Youtube to Spotify Playlist Converter (Backend Implementation)
Hey guys, This is a guide to make a
youtubetospotify musicplaylist converter usingnode jsfor back-end andreact jsfor front-end.“Front-end is in progress and will be posting about it soon.”
Prerequisites : “Knowledge of node js, express js and basics of react states and react routing”
Lets keep it simple !
The front end of the application will be simple, it will have 3 steps. One is to login to google, then login to spotify and then finally giving a youtube playlist URL and name of the the playlist to be created on spotify.
The back-end however will be a ted bit complex as we will have to dealing with oauth 2.0 for both 3rd party application and will have to make multiple external API calls to achieve our end goal.
Here is the flow in short.
First we will be to create the oauth link for the third party application (will be same for any application, in our case google and spotify) in order to take permission to access/modify users data on youtube/spotify, this link is basically a URL to redirect user to the consent screen. As per the oauth 2.0 flow (If you are unfamiliar to the flow i highly recommend seeing the spotify documentation. The flow will remain the same for google as well as it is the generic flow of oauth 2.0) after the consent screen we will have access to the code required to get the expiry time plus the API and refresh token. API tokens will be used to call the third party APIs and the refresh token will be used to get a new API key if the access token is expired, which will be calculated by their expiry. We will save the related information in cockraochDB cloud cluster (However which db to use is irrelevant for our use case and will not make much difference hence, feel free to use any db of your choice).
Setup for back-end.
Installing dependencies.
All required dependencies for server are as follows.
"dependencies": {
"axios": "^1.5.0",
"cors": "^2.8.5",
"dotenv": "^16.3.1",
"express": "^4.18.2",
"jsonwebtoken": "^9.0.2",
"pg": "^8.11.3"
}Use this command to install them.
npm i axios cors dotenv express jsonwebtoken pgFolder structure for our server.
I have used express for creating the back-end server. Code to create one lies in the server.js file. Starting from config we will have configurations related to the service, database and oauth options. The code for the config file is as follows.
module.exports.serverConfig = {
defaultPort: 4040,
defaultHost: 'http://localhost'
}
module.exports.googleOauthOptionsConfig = {
access_type: 'offline',
response_type: 'code',
prompt: 'consent',
scope: [
'https://www.googleapis.com/auth/userinfo.profile',
'https://www.googleapis.com/auth/userinfo.email',
'https://www.googleapis.com/auth/youtube',
'https://www.googleapis.com/auth/youtube.readonly',
].join(' ')
}
module.exports.spotifyOauthOptionsConfig = {
response_type: 'code',
show_dialog: true,
scope: [
'playlist-modify-private',
'playlist-modify-public'
].join(' ')
}
module.exports.cockroachCloudCluster = {
connectionString: 'YOUR_CONNECTION_STRING',
dbName: 'playlistconverter_db'
}The serverConfig key of the exports object will have details like PORT and HOST which will be used to create the server. googleOauthOptionsConfig and spotifyOauthOptionsConfig will have details used to generate the link to redirect user to the consent screen. Once important paramerter to keep in mind is the scope. This is a string of space separated URLs and Options of google or spotify accessible data which (if required) must be configured in the oauth app of the thrid party application. I will walk you through an overview of creating the aouth app in a while so bear with me here. The cockroachCloudCluster configuration will solely depend on your way of connection to you prefered database. In my case i am connection to cockroach cloud using the string parameter.
Now let us start with configuring the third party oauth apps and the db cloud cluster.
Lets start with google app configuration.
Go to your GCP console login or create a new account.
Create a new project or use an existing one.
Go to credentials and click on create credentials. You should be able to see options to generate API key and OAuth Client ID. Clicking on the option should generate an API key in the list below. Make sure the click on show key option and save the key locally somewhere safe or use can create a env file and save your key in that file. In case you do not know how to use an env file checkout this docs for using dotenv package.
Then after click on creating OAuth client ID, select the application type as web application and give a name as per your wish. Add an URI in the redirect options. This is important as it determines the URI to which google app will redirect to after user has given consent to access its data. In our case the redirect URL will be the front-end route. It will also add parameters like code in the query string of the redirect URI, example.
http://localhost:3000/?code=4/0AfJohXldUeyubWfkNF469wL6xyFEgOw_Wa5alEU-GxsHElDzXgTZR0sZmZ8QS4U-g7McEg&scope=email%20profile&authuser=0&prompt=consentThe parameter we are interested in here is code. The idea behind adding our front-end router is that we will use this code and make a call to our back-end server to get the access token for the options given in the scope. Click on create and client id and secret will be created. In the listing page you can click on the download option to save the client id and secret. Use it and create a file name oauth-creeds-google.js in the root directory. Alternatively you can store it in dotenv file as well (It is preferable to store it in env file for more security, however for this explanation i have store it in a js file).
module.exports = Object.freeze({
clientId: CLIENT_ID,
clientSecret: CLIENT_SECRET,
redirectUrl: 'http://localhost:3000/',
rootUrlForAccessToken: 'https://oauth2.googleapis.com/token',
rootUrl: 'https://accounts.google.com/o/oauth2/v2/auth',
})Along with clientId and secret we store the redirectUrl as mentioned on during configuring the app and we store rootUrl (It is used in generating the link for the consent screen)and rootUrlForAccessToken (It is the url used make API call to the third party for getting the access token).
The dashboard should look something like this after API key and client id are generated.
Now we will be configuring the scope in the oAuthconsent screen.
Here i will be filling out only necessary details for local testing for production ready application you must fill out all the details.
First select the user type as external and click on create. Fill out the basic details like app name and support email and put in your application domain. In our case we will just keep it to localhost:3000 and it is sufficient for local development.
The scope added in configuration must be added here as well. Note, you might find some extra scopes in the image below but you don’t need them for the converter. I have added them as i was using the app for some other use cases as well. You can ignore them. Just refer the once mentioned in the configuration.
Spotify app configuration.
The entire process will remain the same if not easy in case of spotify as the UI for the dashboard is basic and easy to grasp, so i will keep it short.
Go to spotify for developers and login or create a new account.
Enter the details and as done previously keep the redirect URI as the react route of the front end application you want to redirect to. In our case it will be http://localhost:3000/spotify.
Get Saumyadixit07’s stories in your inbox
Join Medium for free to get updates from this writer.
Once the app is created you should be able to go and see the client id and secret from settings.
Save the details in the root directory as done before. No need to add scopes separately in spotify you can refer the docs to see the different types of scopes available.
Code base : back-end.
The flow is written in the controller and other files are merely helpers used to keep the code simple and reusable. So stay tuned for that during the next section.
server.js (root folder)
const express = require('express');
const cors = require('cors');
const oauthRoutes = require('./routes/oauthRoutes');
const playlistRoutes = require('./routes/playlistRoutes');
const config = require('./config/config')
const app = express();
app.use(express.json());
app.use(cors());
app.use('/oauth',oauthRoutes);
app.use('/playlist', playlistRoutes);
const port = process.env.PORT || process.argv[2] || config.serverConfig.defaultPort
app.listen(port, (error) => error ?
console.info('Error In Starting The Service', error) :
console.info(`Service Started on ${config.serverConfig.defaultHost}:${port}`));Nothing much going on here make a server using express user server configs and import routes from routers rootDir/routes/featureRoutes.
oauthRoutes.js (routes)
All the routes related to oauth for the application.
const express = require('express');
const oauthRoutes = express.Router();
const {
getGoogleOauthLink,
getSpotifyOauthLink,
spotifyAuthenticate,
googleAuthenticate,
} = require('../controller/oauthController');
oauthRoutes
.route('/oauthlink/google')
.get(getGoogleOauthLink);
oauthRoutes
.route('/oauthlink/spotify')
.get(getSpotifyOauthLink);
oauthRoutes
.route('/authenticate/spotify')
.post(spotifyAuthenticate);
oauthRoutes
.route('/authenticate/google')
.post(googleAuthenticate);
module.exports = oauthRoutes;playlistRoutes.js (routes)
All the routes related to playlist operations like conversion in the application.
const express = require('express');
const playlistRoutes = express.Router();
const refreshTokenIfExpired = require('../middleware/refreshTokenIfExpired');
const {
convertYoutubePlaylist
} = require('../controller/playlistController');
playlistRoutes
.route('/convert-youtube-playlist/:userId')
.post(refreshTokenIfExpired, convertYoutubePlaylist);
module.exports = playlistRoutes;cockroachDbConnection.js (db folder)
const CockraochDBUtils = require('../utils/CockraochDBUtils');
const { cockroachCloudCluster } = require('../config/config');
const cockroach = new CockraochDBUtils({
connectionString: cockroachCloudCluster.connectionString,
database: cockroachCloudCluster.dbName,
});
module.exports = cockroach;In a nutshell importing created custom cockroachdb utilities (Will be explained later) and db configurations and creating a connection instance.
queries.js (db folder)
These file consists of all the database calls both for the user table and for the user token details table.
const cockroach = require('./cockroachDbConnection');
const { cockroachCloudCluster } = require('../config/config');
const database = cockroachCloudCluster.dbName;
const TABLE_NAME_USERS = 'users'
const TABLE_NAME_USER_TOKENS = 'token_details'
// USERS
async function addUser({ email }) {
try {
const fields = ['email', 'created_at', 'modified_at']
const values = [email, new Date(), new Date()];
const query = `
INSERT INTO
${TABLE_NAME_USERS}
(${fields})
VALUES
(${values.map((_, index) => `$${index + 1}`)})
RETURNING id;
`;
const result = await cockroach.executeQuery({
database,
query,
values,
});
if (!result || !result.rows || !result.rows.length) {
return false;
}
return result.rows[0];
} catch (err) {
console.error(err);
throw new Error('Error while adding user');
}
}
async function getUserById({ id, fieldsToQuery }) {
try {
const values = [id];
const query = `
SELECT
${fieldsToQuery ? fieldsToQuery : 'id'}
FROM
${TABLE_NAME_USERS}
WHERE id = $1;
`;
const result = await cockroach.executeQuery({
database,
query,
values,
});
if (!result || !result.rows || !result.rows.length) {
return false;
}
return result.rows[0];
} catch (err) {
console.error(err);
throw new Error('Error while getting user by id');
}
}
async function getUserByEmail({ email, fieldsToQuery }) {
try {
const values = [email];
const query = `
SELECT
${fieldsToQuery ? fieldsToQuery : 'id'}
FROM
${TABLE_NAME_USERS}
WHERE email = $1;
`;
const result = await cockroach.executeQuery({
database,
query,
values,
});
if (!result || !result.rows || !result.rows.length) {
return false;
}
return result.rows[0];
} catch (err) {
console.error(err);
throw new Error('Error while getting user by email');
}
}
// TOKEN
async function addTokenDetails({
userId,
accessTokenGoogle,
accessTokenSpotify,
refreshTokenGoogle,
refreshTokenSpotify,
googleTokenExpiresAt,
spotifyTokenExpiresAt,
createdBy,
}) {
try {
const fields = [
'user_id',
'access_token_google',
'access_token_spotify',
'refresh_token_google',
'refresh_token_spotify',
'google_token_expires_at',
'spotify_token_expires_at',
'created_at',
'created_by',
'modified_at',
'modified_by',
]
const values = [
userId,
accessTokenGoogle,
accessTokenSpotify,
refreshTokenGoogle,
refreshTokenSpotify,
googleTokenExpiresAt,
spotifyTokenExpiresAt,
new Date(),
createdBy,
new Date(),
createdBy,
]
const query = `
INSERT INTO
${TABLE_NAME_USER_TOKENS}
(${fields})
VALUES
(${values.map((_, index) => `$${index + 1}`)})
RETURNING id;
`;
const result = await cockroach.executeQuery({
database,
query,
values,
});
if (!result || !result.rows || !result.rows.length) {
return false;
}
return result.rows[0];
} catch (err) {
console.error(err);
throw new Error('Error while adding user');
}
}
async function updateTokenDetails({
id,
userId,
accessTokenGoogle,
accessTokenSpotify,
refreshTokenGoogle,
refreshTokenSpotify,
googleTokenExpiresAt,
spotifyTokenExpiresAt,
updatedBy
}) {
try {
const fields = ['user_id', 'modified_by', 'modified_at']
const values = [id, userId, updatedBy, new Date()]
if (accessTokenGoogle) {
fields.push('access_token_google')
values.push(accessTokenGoogle)
}
if (accessTokenSpotify) {
fields.push('access_token_spotify')
values.push(accessTokenSpotify)
}
if (refreshTokenGoogle) {
fields.push('refresh_token_google')
values.push(refreshTokenGoogle)
}
if (refreshTokenSpotify) {
fields.push('refresh_token_spotify')
values.push(refreshTokenSpotify)
}
if (googleTokenExpiresAt) {
fields.push('google_token_expires_at')
values.push(googleTokenExpiresAt)
}
if (spotifyTokenExpiresAt) {
fields.push('spotify_token_expires_at')
values.push(spotifyTokenExpiresAt)
}
const query = `
UPDATE
${TABLE_NAME_USER_TOKENS}
SET
${fields.map((field, index) => `${field} = $${index + 2}`)}
WHERE id = $1
RETURNING id;
`;
const result = await cockroach.executeQuery({
database,
query,
values,
});
if (!result || !result.rows || !result.rows.length) {
return false;
}
return result.rows[0];
} catch (err) {
console.error(err);
throw new Error('Error while adding user');
}
}
async function updateTokenDetailsByUserId({
userId,
accessTokenGoogle,
accessTokenSpotify,
refreshTokenGoogle,
refreshTokenSpotify,
googleTokenExpiresAt,
spotifyTokenExpiresAt,
sporitfyUserId,
updatedBy
}) {
try {
const fields = ['modified_by', 'modified_at']
const values = [userId, updatedBy, new Date()]
if (accessTokenGoogle) {
fields.push('access_token_google')
values.push(accessTokenGoogle)
}
if (accessTokenSpotify) {
fields.push('access_token_spotify')
values.push(accessTokenSpotify)
}
if (refreshTokenGoogle) {
fields.push('refresh_token_google')
values.push(refreshTokenGoogle)
}
if (refreshTokenSpotify) {
fields.push('refresh_token_spotify')
values.push(refreshTokenSpotify)
}
if (sporitfyUserId) {
fields.push('spotify_user_id')
values.push(sporitfyUserId)
}
if (googleTokenExpiresAt) {
fields.push('google_token_expires_at')
values.push(googleTokenExpiresAt)
}
if (spotifyTokenExpiresAt) {
fields.push('spotify_token_expires_at')
values.push(spotifyTokenExpiresAt)
}
const query = `
UPDATE
${TABLE_NAME_USER_TOKENS}
SET
${fields.map((field, index) => `${field} = $${index + 2}`)}
WHERE user_id = $1
RETURNING id;
`;
const result = await cockroach.executeQuery({
database,
query,
values,
});
if (!result || !result.rows || !result.rows.length) {
return false;
}
return result.rows[0];
} catch (err) {
console.error(err);
throw new Error('Error while adding user');
}
}
async function getTokenDetailsByUserId({ userId, fieldsToQuery }) {
try {
const values = [userId]
const query = `
SELECT
${fieldsToQuery ? fieldsToQuery : '*'}
FROM
${TABLE_NAME_USER_TOKENS}
WHERE
user_id = $1;
`;
const result = await cockroach.executeQuery({
database,
query,
values,
});
if (!result || !result.rows || !result.rows.length) {
return false;
}
return result.rows[0];
} catch (err) {
console.error(err);
throw new Error('Error while getting token details by user id');
}
}
module.exports = Object.freeze({
// USERS
addUser,
getUserById,
getUserByEmail,
// TOKENS
addTokenDetails,
getTokenDetailsByUserId,
updateTokenDetails,
updateTokenDetailsByUserId,
})The user table is a basically just stores the email and user id. The user id will play an important role here as it will be referencing to the token details which will be stored in the token details table. Token details table will have all information related to users token, like google and spotify access tokens, refresh tokens, expiry and an additional filed spotify id which is required to make spotify API calls.
constants.js (utils folder)
const constants = {
responseSuccess: {
status: 'success',
data: null,
},
responseError: {
status: 'error',
message: 'something went wrong'
},
spotifyAPIs: {
userProfile: {
method: 'get',
url: 'https://api.spotify.com/v1/me'
},
createPlaylist: {
method: 'post',
url: 'https://api.spotify.com/v1/users/:userId/playlists'
},
searchPlaylist: {
method: 'get',
url: 'https://api.spotify.com/v1/search',
config: {
type_track: 'track',
limit: 1,
offset: 0,
}
},
addItemToPlaylist: {
method: 'post',
url: 'https://api.spotify.com/v1/playlists/:playlistId/tracks',
}
},
googleAPIs: {
playlistDetails: {
method: 'get',
url: 'https://youtube.googleapis.com/youtube/v3/playlistItems',
config: {
part: 'snippet',
maxResults: 50, // a max of 50 can be called at a time
}
}
}
}
module.exports = constants;Defining some constants, like base success and error JSON response. Other than that have creates a template objects used for making 3rd party API calls having primarily 3 things method, url, config (optional) which will be used to make API calls to the 3rd Party.
CockroachDBUtils.js (utils folder)
This is a custom utility made to act as a wrapper for making db calls.
const { Pool } = require('pg');
class CockraochDBUtils {
constructor({ connectionString, database }) {
this.cockroachPool = new Pool({
connectionString,
});
this.database = database;
}
async getConnection({ database }) {
const connection = await this.cockroachPool.connect();
await connection.query(`USE ${database}`);
return connection;
}
async executeQuery({ database, query, values }) {
let connection;
try {
connection = await this.getConnection({ database });
return await connection.query({
text: query,
values: values ? values : [],
});
} catch (e) {
throw e;
} finally {
if (connection) {
await connection.release();
}
}
}
}
module.exports = CockraochDBUtils;Upon initialisation of the utility a connection string must be given and the name of default database can be given. You can also specify database explicitly in case you are using multiple db. However here we will be using only one. The executeQuery function will be called to query the database.
GoogleAPIUtils.js (utils folder)
This utility basically have all the functions related to making googles 3rd party API calls. These calls will be using the configurations provided during making the instance and the details of the API calls are pre-defined in the constants.js file as see above. Comments in the code are self explanatory.
const axios = require('axios')
require('dotenv').config()
const { googleAPIs } = require('../utils/constants');
class GoogleAPIUtils {
constructor({
googleOauthConfig,
googleOauthOptionsConfig,
}) {
this.googleOauthOptionsConfig = googleOauthOptionsConfig;
this.googleOauthConfig = googleOauthConfig;
}
// CALL TO GENERATE LINK FOR REDIRECTION TO CONSENT SCREEN
generateOauthLink() {
try {
const options = {
redirect_uri: this.googleOauthConfig.redirectUrl,
client_id: this.googleOauthConfig.clientId,
...this.googleOauthOptionsConfig,
}
const qs = new URLSearchParams(options)
return `${this.googleOauthConfig.rootUrl}?${qs.toString()}`
} catch (e) {
throw e;
} finally {
console.info('[GoogleAPIUtils] :: Processing done ====> generateOauthLink')
}
}
// CALL TO GET TOKEN DETAILS
async getAccessToken({ code }) {
try {
const tokenDetails = await axios({
method: 'post',
url: this.googleOauthConfig.rootUrlForAccessToken,
headers: {
'Content-Type': 'application/x-www-form-urlencoded'
},
data: new URLSearchParams({
client_id: this.googleOauthConfig.clientId,
client_secret: this.googleOauthConfig.clientSecret,
code,
redirect_uri: this.googleOauthConfig.redirectUrl,
grant_type: 'authorization_code'
}).toString()
})
return tokenDetails.data;
} catch (e) {
throw e;
} finally {
console.info('[GoogleAPIUtils] :: Processing done ====> getAccessToken')
}
}
async updateAccessToken({ refreshToken }) {
try {
const tokenDetails = await axios({
method: 'post',
url: this.googleOauthConfig.rootUrlForAccessToken,
headers: {
'Content-Type': 'application/x-www-form-urlencoded'
},
data: new URLSearchParams({
client_id: this.googleOauthConfig.clientId,
client_secret: this.googleOauthConfig.clientSecret,
refresh_token: refreshToken,
grant_type: 'refresh_token'
}).toString()
})
return tokenDetails.data;
} catch (e) {
throw e;
} finally {
console.info('[GoogleAPIUtils] :: Processing done ====> updateAccessToken')
}
}
// CALL TO GET PLAYLIST DETAILS
async getPlaylistDetails({ accessToken, metaData, wantedItems = ['trackNames'], nextPageToken = null }) {
try {
const params = {
...googleAPIs.playlistDetails.config,
playlistId: metaData.playlistId,
key: process.env.GOOGLE_API_KEY
}
if (nextPageToken) params['pageToken'] = nextPageToken;
const googleQueryParams = new URLSearchParams(params)
const playlistDetails = await axios({
method: googleAPIs.playlistDetails.method,
url: `${googleAPIs.playlistDetails.url}?${googleQueryParams.toString()}`,
headers: {
'Authorization': `Bearer ${accessToken}`,
'Content-Type': 'application/json'
}
})
const returnObject = {
nextPageToken: playlistDetails.data.nextPageToken || null,
};
wantedItems.forEach((wantedItem) => {
switch (wantedItem) {
case 'trackNames': {
returnObject['trackNames'] = playlistDetails.data.items.map((item) => item.snippet.title)
}
default: {}
}
});
return returnObject;
} catch (e) {
throw e;
} finally {
console.info('[GoogleAPIUtils] :: Processing done ====> getPlaylistDetails')
}
}
}
module.exports = GoogleAPIUtils;SpotifyAPIUtils.js (utils folder)
Similar utility but for spotify. Read comments for more clarity.
const axios = require('axios')
require('dotenv').config()
const { spotifyAPIs } = require('../utils/constants');
class SpotifyAPIUtils {
constructor({
spotifyOauthConfig,
spotifyOauthOptionsConfig,
}) {
this.spotifyOauthOptionsConfig = spotifyOauthOptionsConfig;
this.spotifyOauthConfig = spotifyOauthConfig;
}
// CALL TO GENERATE LINK FOR REDIRECTION TO CONSENT SCREEN
generateOauthLink({ state }) {
try {
const options = {
redirect_uri: this.spotifyOauthConfig.redirectUrl,
client_id: this.spotifyOauthConfig.clientId,
state,
...this.spotifyOauthOptionsConfig,
}
const qs = new URLSearchParams(options)
return `${this.spotifyOauthConfig.rootUrl}?${qs.toString()}`
} catch (e) {
throw e;
} finally {
console.info('[SpotifyAPIUtils] :: Processing done ====> generateOauthLink')
}
}
// CALL TO GET TOKEN DETAILS
async getAccessToken({ code }) {
try {
const tokenDetails = await axios({
method: 'post',
url: this.spotifyOauthConfig.rootUrlForAccessToken,
headers: {
'Authorization': 'Basic ' + (new Buffer.from(this.spotifyOauthConfig.clientId + ':' + this.spotifyOauthConfig.clientSecret).toString('base64')),
'Content-Type': 'application/x-www-form-urlencoded'
},
data: new URLSearchParams({
code,
redirect_uri: this.spotifyOauthConfig.redirectUrl,
grant_type: 'authorization_code'
}).toString()
})
return tokenDetails.data;
} catch (e) {
throw e;
} finally {
console.info('[SpotifyAPIUtils] :: Processing done ====> getAccessToken')
}
}
async updateAccessToken({ refreshToken }) {
try {
const tokenDetails = await axios({
method: 'post',
url: this.spotifyOauthConfig.rootUrlForAccessToken,
headers: {
'Authorization': 'Basic ' + (new Buffer.from(this.spotifyOauthConfig.clientId + ':' + this.spotifyOauthConfig.clientSecret).toString('base64')),
'Content-Type': 'application/x-www-form-urlencoded'
},
data: new URLSearchParams({
refresh_token: refreshToken,
grant_type: 'refresh_token'
}).toString()
})
return tokenDetails.data;
} catch (e) {
throw e;
} finally {
console.info('[SpotifyAPIUtils] :: Processing done ====> updateAccessToken')
}
}
// CALL TO GET SPOTIFY USER DETAILS
async getSpotifyUserDetails({ accessToken }) {
try {
const getSpotifyUserDetails = await axios({
method: spotifyAPIs.userProfile.method,
url: spotifyAPIs.userProfile.url,
headers: {
'Authorization': `Bearer ${accessToken}`,
}
})
return getSpotifyUserDetails.data
} catch (e) {
throw e;
} finally {
console.info('[SpotifyAPIUtils] :: Processing done ====> getSpotifyUserDetails')
}
}
// CALL TO CREATE SPOTIFY PLAYLIST
async createSpotifyPlaylist({ accessToken, userId, playlistName, playlistDescription }) {
try {
const newSpotifyPlayList = await axios({
method: spotifyAPIs.createPlaylist.method,
url: spotifyAPIs.createPlaylist.url.replace(':userId', userId),
headers: {
'Authorization': `Bearer ${accessToken}`,
},
data: {
"name": playlistName,
"description": playlistDescription,
"public": false
}
})
return newSpotifyPlayList.data
} catch (e) {
throw e;
} finally {
console.info('[SpotifyAPIUtils] :: Processing done ====> createSpotifyPlaylist')
}
}
// CALL TO SEARCH A TRACT ON SPOTIFY
async searchSpotifyTrackByName({ accessToken, trackName }) {
try {
if (!trackName) {
return ''
}
const options = {
// fixed format to search using the remaster%20 or remaster<SPACE> URI option
q: `remaster%20${spotifyAPIs.searchPlaylist.config.type_track}:${trackName}`,
type: spotifyAPIs.searchPlaylist.config.type_track,
limit: spotifyAPIs.searchPlaylist.config.limit,
offset: spotifyAPIs.searchPlaylist.config.offset,
}
const qs = new URLSearchParams(options)
const searchResult = await axios({
method: spotifyAPIs.searchPlaylist.method,
url: `${spotifyAPIs.searchPlaylist.url}?${qs}`,
headers: {
'Authorization': `Bearer ${accessToken}`,
}
})
let unableToConvert = null;
let uri = null;
if (searchResult.data.tracks.items.length) {
uri = searchResult.data.tracks.items[0].uri;
} else {
unableToConvert = trackName;
}
return {unableToConvert, uri};
} catch (e) {
throw e;
} finally {
console.info('[SpotifyAPIUtils] :: Processing done ====> searchSpotifyTrackByName')
}
}
// CALL TO ADD ITEMS TO PLAYLIST (100 ITEMS CAN BE ADDED AT A TIME)
async addItemToSpotifyPlaylist({ accessToken, playlistId, playlistURIs = [] }) {
try {
const response = await axios({
method: spotifyAPIs.addItemToPlaylist.method,
url: spotifyAPIs.addItemToPlaylist.url.replace(':playlistId', playlistId),
headers: {
'Authorization': `Bearer ${accessToken}`,
},
data: {
uris: playlistURIs,
}
})
return response.data;
} catch (e) {
throw e;
} finally {
console.info('[SpotifyAPIUtils] :: Processing done ====> addItemToSpotifyPlaylist')
}
}
}
module.exports = SpotifyAPIUtils;helperFunctions.js (utils folder)
It has all functions which will be used to some miscellaneous tasks having such functions in isolation is a good practice as now they are reusabe and can be used in any controller.
module.exports.generateRandomString = function(length) {
var text = '';
var possibleValues = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789';
for (var i = 0; i < length; i++) {
text += possibleValues.charAt(Math.floor(Math.random() * possibleValues.length));
}
return text;
};
module.exports.wait = async function(timeInSec) {
console.info(`[wait called] ==> WAITING FOR ${timeInSec} SEC`)
await new Promise(resolve => setTimeout(resolve, timeInSec * 1000));
}
module.exports.splitArray = function({parts = 10, arr}) {
let start = 0;
let end = parts;
const response = [];
while (start < arr.length) {
response.push(arr.slice(start, end))
start = end;
end += parts;
if (end > arr.length) end = arr.length
}
return response;
}
module.exports.getTime = function(date) {
if(date) return new Date(date).getTime();
else return new Date();
}Before i discuss the controllers and middleware lets look into the flow of managing the tokens and users in the database. The application flow that we know after the current implementation is as follows. Front-end client will call out /oauthlink/google API to get the link for consent screen. Upon clicking the link user will be redirected to the consent screen and if the consent is given user will be redirected to our front-end application, as soon as the user is redirected the front end will call for /authenticate/spotify where in we will be getting the token details and store them in the database. However for security purpose the token will be valid till 1 hr in case of spotify and 59 mins in case of google, to counter this we will have a middleware for the routes which uses the access token to make requests to the third party (in our case the router to convert the playlist /playlist/convert-youtube-playlist), which will use the refresh token of the third party application request the third party to regenerate the access token, the updated details will be updated in the database as well. Once the process is done the front-end of the will be configured to redirect user to /spotify route (using react routing) of front-end where in the front-end client will as for spotify link i.e. /oauthlink/spotify for consent screen and the same process will be repeated. The user can then paste the youtube playlist link and give the playlist name of the playlist to be created and request the route /playlist/convert-youtube-playlist to perform the task. The task will be can take a lot of time depending upon the playlist length. Hence it is important to implement some kind of loader in the front-end as well.
oauthController.js (controller)
All the controllers related to oauth, the logic of the application goes here.
const jwt = require('jsonwebtoken');
const { responseSuccess, responseError } = require('../utils/constants');
const {
googleOauthOptionsConfig,
spotifyOauthOptionsConfig,
} = require('../config/config')
const { generateRandomString } = require('../utils/helperFunctions');
const googleOauthConfig = require('../oauth-creeds-google');
const spotifyOauthConfig = require('../oauth-creeds-spotify');
const GoogleAPIUtils = require('../utils/GoogleAPIUtils');
const googleAPICall = new GoogleAPIUtils({
googleOauthConfig,
googleOauthOptionsConfig,
});
const SpotifyAPIUtils = require('../utils/SpotifyAPIUtils');
const spotifyAPICall = new SpotifyAPIUtils({
spotifyOauthConfig,
spotifyOauthOptionsConfig,
});
const {
addUser,
getUserById,
getUserByEmail,
updateTokenDetails,
updateTokenDetailsByUserId,
getTokenDetailsByUserId,
addTokenDetails,
} = require('../db/queries');
module.exports.getGoogleOauthLink = function (_, res) {
try {
const oauthLink = googleAPICall.generateOauthLink();
res.status(200).send({ ...responseSuccess, data: oauthLink });
} catch (err) {
res.status(400).send({ ...responseError, message: 'Error while generating oauth link for google' });
}
}
module.exports.googleAuthenticate = async function (req, res) {
try {
if (!req.headers['auth-code']) {
res.status(200).send({ ...responseError, message: 'Auth code not found' });
}
const tokenDetails = await googleAPICall.getAccessToken({ code: req.headers['auth-code'] });
const { email } = jwt.decode(tokenDetails.id_token);
// expires_in is in seconds hence convert it in milliseconds and saving it
const expiryTime = new Date(Date.now() + (tokenDetails.expires_in * 1000));
const userDetails = await getUserByEmail({ email, fieldsToQuery: ['id'] });
let userId = null;
if (!userDetails) {
const newUserDetails = await addUser({ email });
userId = newUserDetails.id;
} else {
userId = userDetails.id;
}
const userTokenDetails = await getTokenDetailsByUserId({ userId });
if (!(userTokenDetails.access_token_google && userTokenDetails.refresh_token_google)) {
await addTokenDetails({
userId,
accessTokenGoogle: tokenDetails.access_token,
refreshTokenGoogle: tokenDetails.refresh_token,
createdBy: userId,
googleTokenExpiresAt: expiryTime
});
} else {
await updateTokenDetails({
id: userTokenDetails.id,
userId,
accessTokenGoogle: tokenDetails.access_token,
refreshTokenGoogle: tokenDetails.refresh_token,
updatedBy: userId,
googleTokenExpiresAt: expiryTime
});
}
res.status(200).send({
...responseSuccess, data: {
message: 'google authenticated',
metaData: {
id: userId
}
}
});
} catch (err) {
res.status(400).send({ ...responseError, message: 'Error while google authentication' });
}
}
module.exports.getSpotifyOauthLink = function (_, res) {
try {
const state = generateRandomString(16);
const oauthLink = spotifyAPICall.generateOauthLink({ state });
res.status(200).send({ ...responseSuccess, data: oauthLink });
} catch (err) {
res.status(400).send({ ...responseError, message: 'Error while generating oauth link for spotify' });
}
}
module.exports.spotifyAuthenticate = async function (req, res) {
try {
if (!req.headers['auth-code']) {
throw new Error('Auth code not found');
}
const userId = req.query.userId;
const userDetails = await getUserById({ id: req.query.userId });
if (!userDetails) {
throw new Error('Please login to google first');
}
const tokenDetails = await spotifyAPICall.getAccessToken({ code: req.headers['auth-code'] });
// expires_in is in seconds hence convert it in milliseconds and saving it
const expiryTime = new Date(Date.now() + (tokenDetails.expires_in * 1000));
const sopritfyUserDetails = await spotifyAPICall.getSpotifyUserDetails({ accessToken: tokenDetails.access_token });
await updateTokenDetailsByUserId({
userId,
accessTokenSpotify: tokenDetails.access_token,
refreshTokenSpotify: tokenDetails.refresh_token,
updatedBy: userId,
sporitfyUserId: sopritfyUserDetails.id,
spotifyTokenExpiresAt: expiryTime
});
res.status(200).send({ ...responseSuccess, data: 'sportify authenticated' });
} catch (err) {
res.status(400).send({ ...responseError, message: 'Error while sportify authentication' });
}
}Let’s see the controllers one by one.
- getGoogleOauthLink and getSpotifyOauthLink is pretty self explanatory.
- googleAuthenticate is the controller that is mainly responsible for getting the token details based on the code generated during getting consent from the user. However it also deals with storing new user details after google authentication. You see in after making the call to get token details we have get fields like id_token, access_token, refresh_token and expires_in. The id_token is simply a jwt token, upon decrypting it we get our user details one of which is users email. We can then take this email and create a user in the users table. Using the id of the user generated we create a new entry or update the existing entry in the token table storing access token, refresh token, expiry and update details like updated by and updated at. In our case at this scale we technically we do not need update by as the app does not have a admin user of some other which can login and manuplate some other users details. However it can be implemented in the future and as per standard practice i am storing the updated by field as well. Once the process is completed the generated user id or the preexisting user id will be sent in the response. Which will be used by the front end to make other API calls.
- Similar to the one explained above spotifyAuthenticate is the controller that is mainly responsible for getting the token details based on the code generated during getting consent from the user. It then uses the user id (which by this time must be available as discussed above) to update the user token details. It first makes call to get spotify token details we get fields like access_token, refresh_token and expires_in. However we will also need the spotify user id to make spotify API calls. Hence, we make call to get user details and store the spotify user id in the database along with access_token, refresh_token and expires_in. Note that we will store the exact expiry date time instead of time in seconds for better understating of the of expiry while examining the database.
refreshTokenIfExpired.js (middleware)
This middleware will be used making API calls using the tokens as we know that the access tokens have an expiry hence we must update the access token if it is expired.
const { responseError } = require('../utils/constants')
const { googleOauthOptionsConfig, spotifyOauthOptionsConfig } = require('../config/config')
const { getTime } = require('../utils/helperFunctions')
const googleOauthConfig = require('../oauth-creeds-google')
const spotifyOauthConfig = require('../oauth-creeds-spotify')
const {
updateTokenDetailsByUserId,
getTokenDetailsByUserId,
} = require('../db/queries');
const GoogleAPIUtils = require('../utils/GoogleAPIUtils');
const googleAPICall = new GoogleAPIUtils({
googleOauthConfig,
googleOauthOptionsConfig,
});
const SpotifyAPIUtils = require('../utils/SpotifyAPIUtils');
const spotifyAPICall = new SpotifyAPIUtils({
spotifyOauthConfig,
spotifyOauthOptionsConfig,
});
const refreshTokenIfExpired = async function (req, res, next) {
const userId = req.params.userId;
const tokenDetails = await getTokenDetailsByUserId({ userId });
if (!tokenDetails) {
res.status(404).send({ ...responseError, message: 'Please login to google first' })
}
const googleExpiryDate = getTime(tokenDetails.google_token_expires_at);
const spotifyExpiryDate = getTime(tokenDetails.spotify_token_expires_at);
// checking if tine difference between expiry and now is more than 10min to be sage as the conversion process can take a long time based on playlist items
const maxExpiryAllowed = getTime() - 300000;
if (googleExpiryDate < maxExpiryAllowed) {
const updatedTokenDetails = await googleAPICall.updateAccessToken({ refreshToken: tokenDetails.refresh_token_google });
const expiryTime = new Date(Date.now() + (updatedTokenDetails.expires_in * 1000));
await updateTokenDetailsByUserId({
userId,
accessTokenGoogle: updatedTokenDetails.access_token,
updatedBy: userId,
googleTokenExpiresAt: expiryTime
});
}
if (spotifyExpiryDate < maxExpiryAllowed) {
const updatedTokenDetails = await spotifyAPICall.updateAccessToken({ refreshToken: tokenDetails.refresh_token_spotify });
const expiryTime = new Date(Date.now() + (updatedTokenDetails.expires_in * 1000));
await updateTokenDetailsByUserId({
userId,
accessTokenSpotify: updatedTokenDetails.access_token,
updatedBy: userId,
spotifyTokenExpiresAt: expiryTime
});
}
next();
}
module.exports = refreshTokenIfExpired;We first take the user Id and get the user token details in which we have the token expiry date time. As we know the conversion process can take a lot of time hence for safety we make sure that the expiry date time is less than the current date time minus 5 mins. If that is not the case we just make the call to update the access token and the new expiry date time before starting any process which make third party API calls.
playlistController.js (controller)
This controller does the job of creating a new playlist and adding the tracks to that playlist.
const { responseSuccess, responseError } = require('../utils/constants');
const {
googleOauthOptionsConfig,
spotifyOauthOptionsConfig,
} = require('../config/config')
const { wait, splitArray } = require('../utils/helperFunctions');
const googleOauthConfig = require('../oauth-creeds-google');
const spotifyOauthConfig = require('../oauth-creeds-spotify');
const GoogleAPIUtils = require('../utils/GoogleAPIUtils');
const googleAPICall = new GoogleAPIUtils({
googleOauthConfig,
googleOauthOptionsConfig,
});
const SpotifyAPIUtils = require('../utils/SpotifyAPIUtils');
const spotifyAPICall = new SpotifyAPIUtils({
spotifyOauthConfig,
spotifyOauthOptionsConfig,
});
const {
getTokenDetailsByUserId,
} = require('../db/queries');
module.exports.convertYoutubePlaylist = async function (req, res) {
try {
const userId = req.params.userId;
const playlistUrl = req.body.playlist_url;
const playlistName = req.body.playlist_name;
const playlistDescription = req.body.playlist_description ? req.body.playlist_description : '';
if (!playlistName) {
throw new Error('Please provide playlist name!');
}
// Getting tokens associated to users
const userTokenDetails = await getTokenDetailsByUserId({ userId });
if (!userTokenDetails) {
throw new Error('User token not found. Please login to google first!');
}
// Validating Youtube Playlist URL
const ytRegex = /^https:\/\/www\.youtube\.com\/playlist\?list=[A-Za-z0-9_\-]+(&si=[A-Za-z0-9_\-]+)?$/gm;
const serchParams = playlistUrl.split('?')[1];
const playlistUrlSearchParams = new URLSearchParams(serchParams);
if (!ytRegex.test(playlistUrl)) {
throw new Error('Invalid yt link found!');
} else {
if (!playlistUrlSearchParams.get('list')) {
throw new Error('Please provided valid playlist list id!');
}
}
let nextPageToken = null;
let spotifyTracksUriList = [];
let failedToFindOnSpotify = [];
do {
// Getting youtube playlist playlist details (Can get a max of 50 items)
const googlePlaylistDetails = await googleAPICall.getPlaylistDetails({
accessToken: userTokenDetails.access_token_google,
metaData: {
playlistId: playlistId: playlistUrlSearchParams.get('list'),
},
wantedItems: ['trackNames'],
nextPageToken,
});
nextPageToken = googlePlaylistDetails.nextPageToken;
const spotifyTrackItemsStatus = await Promise.all(googlePlaylistDetails.trackNames.map(async (name) => {
return await spotifyAPICall.searchSpotifyTrackByName({
accessToken: userTokenDetails.access_token_spotify,
trackName: name
});
}));
spotifyTracksUriList.push(...spotifyTrackItemsStatus.filter((item) => item.uri).map((item) => item.uri));
failedToFindOnSpotify.push(...spotifyTrackItemsStatus.filter((item) => item.unableToConvert).map((item) => item.unableToConvert));
if(nextPageToken) await wait(5) // Addded to deal with spotify rate limiting
} while (nextPageToken);
// Once all data is recieved from sportify we we create a new playlist
const newSpotifyPlayList = await spotifyAPICall.createSpotifyPlaylist({
accessToken: userTokenDetails.access_token_spotify,
userId: userTokenDetails.spotify_user_id,
playlistName,
playlistDescription,
})
const splittedSpotifyTracksUriList = splitArray({parts: 100, arr: spotifyTracksUriList});
// Add data to that playlist (Can add A maximum of 100 items)
splittedSpotifyTracksUriList.map(async (uriList) => {
return await spotifyAPICall.addItemToSpotifyPlaylist({
accessToken: userTokenDetails.access_token_spotify,
playlistId: newSpotifyPlayList.id,
playlistURIs: uriList,
})
})
res.status(200).send({ ...responseSuccess, data: {
url: newSpotifyPlayList.external_urls.spotify,
unableToCopy: newSpotifyPlayList
}})
} catch (err) {
console.log(err);
res.status(404).send({ ...responseError, message: 'Error while converting playlist' })
}
}We will have playlist name and description (optional) to be created in spotify and the youtube playlist URL. After validating the youtube URL we make API calls to get all items of the given playlist. As per the documentation we can only get 50 items at a time, after getting all the items we make repetative calls to spotify API which searches for the track based on the track name from google playlist. If it finds the track we take the track uri and make a collection of it, if does not find the track or some error occurs while finding the track we store the track name in a collection for enable to convert tracks, and we do get a parameter which give the next page token from google API using which we can call the API again to get the data and make a list out of it. However spotify API has rate limiting which can cause issues with redundent API calls. Hence, for safety we keep a 5 sec wait time after processing 50 items. Once we have the collection of uris we create a new playlist in spotify and add the uris to that playlist, but we can add only 100 tracks at a time hence we divide the collection of uris in lists of 100 each and process the data. Then after we send the link of the newly created playlist (by default the playlist will be private to the user account, can be changed by the user) and list of items whose conversion failed.
Implementing worker threads — Solving a critical issue
If you even have a little experience with back-end development in node. You might have noticed this problem. The conversion process will take a long time and hence will block the main thread. Meaning, requests to other APIs will be pending make our site super slow and will drastically effect the user experience. Hence, a better approach is to use worker thread to perform the continuous API calls.
Here will will make changes to our playlistController.js file and will add a new file in a new folder <root>/web worker/converterWorker.js
We will basically spawn a new worker thread at the start of the conversion process. So basically. the process of getting youtube playlist items searching them, creating a new playlist in spotify and adding items will be done in the worker thread. We will send required data from parent to child thread and will process the items in the child thread leaving the main thread free to use. After completion of the processing, whenever the main thread gets free the data will be sent to parent which will then send the response as before. The communication will happen based on events in node js. For implementation we will need to change the playlistConverter.js file and will have to add a new file converterWorker.js which will be executed in the worker.
playlistController.js (controller) — Updated
Will remove the unnecessary code, which will be move to the converterWorker.js and will add a variable to get the file location which must be executed in worker.
const { responseSuccess, responseError } = require('../utils/constants');
const { Worker } = require('worker_threads')
const { join } = require('path');
const converterWorkerFile = join(__dirname, '..', 'web workers', 'converterWorker.js')
const {
getTokenDetailsByUserId,
} = require('../db/queries');
module.exports.convertYoutubePlaylist = async function (req, res) {
try {
const userId = req.params.userId;
const playlistUrl = req.body.playlist_url;
const playlistName = req.body.playlist_name;
const playlistDescription = req.body.playlist_description ? req.body.playlist_description : '';
if (!playlistName) {
throw new Error('Please provide playlist name!');
}
// Getting tokens associated to users
const userTokenDetails = await getTokenDetailsByUserId({ userId });
if (!userTokenDetails) {
throw new Error('User token not found. Please login to google first!');
}
// Validating Youtube Playlist URL
const ytRegex = /^https:\/\/(www\.)?youtube\.com\/playlist\?list=[A-Za-z0-9_\-]+(&si=[A-Za-z0-9_\-]+)?$/gm;
const serchParams = playlistUrl.split('?')[1];
const playlistUrlSearchParams = new URLSearchParams(serchParams);
if (!ytRegex.test(playlistUrl)) {
throw new Error('Invalid yt link found!');
} else {
if (!playlistUrlSearchParams.get('list')) {
throw new Error('Please provided valid playlist list id!');
}
}
const workerData = { userTokenDetails, playlistId: playlistUrlSearchParams.get('list'), playlistName, playlistDescription };
// creating worker thread to avoid blocking of main thread while complex converstion process
const conversionResponse = await new Promise((resolve, reject) => {
const converterWorker = new Worker(converterWorkerFile, { workerData });
converterWorker.on('message', resolve);
converterWorker.on('error', reject);
converterWorker.on('exit', (code) => {
if (code !== 0)
reject(new Error(`Worker stopped with exit code ${code}`));
})
});
const { generatedUrl, failedToFindOnSpotify } = conversionResponse;
res.status(200).send({
...responseSuccess, data: {
url: generatedUrl,
unableToCopy: failedToFindOnSpotify
}
})
} catch (err) {
console.error(err);
res.status(404).send({ ...responseError, message: 'Error while converting playlist' })
}
}converterWorker.js (web workers)
Here as you can see their is no change in the flow the only change is that we take the required data from parent and then after processing we send the processed data to the parent.
const { workerData, parentPort } = require('worker_threads');
const { wait, splitArray } = require('../utils/helperFunctions');
const {
googleOauthOptionsConfig,
spotifyOauthOptionsConfig,
} = require('../config/config')
const googleOauthConfig = require('../oauth-creeds-google');
const spotifyOauthConfig = require('../oauth-creeds-spotify');
const GoogleAPIUtils = require('../utils/GoogleAPIUtils');
const googleAPICall = new GoogleAPIUtils({
googleOauthConfig,
googleOauthOptionsConfig,
});
const SpotifyAPIUtils = require('../utils/SpotifyAPIUtils');
const spotifyAPICall = new SpotifyAPIUtils({
spotifyOauthConfig,
spotifyOauthOptionsConfig,
});
let nextPageToken = null;
let spotifyTracksUriList = [];
let failedToFindOnSpotify = [];
(async () => {
try {
const { userTokenDetails, playlistId, playlistName, playlistDescription } = workerData;
do {
// Getting youtube playlist playlist details (Can get a max of 50 items)
const googlePlaylistDetails = await googleAPICall.getPlaylistDetails({
accessToken: userTokenDetails.access_token_google,
metaData: {
playlistId,
},
wantedItems: ['trackNames'],
nextPageToken,
});
nextPageToken = googlePlaylistDetails.nextPageToken;
const spotifyTrackItemsStatus = await Promise.all(googlePlaylistDetails.trackNames.map(async (name) => {
return await spotifyAPICall.searchSpotifyTrackByName({
accessToken: userTokenDetails.access_token_spotify,
trackName: name
});
}));
spotifyTracksUriList.push(...spotifyTrackItemsStatus.filter((item) => item.uri).map((item) => item.uri));
failedToFindOnSpotify.push(...spotifyTrackItemsStatus.filter((item) => item.unableToConvert).map((item) => item.unableToConvert));
if (nextPageToken) await wait(5) // Addded to deal with spotify rate limiting
} while (nextPageToken);
// Once all data is recieved from sportify we we create a new playlist
const newSpotifyPlayList = await spotifyAPICall.createSpotifyPlaylist({
accessToken: userTokenDetails.access_token_spotify,
userId: userTokenDetails.spotify_user_id,
playlistName,
playlistDescription,
})
const splittedSpotifyTracksUriList = splitArray({ parts: 100, arr: spotifyTracksUriList });
// Add data to that playlist (Can add A maximum of 100 items)
splittedSpotifyTracksUriList.map(async (uriList) => {
return await spotifyAPICall.addItemToSpotifyPlaylist({
accessToken: userTokenDetails.access_token_spotify,
playlistId: newSpotifyPlayList.id,
playlistURIs: uriList,
})
})
parentPort.postMessage({ generatedUrl: newSpotifyPlayList.external_urls.spotify, failedToFindOnSpotify });
} catch (error) {
// Send an error message to the parent thread
console.error(error);
parentPort.postMessage({ error: error.message ? error.message : 'conversion process error' });
}
})()Possible Improvements
- Can add capability to completely customise the details of newly created playlist like album cover, letting user decide whether to keep the playlist private or public and so on.
- Can add API to update an existing playlist of the user.
However the base remains the same. Feel free to use make adjustments as per need. You can access the source code using this link.
