Scraper for IIITB LMS Client. This project is under active development and is not ready for production, just yet.
Scrapes IIITB Moodle LMS and returns data as json. For use as API layer of IIITB LMS Client (yet to be built).
Please note that the API is expected to change rapidly before a stable release.
This is a reiteration of LMS client built in Java as a part of course curriculum last year. This NodeJS scraper is expected to be more performant than the last one.
clone and install dependencies to get started.
git clone https://github.com/zeroby0/lms-scraperyarnornpm install
If yarn fails with Failed to execute 'node-gyp clean' (Error: spawn node-gyp ENOENT), install with npm first and use yarn from then on.
Note: Make sure you define your username and password in your environment. Either run
export LMS_USERNAME="Your Username"andexport LMS_PASSWORD="Password"in every terminal session before you run scraper or add them to yourbash_profile.
Note: All settings can be changed by editing config/default.yml.
- Login
- Courses and Private Files from home page
- List of Forums, Activities (Assignments), and Course Materials and link to, descriptions for them
- Activity (Assignment) details, submission and feedback recieved.
- Files uploaded in Course material section
- Forum content not supported right now.
Requireindex.jsfrom where you'd like to use the scraper. like so,const API = require('./index').- API.getContent('your-url')
- Get back JSON
- ??
- Profit!
API automatically resolves the URL and returns a Promise which resolves into JSON.
Example:
api.getContent(url)
.then((result) => {
console.log(result)
})JSON responses are defined in config/default.yml. Here is the reference:
{
context: '<context>',
status: '<status>',
code: '<error-code>',
data: {
data_single: '<data>',
data_multiple: ['<data>']
},
error: '<Error if any>'
}context: indicates the content of the response.
status: indicates the status of the API call. status is success if everything went well.
code: The error code. It is preferable to rely on status than code.
data: The data returned by the call.
url: The url passed in call. Though URL is returned, it's not included in all responses. Errors for example. Do not rely on URL for identifying the response, maintain a record of call and response.
{ // Successful
context:'login',
status:'authenticated',
code:0,
data:{
status:'You are logged in as <Name and rollnumber> (Log out)',
cookies:{
MoodleSession:'*&#%*#&%*&%*#~&$%*&'
}
}
}inavlidCredentials: { context: 'login', status: 'invalidCredentials', code: 1 } // Invalid username or password
cannotConnect: { context: 'login', status: 'cannotConnect', code: 3 } // Network Unavailable
unknownError: { context: 'login', status: 'Error', error: '<error>', code: 4 } // an unexpected error occured{
courseList:{
context:'courseList',
status:'success',
data:{
links:[Array],
names:[Array]
},
code:0
},
privateFiles:{
context:'privateFiles',
status:'success',
data:{
links:[Array],
names:[Array]
},
code:0
}
}cannotConnect: { context: 'home', status: 'cannotConnect', code: 1 } // Network unreachable
unknownError: { context: 'home', status: 'error', error: '<Error>', code: 2 } // An unexpected error occured{
context:'course',
status:'success',
url: '<course-url>',
data:{
forums:{
context:'forumList',
status:'success',
data:[
Object
],
code:0
},
activities:{
context:'activityList',
status:'success',
data:[
Object
],
code:0
},
materials:{
context:'materialList',
status:'success',
data:[
Object
],
code:0
}
},
code:0
}cannotConnect: { context: 'course', status: 'cannotConnect', code: 1 } // No network
noForum: { context: 'forumList', status: 'noForum', code: 1 } // No forum for this course
noActivity: { context: 'activityList', status: 'noActivity', code: 1 } // No activities for this course
noMaterial: { context: 'materialList', status: 'noMaterial', code: 1 } // No materials for this course
unknownError: { context: 'course', status: 'Error', error: '<Error>', code: 2 } // An unexpected error occured{
context:'material',
url:'<URL>',
status:'success',
code:0,
data:{
links:[
'<List of links to course materials>'
],
names:[
'<List of names in same order>'
]
}
}cannotConnect: { context: 'material', status: 'cannotConnect', code: 1 } // cannot connect to the internet
noContent: { context: 'material', status: 'noContent', code: 2 } // No content in this section
unknownError: { context: 'material', status: 'Error', error: '<Error>', code: 3 } // An unexpected error occured{
context:'activity',
url:'<URL>',
status:'success',
code:0,
data:{
intro:{
linkTitles:[Array],
links:[Array],
lists:[Array],
text:'<Intro text>'
},
feedback:{
left:Array],
right:[Array]
},
submission:{
left:[Array],
right:[Array],
filesLink:'<Link to subitted file>'
}
}
}cannotConnect: { context: 'activity', status: 'cannotConnect', code: 1} // Network unreachable
noContent: { context: 'activity', status: 'noContent', code: 2} // No content in this section
unknownError: { context: 'activity', status: 'Error', error: '<Error>', code: 3 } // An unexpected error occuredunknownContext: { context: 'unknown', status: 'unknownContext', url: '<URL>', code: 1 }