Skip to content

Commit d3915fb

Browse files
zeroby0zeroby0
committed
Added docs and tweaked some settings
1 parent fcc177f commit d3915fb

File tree

3 files changed

+344
-10
lines changed

3 files changed

+344
-10
lines changed

‎README.md‎

Lines changed: 328 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,331 @@
11
# LMS Scraper
22

3-
LMS scraper for IIITB. Reiteration of Java project done last year.
3+
Scraper for IIITB LMS Client. This project is under active development and is not ready for production, just yet.
44

5-
more info to be added later.
5+
Scrapes [IIITB Moodle LMS](https://lms.iiitb.ac.in/moodle/login/index.php) and returns data as `json`. For use as API layer of IIITB LMS Client (yet to be built).
6+
7+
Please note that the API is expected to change rapidly before a stable release.
8+
9+
## Get Started
10+
11+
This is a reiteration of [LMS client](https://github.com/zeroby0/IIITB-LMS-Client-OLD) built in Java as a part of course curriculum last year. This NodeJS scraper is expected to be more performant than the last one.
12+
13+
clone and install dependencies to get started.  
14+
* `git clone https://github.com/zeroby0/lms-scraper`
15+
* `yarn` or `npm install`
16+
17+
If `yarn` fails with `Failed to execute 'node-gyp clean' (Error: spawn node-gyp ENOENT)`, install with `npm` first and use `yarn` from then on.
18+
19+
## Documentation
20+
21+
All source is under `src` directory. Behaviour is expected to change as the project matures.
22+
23+
All settings can be changed by editing `config/default.yml`.
24+
25+
#### Index
26+
* [Auth](#auth)
27+
* [isLoggedin](#isloggedin)
28+
* [login](#login)
29+
* [keepLoggedIn](#keeploggedin)
30+
* [Course](#course)
31+
* [getForums](#getforums)
32+
* [getActivities](#getactivities)
33+
* [getMaterials](#getmaterials)
34+
* [Course > Activity](#course--activity)
35+
* [getContent](#getcontent)
36+
* [Course > Material](#course--material)
37+
* [getContent](#getcontent-1)
38+
* [User](#user)
39+
* [isLoggedin](#isloggedin-1)
40+
* [login](#login-1)
41+
* [keepLoggedIn](#keeploggedin-1)
42+
* [getCourses](#getcourses)
43+
* [getPrivateFiles](#getprivatefiles)
44+
* [getCookies](#getcookies)
45+
* [setCookies](#setcookies)
46+
47+
> Note: Most functions expect a user-agent instance to be passed. Login with `User.login` before passing it to other functions. Most functions need the user to be logged in to work. `User.login` authorises and sets cookies on the user-agent. All functions return a promise.
48+
49+
### Auth
50+
Authorisation functions. Not supposed to be used directly, login with `User.login` instead.
51+
Returns user status from lms in json. Either `'You are not logged in.'` or `'You are logged in as <userID> <Full name> (Log out)'`. Configuration options in `config/default.yaml` are in `eventFlags > auth > isLoggedin`.
52+
* #### isLoggedin
53+
``` js
54+
static async isLoggedin(user-agent)
55+
```
56+
Checks if a user is logged in.
57+
58+
* ###### success
59+
promise resolves with
60+
``` js
61+
{ status: 'success', code: 0, data: '<status>' }
62+
```
63+
* ###### cannotConnect
64+
Network failed. Check your internet connection.
65+
``` js
66+
{ status: 'cannotConnect', code: 1}
67+
```
68+
* ###### nullValue
69+
User status is null for an unknown reason.
70+
``` js
71+
{ status: 'nullValue', error: 2 }
72+
```
73+
* ###### unknownError
74+
An unexpected error occurred. Please let me know along with details in [issues](https://github.com/zeroby0/lms-scraper/issues).
75+
``` js
76+
{ status: 'Error', error: <Error>, code: 3 }
77+
```
78+
* #### login
79+
``` js
80+
static async login(username, password, agent)
81+
```
82+
83+
Authorises user-agent and sets cookies. Returns `json` with cookies if _successfully logged in_.
84+
* ###### success
85+
Successfully logged in.
86+
``` js
87+
{ status: 'authenticated', code: 0, cookies: <Cookies>, data: <User-Status> }
88+
```
89+
90+
* ###### invalidCreds
91+
Invalid credentials supplied.
92+
``` js
93+
{status: 'invalidCredentials', code: 1}
94+
```
95+
* ###### cannotConnect
96+
Network connectivity failed.
97+
``` js
98+
{ status: 'cannotConnect', code: 3 }
99+
```
100+
* ###### unknownError
101+
An unexpected error occurred. Please let me know along with details in [issues](https://github.com/zeroby0/lms-scraper/issues).
102+
``` js
103+
{ status: 'Error', error: null, code: 4 }
104+
```
105+
* #### keepLoggedIn
106+
``` js
107+
static async keepLoggedIn(username, password, agent, interval = 3000)
108+
```
109+
Keeps user logged in by trying to login every few seconds. Default value is 3000 ms. You should use a higher value.
110+
Calls `Auth.login` every `interval` seconds, doesn't return anything.
111+
112+
Uses recursion and `setTimeout` to keep looping. _Might_ cause a stack overflow if scraper is run continuously on a potato™ Micro controller.
113+
114+
### Course
115+
``` js
116+
class Course
117+
constructor(url)
118+
```
119+
`Course` class handles course level functions such as Forums, Course Materials and Activities. Getting their content is delegated to corresponding classes. For example, content _in Course Materials_ is handled by `Material` class in `src/Course/Material.js`. Constructor accepts course url. This is expected to change in future versions.
120+
* #### getForums
121+
``` js
122+
async getForums(user-agent)
123+
```
124+
Fetches list of Forums and links to them.
125+
126+
* ###### success
127+
Succefully obtained list of forum names and links.
128+
``` js
129+
{ status: 'success', code: 0, link: [<forum-links>], name: [<Forum-name>] }
130+
```
131+
The response is expected to change to be more consistent with others.
132+
* ###### noForum
133+
Course has no forums.
134+
``` js
135+
{ status: 'noForum', code: 1 }
136+
```
137+
* ###### cannotConnect
138+
Network connectivity failed.
139+
``` js
140+
{ status: 'cannotConnect', code: 2 }
141+
```
142+
* ###### unknownError
143+
An unexpected error occurred. Please let me know along with details in [issues](https://github.com/zeroby0/lms-scraper/issues).
144+
``` js
145+
{ status: 'Error', error: <Error>, code: 3 }
146+
```
147+
* #### getActivities
148+
``` js
149+
async getActivities(user-agent)
150+
```
151+
Fetches list of activities in course Activity section.
152+
* ###### success
153+
Successfully fetched list of activities from course.
154+
``` js
155+
{ status: 'success', code: 0, summary: <activity-section-summary>, \
156+
link: [<activity-links>], name: [<activity-name>], \
157+
desc: [<activity-description>] }
158+
```
159+
* ###### noActivity
160+
No activity section in course.
161+
``` js
162+
{ status: 'noActivity', code: 1 }
163+
```
164+
* ###### cannotConnect
165+
Network connectivity failed.
166+
``` js
167+
{ status: 'cannotConnect', code: 2 }
168+
```
169+
* ###### unknownError
170+
An unexpected error occurred. Please let me know along with details in [issues](https://github.com/zeroby0/lms-scraper/issues).
171+
``` js
172+
{ status: 'Error', code: 3, error: <Error> }
173+
```
174+
* #### getMaterials
175+
``` js
176+
async getMaterials(user-agent)
177+
```
178+
get course material name and links. This is usually named `slides folder` and `notes folder`.
179+
180+
* ###### success
181+
successfully fetched list of materials from course.
182+
``` js
183+
{ status: 'success', code: 0, summary: <material-section-summary>, link: [<material-links>], name: [<material-name>], desc: [<material-description>] }
184+
```
185+
* ###### noMaterial
186+
No course material.
187+
``` js
188+
{ status: 'noMaterial', code: 1 }
189+
```
190+
* ###### cannotConnect
191+
Network connectivity failed.
192+
``` js
193+
{ status: 'cannotConnect', code: 2 }
194+
```
195+
* ###### unknownError
196+
An unexpected error occurred. Please let me know along with details in [issues](https://github.com/zeroby0/lms-scraper/issues).
197+
``` js
198+
{ status: 'Error', code: 3, error: <Error> }
199+
```
200+
201+
### Course > Activity
202+
``` js
203+
class Activity
204+
constructor(url)
205+
```
206+
class `Activity` is in `src/Course/Activity.js`. Handles `Course Activity` level functions like fetching contents of an activity. Activity is usually an assignment. Submission details and feedback given can be obtained by using this class. Constructor expects an url to course activity. urls can be obtained from `Course.getActivities`.
207+
208+
* #### getContent
209+
``` js
210+
async getContent(user-agent)
211+
```
212+
Get details of activity. This includes submission details and feedback. `submissionLeft` and `submissionRight` are arrays with content on left and rigth side of submission status in LMS. `feedbackLeft` and `feedbackRight` similary correspond to feedback section.
213+
* ###### success
214+
``` js
215+
{ status: 'success', code: 0, title: <Activity-title>, intro: <Activity-intro>, \
216+
submissionLeft: [<left column of submission>], submissionRight: [<right column of submission>], \
217+
feedbackLeft: [<left column of feedback>], feedbackRight: [<right column of feedback>], \
218+
filesLink: <link to file submitted, if any>}
219+
```
220+
Note that values to keys are empty if no such content exists.
221+
* ###### noContent
222+
``` js
223+
{ status: 'noContent', code: 1 }
224+
```
225+
no content exists in activity. This shouldn't usually happen.
226+
* ###### cannotConnect
227+
Network connectivity failed.
228+
``` js
229+
{ status: 'cannotConnect', code: 2 }
230+
```
231+
* ###### unknownError
232+
An unexpected error occurred. Please let me know along with details in [issues](https://github.com/zeroby0/lms-scraper/issues).
233+
``` js
234+
{ status: 'Error', code: 3, error: <Error> }
235+
```
236+
### Course > Material
237+
``` js
238+
class Material
239+
constructor(url)
240+
```
241+
Course Material. This is the content in what is usually `slides folder` or `notes folder`. Constructor expects url to course material, this can be obtained from `Course.getMaterial`. `Material` class is at `src/Course/Material.js`.
242+
* #### getContent
243+
``` js
244+
async getContent(user-agent)
245+
```
246+
Fetches name and links to contents in course material.
247+
248+
* ###### success
249+
successfully fetched contents.
250+
``` js
251+
{ status: 'success', code: 0, link: [<material-links>], name: [<material-name>] }
252+
```
253+
* ###### noContent
254+
No content in course Material
255+
``` js
256+
{ status: 'noContent', code: 1 }
257+
```
258+
* ###### cannotConnect
259+
Network failed.
260+
``` js
261+
{ status: 'cannotConnect', code: 2 }
262+
```
263+
* ###### unknownError
264+
An unexpected error occurred. Please let me know along with details in [issues](https://github.com/zeroby0/lms-scraper/issues).
265+
``` js
266+
{ status: 'Error', code: 3, error: <Error> }
267+
```
268+
269+
### User
270+
``` js
271+
class User extends Person
272+
constructor(username, password)
273+
```
274+
User class that handles User level functions like authorisation and gets courses and private files list. Apart from that, `User` extends `Person` and can set and get name, etc . Checkout `Person` in `src/User/Person.js`.
275+
* #### isLoggedin
276+
``` js
277+
static async isLoggedin(user-agent)`
278+
```
279+
Wrapper around `Auth.isLoggedin`. Checkout `Auth.isLoggedin` for more details.
280+
* #### login
281+
```js
282+
async login(user-agent)
283+
```
284+
Authorises user-agent with credentials of user. User credentials passed during _initialisation_ are used. Wraps around `Auth.login`, checkout that section to see what Promises resolve or reject to.
285+
* #### keepLoggedIn
286+
```js
287+
async keepLoggedIn(user-agent)
288+
```
289+
wraps around `Auth.keepLoggedIn`. Uses `timeout` set in `config` file. You can set that at `params > onlineTimeout` in `config/default.yaml` file.
290+
* #### getCourses
291+
```js
292+
static async getCourses(user-agent)
293+
```
294+
Note that this is a `static` function. Assumes that user-agent is already authorised by passing to `User.login()`.
295+
* ###### success
296+
Successfully fetched list of courses. Note that this function assumes user credentials are correct. List of courses and links to them are returned in json.
297+
``` js
298+
{ status: 'success', code: 0, data: [[<course-link>], [<course-name>] ] }
299+
```
300+
* ###### error
301+
An unexpected error occurred. Please let me know along with details in [issues](https://github.com/zeroby0/lms-scraper/issues).
302+
``` js
303+
{ status: 'error', code: 1, error: <Error> }
304+
```
305+
* #### getPrivateFiles
306+
``` js
307+
static async getPrivateFiles(user-agent)
308+
```
309+
Get user private files hosted in LMS. Note that this is a static function and assumes user-agent is authorised with correct credentials.
310+
Results are identical to `User.getCourses`.
311+
* ###### success
312+
Successfully fetched private files.
313+
``` js
314+
{ status: 'success', code: 0, data: [[<files-link>], [<file-names>]] }
315+
```
316+
* ###### error
317+
An unexpected error occurred. Please let me know along with details in [issues](https://github.com/zeroby0/lms-scraper/issues).
318+
``` js
319+
{ status: 'error', code: 1, error: <Error> }
320+
```
321+
* #### getCookies
322+
Get cookies from user-agent. Expects no arguments.
323+
``` js
324+
getCookies()
325+
```
326+
Retrieves cookies stored in `User` object.
327+
* #### setCookies
328+
Fetches cookies from user agent and stores in User object.
329+
``` js
330+
setCookies(user-agent)
331+
```

‎config/default.yaml‎

Lines changed: 9 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,10 @@ eventFlags:
1919
getCourses:
2020
success: { status: 'success', code: 0 }
2121
unknownError: { status: 'error', code: 1 }
22+
23+
getPrivateFiles:
24+
success: { status: 'success', code: 0 }
25+
unknownError: { status: 'error', code: 1 }
2226
auth:
2327
login:
2428
success: { status: 'authenticated', code: 0 }
@@ -28,7 +32,7 @@ eventFlags:
2832
unknownError: { status: 'Error', error: null, code: 4 }
2933

3034
isLoggedin:
31-
success: { status: 'success', code: 0}
35+
success: { status: 'success', code: 0}
3236
cannotConnect: { status: 'cannotConnect', code: 1 }
3337
nullValue: { status: 'nullValue', error: 2 }
3438
unknownError: { status: 'Error', code: 3 }
@@ -38,7 +42,7 @@ eventFlags:
3842
noContent: { status: 'noContent', code: 1 }
3943
cannotConnect: { status: 'cannotConnect', code: 2 }
4044
unknownError: { status: 'Error', code: 3 }
41-
45+
4246
material: # material class
4347
getContent:
4448
success: { status: 'success', code: 0 }
@@ -82,7 +86,7 @@ selector:
8286
selector: "#page-content"
8387
link: ".corner-box .content .course_list h2 @href"
8488
name: ".corner-box .content .course_list h2"
85-
89+
8690
activity:
8791
activity: "#region-main"
8892
title: "#region-main > div > div > h2"
@@ -105,17 +109,16 @@ selector:
105109
link: ".modtype_assign > div > div > div:nth-child(2) > div.activityinstance > a @href"
106110
name: ".modtype_assign > div > div > div:nth-child(2) > div.activityinstance > a"
107111
desc: ".modtype_assign > div > div > div:nth-child(2) > div.contentafterlink > div > div > p"
108-
112+
109113
forum:
110114
forum: "#section-0"
111115
link: ".modtype_forum > div > div > div:nth-child(2) > div > a @href"
112116
name: ".modtype_forum > div > div > div:nth-child(2) > div > a"
113-
117+
114118

115119
material:
116120
material: "#section-1"
117121
summary: ".summary"
118122
link: ".modtype_folder a @href"
119123
name: ".modtype_folder a"
120124
desc: ".contentafterlink p"
121-

0 commit comments

Comments
 (0)