Skip to content

Commit a24fcf9

Browse files
zeroby0zeroby0
committed
Alpha 1
1 parent 9e8de80 commit a24fcf9

File tree

10 files changed

+1253
-336
lines changed

10 files changed

+1253
-336
lines changed

‎README.md‎

Lines changed: 197 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -18,10 +18,204 @@ If `yarn` fails with `Failed to execute 'node-gyp clean' (Error: spawn node-gyp
1818

1919
> Note: Make sure you define your username and password in your environment. Either run `export LMS_USERNAME="Your Username"` and `export LMS_PASSWORD="Password"` in every terminal session before you run scraper or add them to your `bash_profile`.
2020
21+
<hr>
22+
2123
## Documentation
2224

23-
All source is under `src` directory. Behaviour is expected to change as the project matures.
25+
Note: All settings can be changed by editing `config/default.yml`.
26+
27+
### Supported
28+
* Login
29+
* Courses and Private Files from home page
30+
* List of Forums, Activities (Assignments), and Course Materials and link to, descriptions for them
31+
* Activity (Assignment) details, submission and feedback recieved.
32+
* Files uploaded in Course material section
33+
34+
### Limitations
35+
* Forum content not supported right now.
36+
37+
### Usage
38+
1. `Require` `index.js` from where you'd like to use the scraper. like so, `const API = require('./index')`.
39+
2. API.getContent('your-url')
40+
3. Get back JSON
41+
4. ??
42+
5. Profit!
43+
44+
API automatically resolves the URL and returns a `Promise` which resolves into JSON.
45+
Example:
46+
``` js
47+
api.getContent(url)
48+
.then((result) => {
49+
console.log(result)
50+
})
51+
```
52+
JSON responses are defined in `config/default.yml`. Here is the reference:
53+
### General response
54+
``` js
55+
{
56+
context: '<context>',
57+
status: '<status>',
58+
code: '<error-code>',
59+
data: {
60+
data_single: '<data>',
61+
data_multiple: ['<data>']
62+
},
63+
error: '<Error if any>'
64+
}
65+
```
66+
67+
`context`: indicates the content of the response.
68+
`status`: indicates the status of the API call. `status` is `success` if everything went well.
69+
`code`: The error code. It is preferable to rely on `status` than `code`.
70+
`data`: The data returned by the call.
71+
`url`: The url passed in call. Though URL is returned, it's not included in all responses. Errors for example. Do not rely on URL for identifying the response, maintain a record of call and response.
72+
73+
### Login:
74+
``` js
75+
{ // Successful
76+
context:'login',
77+
status:'authenticated',
78+
code:0,
79+
data:{
80+
status:'You are logged in as <Name and rollnumber> (Log out)',
81+
cookies:{
82+
MoodleSession:'*&#%*#&%*&%*#~&$%*&'
83+
}
84+
}
85+
}
86+
```
87+
``` js
88+
inavlidCredentials: { context: 'login', status: 'invalidCredentials', code: 1 } // Invalid username or password
89+
cannotConnect: { context: 'login', status: 'cannotConnect', code: 3 } // Network Unavailable
90+
unknownError: { context: 'login', status: 'Error', error: '<error>', code: 4 } // an unexpected error occured
91+
```
92+
### Home
93+
``` js
94+
{
95+
courseList:{
96+
context:'courseList',
97+
status:'success',
98+
data:{
99+
links:[Array],
100+
names:[Array]
101+
},
102+
code:0
103+
},
104+
privateFiles:{
105+
context:'privateFiles',
106+
status:'success',
107+
data:{
108+
links:[Array],
109+
names:[Array]
110+
},
111+
code:0
112+
}
113+
}
114+
```
115+
``` js
116+
cannotConnect: { context: 'home', status: 'cannotConnect', code: 1 } // Network unreachable
117+
unknownError: { context: 'home', status: 'error', error: '<Error>', code: 2 } // An unexpected error occured
118+
```
119+
### Course
120+
``` js
121+
{
122+
context:'course',
123+
status:'success',
124+
url: '<course-url>',
125+
data:{
126+
forums:{
127+
context:'forumList',
128+
status:'success',
129+
data:[
130+
Object
131+
],
132+
code:0
133+
},
134+
activities:{
135+
context:'activityList',
136+
status:'success',
137+
data:[
138+
Object
139+
],
140+
code:0
141+
},
142+
materials:{
143+
context:'materialList',
144+
status:'success',
145+
data:[
146+
Object
147+
],
148+
code:0
149+
}
150+
},
151+
code:0
152+
}
153+
```
154+
``` js
155+
cannotConnect: { context: 'course', status: 'cannotConnect', code: 1 } // No network
156+
157+
noForum: { context: 'forumList', status: 'noForum', code: 1 } // No forum for this course
158+
noActivity: { context: 'activityList', status: 'noActivity', code: 1 } // No activities for this course
159+
noMaterial: { context: 'materialList', status: 'noMaterial', code: 1 } // No materials for this course
160+
161+
unknownError: { context: 'course', status: 'Error', error: '<Error>', code: 2 } // An unexpected error occured
162+
```
163+
### Materials
164+
``` js
165+
{
166+
context:'material',
167+
url:'<URL>',
168+
status:'success',
169+
code:0,
170+
data:{
171+
links:[
172+
'<List of links to course materials>'
173+
],
174+
names:[
175+
'<List of names in same order>'
176+
]
177+
}
178+
}
179+
```
180+
``` js
181+
cannotConnect: { context: 'material', status: 'cannotConnect', code: 1 } // cannot connect to the internet
182+
noContent: { context: 'material', status: 'noContent', code: 2 } // No content in this section
183+
unknownError: { context: 'material', status: 'Error', error: '<Error>', code: 3 } // An unexpected error occured
184+
```
24185

25-
All settings can be changed by editing `config/default.yml`.
186+
### Activity
187+
``` js
188+
{
189+
context:'activity',
190+
url:'<URL>',
191+
status:'success',
192+
code:0,
193+
data:{
194+
intro:{
195+
linkTitles:[Array],
196+
links:[Array],
197+
lists:[Array],
198+
text:'<Intro text>'
199+
},
200+
feedback:{
201+
left:Array],
202+
right:[Array]
203+
},
204+
submission:{
205+
left:[Array],
206+
right:[Array],
207+
filesLink:'<Link to subitted file>'
208+
}
209+
}
210+
}
211+
```
212+
``` js
213+
cannotConnect: { context: 'activity', status: 'cannotConnect', code: 1} // Network unreachable
214+
noContent: { context: 'activity', status: 'noContent', code: 2} // No content in this section
215+
unknownError: { context: 'activity', status: 'Error', error: '<Error>', code: 3 } // An unexpected error occured
216+
```
26217

27-
One function call for anything. Call API.getContent() and it takes care of the rest.
218+
### Malformed URL
219+
``` js
220+
unknownContext: { context: 'unknown', status: 'unknownContext', url: '<URL>', code: 1 }
221+
```

‎config/default.yaml‎

Lines changed: 14 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,8 @@ params:
2424
onlineTimeout: 3000
2525

2626
response:
27+
global:
28+
unknownContext: { context: 'unknown', status: 'unknownContext', url: null, code: 1 }
2729
user:
2830
login:
2931
success: { context: 'login', status: 'authenticated', code: 0 }
@@ -32,14 +34,25 @@ response:
3234
cannotConnect: { context: 'login', status: 'cannotConnect', code: 3 }
3335
unknownError: { context: 'login', status: 'Error', error: null, code: 4 }
3436
getCourseContent:
35-
success: { context: 'course', status: 'success', data: { forums: null, activities: null, materials: null }, code: 0 }
37+
success: { context: 'course', status: 'success', url: null, data: { forums: null, activities: null, materials: null }, code: 0 }
3638
cannotConnect: { context: 'course', status: 'cannotConnect', code: 1, error: null }
3739
forumSuccess: { context: 'forumList', status: 'success', data: { names: null, links: null }, code: 0 }
3840
noForum: { context: 'forumList', status: 'noForum', code: 1 }
3941
activitySuccess: { context: 'activityList', status: 'success', data: { summary: null, links: null, names: null, descriptions: null }, code: 0 }
4042
noActivity: { context: 'activityList', status: 'noActivity', code: 1 }
4143
materialSuccess: { context: 'materialList', status: 'success', data: { summary: null, links: null, names: null, descriptions: null }, code: 0 }
4244
noMaterial: { context: 'materialList', status: 'noMaterial', code: 1 }
45+
unknownError: { context: 'course', status: 'Error', error: null, code: 2 }
46+
getMaterialContent:
47+
success: { context: 'material', url: null, status: 'success', code: 0, data: { links: null, names: null } }
48+
cannotConnect: { context: 'material', status: 'cannotConnect', code: 1, error: null }
49+
noContent: { context: 'material', status: 'noContent', code: 2, error: null }
50+
unknownError: { context: 'material', status: 'Error', error: null, code: 3 }
51+
getActivityContent:
52+
success: { context: 'activity', url: null, status: 'success', code: 0, data: { intro: { linkTitles: null, links: null, lists: null }, feedback: { left: null, right: null }, submission: { left: null, right: null, filesLink: null } } }
53+
cannotConnect: { context: 'activity', status: 'cannotConnect', code: 1, error: null }
54+
noContent: { context: 'activity', status: 'noContent', code: 2, error: null }
55+
unknownError: { context: 'activity', status: 'Error', error: null, code: 3 }
4356
getHomeContent:
4457
success: { context: 'home', status: 'success', data: { courseList: null, privateFiles: null }, code: 0 }
4558
cannotConnect: { context: 'home', status: 'cannotConnect', error: null, code: 1 }

‎index.js‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,12 +33,15 @@ class API extends Scraper {
3333
break
3434
case url.includes(material):
3535
debug('getContent: material - %s', url)
36+
promise = this.getMaterialContent(url)
3637
break
3738
case url.includes(activity):
3839
debug('getContent: activity - %s', url)
40+
promise = this.getActivityContent(url)
3941
break
4042
default:
4143
debug('getContent: can\'t resolve - %s', url)
44+
promise = this.unknownContext(url)
4245
}
4346

4447
return promise

0 commit comments

Comments
 (0)