implemented the flask api module and fixed some errors

Author: ihgalis

README.md

@@ -20,8 +20,6 @@ The results will be saved within a MongoDB collection. Another script will be tr
 
 The second step results in new collections where each keyword gets one new collection and all found pastebin entries will be copied there.
 
-The third step might be the accumulator. It identifies specific words (similiar to step 2) but also specials like eMail addresses, Bitcoin addresses, URLs, IP Adresses etc. 
-
 ### 1. pastebin_scrape.py
 
 For this one to work good you need an API key. I bought a lifetime access to the pastebin API a while ago for 29,99 USD. It doesn't make you poor.
@@ -32,7 +30,7 @@ You will also need to update your Scraping IP, in order to make it work: [Change
                                   -db 1 \                               # save to DB (without this, nothing will be saved)
                                   -api <YOUR_PASTE_BIN_API_KEY> \
                                   -mongodbhost <mongo_db_hostname> \    # default: localhost
-                                  -mongodbport <mongo_db_port> \        # default: 27017
+                                  -mongodbport <mongo_db_port>          # default: 27017
 
 ### 2. pastebin_analyze.py
 
@@ -42,19 +40,20 @@ When you have finished, start the analze module:
 
         python pastebin_analyze.py -f <path_to_keyword_file> \
                                    -mongodbhost <mongo_db_hostname> \   # default: localhost
-                                   -mongodbport <mongo_db_port<         # default: 27017
+                                   -mongodbport <mongo_db_port>         # default: 27017
                                    
 Finally it will create collections for all of the keywords it found and copy the pastebin into that collection. There might also be empty collections. Sometimes you
-just can't find anything you are searching for :-()       
-### 3. pastebin_accumulate.py
+just can't find anything you are searching for.
 
 ### Access Data via Flask API
 
 Finally you can either write yourself a clean data retriever or you can use this Flask API implementation here:
 
 ```
-# start it in debug and verbose mode first!
-python pastebin_api.py -d -v
+# start it in debug mode first!
+python pastebin_api.py -d \
+                       -mongodbhost <mongo_db_hostname> \
+                       -mongodbport <mongo_db_port> 
 ```
 
 Well there is only one API method. Grab yourself a browser or use curl:

keywords_example.txt

@@ -0,0 +1,7 @@
+ip
+malware
+glock
+android
+ios
+lenovo
+

pastebin_analyze.py

@@ -13,6 +13,13 @@
 
 
 def main():
+    """
+    starts the entire process of analyzation by creating new collections and appending new documents into existing
+    collections based on keywords.
+
+    :return:
+    """
+
     client = MongoClient(str(args['mongodbhost']), int(args['mongodbport']))
     db = client.scrape
     logger.info("MongoDB Connection created")

pastebin_api.py

@@ -0,0 +1,88 @@
+from flask import Flask, jsonify, make_response
+from pymongo import MongoClient
+from bson import json_util
+
+import argparse
+import json
+
+app = Flask(__name__)
+
+api_version = "1.0"
+
+
+@app.errorhandler(404)
+def not_found(error):
+    """
+    some standard error handling for unknown pages.
+
+    :param error:
+    :return:
+    """
+    return make_response(jsonify({'error': 'Notfound'}), 404)
+
+
+@app.route('/')
+def get_index():
+    """
+    standard output when nothing is set
+    :return:
+    """
+
+    basic_info = [
+        {
+            'api': '1.0',
+            'name': 'PastebinPython Flask Accessing API',
+            'author': 'Andre Fritsche / ihgalis'
+        }
+    ]
+
+    return jsonify({'basic_info': basic_info})
+
+
+@app.route('/api/getpastebins/<string:keyword>', methods=['GET'])
+def get_pastebins(keyword):
+    """
+    method gets all documents related to the specified keyword. It accesses the corresponding collections so you will
+    always get only the documents that have been identified by the pastebin_analyze.py script.
+
+    :param keyword: string
+    :return: JSON based dictionary
+    """
+
+    client = MongoClient(str(args['mongodbhost']), int(args['mongodbport']))
+    db = client.scrape
+
+    tlist = list()
+
+    dbcursor = db[keyword].find({})
+    for document in dbcursor:
+        sanitized = json.loads(json_util.dumps(document))
+        tlist.append(sanitized)
+
+    return jsonify(tlist)
+
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser(description="PastebinPython Flask Accessing API")
+
+    parser.add_argument('-mongodbhost',
+                        help="A string with the URL to your MongoDB Server.",
+                        default="localhost",
+                        required=True)
+
+    parser.add_argument('-mongodbport',
+                        help="THe port to which your MongoDB listens.",
+                        default=27017,
+                        required=True)
+
+    parser.add_argument('-d',
+                        action="store_true",
+                        help="Debug in Flask active or not.",
+                        default=0)
+
+    args = vars(parser.parse_args())
+
+    if args['d']:
+        app.run(debug=True)
+    else:
+        app.run(debug=False)

requirements.txt

@@ -0,0 +1,10 @@
+pymongo
+argparse
+logging
+re
+time
+sys
+pprint
+flask
+bson
+requests