mining GitHub Advanced Security data using MongoDB
This was inspired by the work published by Dakota Riley at https://blog.aquia.us/blog/2024-02-27-gh-advisory-db/
I was thinking — since the NVD CVE data is provided in the form of JSON files, why not dump them in a document database and plot them in MongoDB dashboard for fun?
I used the below to down and dump the data to MongoDB as a collection,
Get Shi’s stories in your inbox
Join Medium for free to get updates from this writer.
git clone https://github.com/github/advisory-database
uri = "mongodb+srv://shichao:password@clusterm10.tf52c.mongodb.net/?retryWrites=true&w=majority&appName=ClusterM10"
client = MongoClient(uri, server_api=ServerApi('1'))
db = client['CVE_data']
collection = db['github_advisory_database']
try:
client.admin.command('ping')
print("Pinged your deployment. You successfully connected to MongoDB!")
except Exception as e:
print(e)
repo_path = './advisory-database/advisories/github-reviewed/**/*.json'
json_files = glob.glob(repo_path, recursive=True)
for file in json_files:
with open(file, 'r') as f:
json_data = json.load(f)
try:
result = collection.insert_one(json_data)
print(json_data["id"] + ": " + json_data["summary"])
print(f"Successfully inserted {result.inserted_id} documents into MongoDB.")
except Exception as e:
print(f"An error occurred while inserting documents: {e}")then the MongoDB dashboard easily plotted the top CWEs and top package managers without writing any code:
obviously CWE-79 XSS is still the top, and far more dangerous than everything else.
Note that each GHAS entry might have multiple affected package managers, so the total number of entries sum by the below is more than the total number of GHAS entries.
I am slightly surprised that PyPI is an even more meaty target than maven, which I believe is the most popular package manager. I am also surprised that npm is only ranked №4, which is lower than I initially assumed.
