What is content?
→ anything presented and hidden from end user on the site, like images, videos, file, backup, site features etc.
What are we trying to discover?
→ obvious things are clearly visible on the site, so we have to look for something which is hidden from user or wasnt meant for user to see.
Eg: portals for employees, hidden pages, old version of site, config files, backups etc.
Q. What is the Content Discovery method that begins with M?
A → manually
Q. What is the Content Discovery method that begins with A?
A → automated
Q. What is the Content Discovery method that begins with O?
A → osint
Before running scripts to find info, we can manually search a few things, makes our work faster:
/robots.txt — tells search engines which pages shouldnt be shown by the SE, also can ban certain SE from crawling the website altogether.
Q. What is the directory in the robots.txt that isn’t allowed to be viewed by web crawlers?
A → /staff-portal
favicon — you see the small icon beside tab name? thats a favicon. if the site is built using web frameworks, the default ones get leftover and they can hint us version of the framework. further info can be found about it on the web.
Eg: you download the favicon, get its md5 hash, go to this site, search for the hash and there you have the framework version.
Q. What framework did the favicon belong to?
A → cgiirc
sitemap.xml — gives list of every file of website owner wants listed on an SE. may contain old pages which arent used anymore but still present.
Q. What is the path of the secret area that can be found in the sitemap.xml file?
A → /s3cr3t-area
HTTP Headers — in response from server, one can find info like webserver s/w, language used, versions etc.
Q. What is the flag value from the X-FLAG header?
A → THM{HEADER_FLAG}
framework stack — finding the frameworks website and then getting more info from there, it could give us with more links, potential vulnerability clues etc.
Get eXpl017’s stories in your inbox
Join Medium for free to get updates from this writer.
Q. What is the flag from the framework’s administration portal?
A → THM{CHANGE_DEFAULT_CREDENTIALS}
Now we can go ahead and use OSINT to get more info by using:
google dorking— uses googles advanced search features to find openly available info on the web, which could be hard to reach/search without this help. Eg. of some filters:
Q. What Google dork operator can be used to only show results from a particular site?
A → site:
wappalyzer — a tool which finds the tech stack of a website, like frameworks, Content Management System (CMS), payment processors, and even versions.
Q. What online tool can be used to identify what technologies a website is running?
A → wappalyzer
web archive — a historical archive of websites scraped, just enter domain and it will give the times when those sites were scraped, could be none too.
Q. What is the website address for the Wayback Machine?
A → https://archive.org/web
github — finding repositories related to the target can be helpful, may not be much useful in real life but humans make mistakes often and we need to take advantage of it.
Q. What is Git?
A → version control system
s3 buckets — storage service provided by AWS, access permissions could be set wrongly and we can use that. links are of the format http(s)://{name}.s3.amazonaws.com, where owner sets the name.
can automate the process of searching such links from the website.
One common automation method is by using the company name followed by common terms such as {name}-assets, {name}-www, {name}-public, {name}-private, etc.
Q. What URL format do Amazon S3 buckets end in?
A → .s3.amazonaws.com
One of the best ways of discovering content is using automation.
Automated Discovery is using tools to discover content rather than doing it manually, can include hundreds of thousands of requests to server and get as much info as possible.
As running dirb gave me all the answers, i didnt run Gobuster.
Q. What is the name of the directory beginning “/mo….” that was discovered?
A → /mothly
Q. What is the name of the log file that was discovered?
A → /development.log
Thats it for today!
THANKYOU!!!
