Use of web crawling as a technique to determine online security of websites

Abstract

This study developed a system that utilized web crawling techniques to analyze website security. The objective was to detect misconfigurations such as invalid cryptographic certificates, broken authentication, and exposed private data. A prototyping methodology involving analysis, design, and implementation of a proof-of-concept testbed was used. The system was validated by measuring crawl time, detection accuracy, and the ability to notify site owners about vulnerabilities. The research contributes an automated approach to efficiently assess website vulnerabilities. Web crawling is a technique for automating interactions between a web client and a web server. This research proposed use of web crawling techniques to analyze websites and detect misconfiguration such as invalid cryptographic certificates, broken authentication, and exposed private data. A prototyping methodology involving analysis, design and implementation of a proof-of-concept testbed was used. The system would be validated using amount of time required to crawl and process a website, and accuracy in detecting deliberate misconfigurations. The research will also develop an automated system for efficiently notifying website owners about newly identified vulnerabilities.

Key Words: Access Control, Automated Scanning, Cyber Threats, Cybersecurity, Dark Web Monitoring, Intrusion Threat Intelligence, Vulnerability Assessment, Web Crawling.