Researchers at Carnegie Mellon University have developed a means for predicting if a currently uncompromised website will become malicious before it happens. According to their results, nearly 3 million web pages are vulnerable to possible exploitation within the next year. Kyle Soska and Nicolas Christin used the Internet Archive, which periodically stores snapshots of large parts of the Internet, to comb through recent history and look for common traits of websites that become compromised by Internet attackers. According to a paper presented at the recent USENIX Security Symposium, the authors of the study “… manage[d] to achieve good detection accuracy over a one-year horizon; that is, we generally manage to correctly predict that currently benign websites will become compromised within a year.”
The authors employed an intelligent algorithm, using samples of malicious sites from blacklists such as PhishTank to train their system to recognize a compromised site. They then used the Internet Archive’s Wayback machine, which searches the state of the Internet at previous points in recent history, to look for common characteristics of these sites before they were compromised. The assessment ignored user-supplied content and focused on factors such as unpatched web services and site structure, as well as anomalies in web traffic. The system learned to identify vulnerable sites on the verge of becoming compromised three to 12 months in advance.
In theory, this method could help organizations find flaws in their sites that could eventually lead to compromise. Search engines could also use a version of this technique to warn users about possible vulnerable pages that appear on the search list, which would provide a big incentive for webmasters to put their sites in order.