Using Shodan Web Crawlers to find vulnerable servers or websites on Internet

by vault . 14 June 2018

Using Shodan Web Crawlers to find vulnerable servers or websites on Internet

In this introductory tutorial of Shodan, the hacker's search engine, i'll breifly explain its working and will try to make you familiar with various shodan filters. We will see how shodan works, how it grab banners, search maps, how often the database changes are commited, how queries are being defined, filters, and eventually how to find the vulnerable servers, i.e. servers running the vulnerable services.

Banner

What is a Banner? A banner is collection of text data that give details of a service running on a host like Content Type, Cookies, Web Server and Content-length. Banners are always different for different kind of services and keep on changing time to time. Here's an example of a banner returned in response of a request:

HTTP/1.1 200 OK
Server: apache2
Date: Sun, 13 May 2018 02:12:34 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 9879
Connection: keep-alive

Database

Shodan databases are updated 24 hours a day and 7 days a week. So, it means anytime you search, you are retreiving the latest results on the Internet.

How it Works?

Whenever shodan gets a query from a user, it generates randomized IPv4 addresses and then on various randomly choosen ports, retreive banners. It then analyze the banners, process the logic and give back the results. Lets say we search for facebook. It will look at the banners for matched words. If a word gets found, then this banner will be included in the results.

facebook...

Shodan uses OR operator by default for filtering queries. If you want to search for a word that include spaces or want combine two different filters, you can use +. It will work as AND operator. Lets query for iis 8.0.

iis+8.0
iis+8.0

Meta Data

In addition to Banners, shodan crawlers also look for the meta data of an IP address and show results from the past month. Meta Data are the information collected from an IP address like its Physical location, Geo Coordinates and ISP etc.

Filters

Introducing shodan filters. But before you start using filter, you would need an account. Go to shodan.io and get youself a new account. Filters are the key to find something potential on shodan. Shodan provides various filters to narrow down the result queries to further and further level. Here's the syntax of providing filters:

filtername1:value filtername2:value ...

And here's the common used filters:

  • country:string country code like US, PK, FR, IN etc
  • city:string, having the name of city
  • ip:string containing the IP address
  • os:string, operating system
  • product:string, name of software
  • port:integer, port to search on
  • geo:comma-seperated string, longitude and lattitude
  • version:string version of a product

Note that, there is no space on either side of ':'. Multiple values for a single filter can also be provided by seperating them by comma ',' except for filters that already use comma-seperated strings.

country and city:

country:"FR" city:"paris" nginx

The above query will search for the word "nginx" in banners retreived from the IP addresses which are located in Paris, France.

IP filter:

ip:'119.159.229.144'

This will look for the banners from the IP 119.159.229.144.

os, product and port:

product:MySQL os:windows

Now, this will search for MySQL databases running the Windows Operating System and currently it gave 66,377 results from the Internet. It means that 66,377 hosts on the internet are using MySQL. More importantly, this is how you can get the statistics of a service or a widely used software.

From this, now you be able to compare which database is more widely deployed? Postgresql or MySQL.

Now, let's come to geo filter. It accepts a string a two comma-seperated values which respectively are longitude and lattitude. I've picked up random geo coordinates from Jakarta, Indonesia. The downward query will look for IIS web Servers in Jakarta.

product:'microsoft-iis' geo:'-6.2267661, 106.820036'

HTTP filters

Besides, the general filters, shodan also provides some http filters. These filters are to fetch some of the important details from within the document like document title, technologies used. General used Shodan filters are:

  • http.component: value must be the name of technologies used like wordpress, JQuerey, Drupal, Django etc.
  • http.title: Title for the website
  • http.status: Response Status Code

Vulnerable Services

Until now, you must have got the picture of using Shodan. Well, if you do really understand the filters, then you would know that, it's all just the matter of swaping the filters. To dive more deep, i suggest exploring the shared examples on shodan.

Conclusion

We've seen how we could use shodan to find things on the Internet. Our concern is not the things but the vulnerable parts of them, like vulnerable versions of FTP, SMTP etc. Moreover, we could search for vulnerable Operating Systems like Windows XP, service pack 1 which are not yet patched for Netapi Vulnerability and could be easily exploit with a few couple commands. So, basically what we have to do is to look out for vulnerable implementation of a system.