Using Nmap to Screenshot Web Services Troubleshooting - Pentest Geek

Using Nmap to Screenshot Web Services Troubleshooting

Author: Brandon McCann Posted In Penetration Testing Tutorials On: 2012/07/11 Comments: 17
Using Nmap to Screenshot Web Services Troubleshooting

Recently a member from the Trustwave SpiderLabs team created an nmap NSE script that could be used to take a screenshot of webpages as it scanned the network. Working for a top 10 accounting firm, I conduct a lot of internal penetration tests for clients that operate on very large networks, and sometimes I’m required to audit entire counties.  Having the ability to view all the webpages on the internal network without being required to manually type in each addresses into the browser sounded amazing.  This was very exciting news now that there was a way to automate this process and have the ability to scale.  I dove in right away to get started by installing the script based on the instructions in the link listed below:

http://blog.spiderlabs.com/2012/06/using-nmap-to-screenshot-web-services.html

I highly suggest you look over the article above as I wrote this article in hopes that it would help assist anyone when having issues getting the http-screenshot NSE script to function properly with the latest version of nmap.

I encountered some adversity while following the tutorial since I wasn’t running BT5.  I was running Ubuntu 10.04.04 amd64 and had just installed the latest version of nmap (6.02) via svn and compiled from source.

The first problem with the instructions was the version of wkhtmltoimage-i386 didn’t work on my system, most likely due to my x64 architecture.  Be sure to test your version of wkhtmltoimage before continuing.  The command should display a help menu with options when ran without any arguments.

Not working:
[crayon toolbar=”false”]
root@lockout:~# file ./wkhtmltoimage-i386
./wkhtmltoimage-i386: ELF 32-bit LSB executable, Intel 80386, version 1 (GNU/Linux), statically linked, stripped
root@lockout:~# ./wkhtmltoimage-i386
root@lockout:~#

[/crayon]

Make sure you download the correct version that will work with your OS, and test it before continuing.  wkhtmtoimage downloads can be found here.

In my case I tweak the tutorial to this:

wget https://wkhtmltopdf.googlecode.com/files/wkhtmltoimage-0.11.0_rc1-static-amd64.tar.bz2
tar -jxvf wkhtmltoimage-0.11.0_rc1-static-amd64.tar.bz2
cp wkhtmltoimage-amd64 /usr/local/bin/

This was just the beginning of my problems as I attempted to get this NSE script to function properly.   I continued to follow the instructions like the tutorial had listed

git clone git://github.com/SpiderLabs/Nmap-Tools.git
cd Nmap-Tools/NSE/
cp http-screenshot.nse /usr/local/share/nmap/scripts/
nmap –script-updatedb

The problems started when I tried to update my nmap database that keeps track of the NSE scripts.  Here is what the output looked like on my initial run:

[crayon toolbar=”false”]

root@lockout:~/Nmap-Tools/NSE# nmap –script-updatedb
Starting Nmap 6.02 ( http://nmap.org ) at 2012-07-11 23:36 CDT
NSE: Updating rule database.
NSE: Failed to load /usr/local/bin/../share/nmap/scripts//http-screenshot.nse:
/usr/local/bin/../share/nmap/scripts//http-screenshot.nse:30: variable ‘shortport’ is not declared
stack traceback:
[C]: in function ‘error’
/usr/local/bin/../share/nmap/nselib/strict.lua:80: in function ‘__index’
/usr/local/bin/../share/nmap/scripts//http-screenshot.nse:30: in function </usr/local/bin/../share/nmap/scripts//http-screenshot.nse:1>
NSE: failed to initialize the script engine:
/usr/local/bin/../share/nmap/nse_main.lua:460: could not load script
stack traceback:
[C]: in function ‘error’
/usr/local/bin/../share/nmap/nse_main.lua:460: in function ‘new’
/usr/local/bin/../share/nmap/nse_main.lua:1113: in main chunk
[C]: in ?

QUITTING!
root@lockout:~/Nmap-Tools/NSE#

[/crayon]

The first problem that I noticed right away was “variable ‘shortport’ is not declared on live 5 above”.  I opened up the script to try and make some changes to the http-screenshot NSE script.

On line #28 you should see the line below:

[crayon toolbar=”false”]

require “shortport”

[/crayon]

replace this line with the following to declare the variable.

[crayon toolbar=”false”]

local shortport = require “shortport”

[/crayon]

Once you have saved your changes, lets try to update the nmap database.  If everything went successfully, you should have some output like below that shows the database updated.  If you still have issues related to variables, place the string ‘local’ in front of the variable causing issues, and that should fix your problem.

[crayon toolbar=”false”]

root@lockout:~/Nmap-Tools/NSE# nmap –script-updatedb
Starting Nmap 6.02 ( http://nmap.org ) at 2012-07-11 23:43 CDT
NSE: Updating rule database.
NSE: Script Database updated successfully.
Nmap done: 0 IP addresses (0 hosts up) scanned in 1.04 seconds
root@lockout:~/Nmap-Tools/NSE#
[/crayon]

Before attempting to run the script, we will need to update a few items to make the http-screenshot NSE script to run properly.  The first item we needed to modify was the line that called wkhtmltoimage-i386 to wkhtmltoimage-amd64 since we downloaded and installed a different version above.

On line #48 you should see the line below:

[crayon toolbar=”false”]

local cmd = “wkhtmltoimage-i386 -n ” .. prefix .. “://” .. host.ip .. “:” .. port.number .. ” ” .. filename .. ” 2> null   >null”

[/crayon]

replace the line above with the following to changes listed below. We first want to change the quality that wkhtmtoimage uses to create screenshots of the webpage.  This will save disk space on your hdd and speed up the downloading of screenshots as well.  In my testing, I noticed that a quality of 20 (default 94) was not very noticeable to the eye, but decreased the size of the screenshots significantly.

[crayon toolbar=”false”]

local cmd = “wkhtmltoimage-amd64 –quality 20 -n ” .. prefix .. “://” .. host.ip .. “:” .. port.number .. ” ” .. filename .. ” 2> null   >null”

[/crayon]

Make sure you update the nmap database everytime you make a change to the NSE script.

[crayon toolbar=”false”]
root@lockout:~/Nmap-Tools/NSE# nmap –script-updatedb
[/crayon]

At this point I thought I was smooth sailing now that nmap was scanning properly, and I wasn’t getting any errors.  Well, I was wrong.  Nmap would continually hang when scanning the network, and specifically when running the NSE portion of the namp scan.  It was getting hung on the http-screenshot NSE script specifically, and I needed to find out what line of code was causing the issue.

http-screenshot.nse hanging

[crayon toolbar=”false”]


Completed SYN Stealth Scan against 192.168.51.14 in 45.47s (10 hosts left)
Completed SYN Stealth Scan against 192.168.51.12 in 45.50s (9 hosts left)
Completed SYN Stealth Scan against 192.168.51.0 in 45.51s (8 hosts left)
Completed SYN Stealth Scan against 192.168.51.5 in 45.51s (7 hosts left)
Completed SYN Stealth Scan against 192.168.51.22 in 45.53s (6 hosts left)
Completed SYN Stealth Scan against 192.168.51.2 in 45.53s (5 hosts left)
Completed SYN Stealth Scan against 192.168.51.1 in 45.55s (4 hosts left)
Completed SYN Stealth Scan against 192.168.51.3 in 45.55s (3 hosts left)
Completed SYN Stealth Scan against 192.168.51.15 in 45.55s (2 hosts left)
Completed SYN Stealth Scan against 192.168.51.19 in 45.61s (1 host left)
Completed SYN Stealth Scan at 23:49, 47.00s elapsed (38000 total ports)
NSE: Script scanning 38 hosts.
Initiating NSE at 23:49
[/crayon]

I noticed that the script called the binary wkhtmltoimage-amd64, but never specified a timeout if the program never returned a result in a timely fashion.  I started looking for options in wkhtmtoimage to see if a timeout value could be specifed that would give up after set amount of time. That was when I stumbled up this article here that essential states the pre packaged binary that I downloaded did not have a timeout by default.

I knew linux had a ‘timeout’ package that I could download right from the Ubuntu repositories.  Since the NSE script was making a system call, I could install the ‘timeout’ package and use it directly in the NSE script.

apt-get install timeout -y

Now that I have the timeout utility installed on my OS, I was ready to modify the script for one last time.

[crayon toolbar=”false”]
local cmd = “wkhtmltoimage-amd64 –quality 20 -n ” .. prefix .. “://” .. host.ip .. “:” .. port.number .. ” ” .. filename .. ” 2> null   >null”
[/crayon]

[crayon toolbar=”false”]
local cmd = “timeout 20 wkhtmltoimage-amd64 –quality 20 -n ” .. prefix .. “://” .. host.ip .. “:” .. port.number .. ” ” .. filename .. ” 2> null   >null”
[/crayon]

Make sure that when you run the http-screenshot.nse script that you utilize the -sV flag in nmap to properly enumerate http vs https services.
[crayon toolbar=”false”]
root@lockout:~/Nmap-Tools/NSE# nmap -sV -p 80,443 –script=http-screenshot google.com
[/crayon]

If your nmap output continues to tell you “failed (verify wkhtmltoimage-i386 is in your path)” but the script is still output images to your current working directory, this is because there is a flaw in the programs logic.  The code below will always make result = “failed (verify wkhtmltoimage-i386 is in your path)” because ret is actually a boolean value and not a string or decimal.  This means that the ‘ret’ variable will never == 0 and therefore never give you the output you wish to see such as “Saved to screenshot-nmap-74.125.225.7:443.png”

[crayon toolbar=”false”]
— If the command was successful, print the saved message, otherwise print the fail message
local result = “failed (verify wkhtmltoimage-i386 is in your path)”

if ret == 0 then
result = “Saved to ” .. filename
end
[/crayon]

In order to make the code function how it was intented we will change the code to the following:

[crayon toolbar=”false”]
— If the command was successful, print the saved message, otherwise print the fail message
local result = “failed (verify wkhtmltoimage-i386 is in your path)”

if ret then
result = “Saved to ” .. filename
end
[/crayon]

I tested these changes and it helped my performance significantly.  Nmap was scanning entire class C networks in a timely fashion and I was no longer getting hung on the NSE scripts!

Hope this helps some of you out there who may have had issues getting the http-screenshot NSE script to run properly initially.

Share this article

Facebooktwittergoogle_plusredditpinterestlinkedinmailFacebooktwittergoogle_plusredditpinterestlinkedinmail
17 Comments
  • Wow. Normally don’t comment on blogs but thank you so much for that last part about setting the timeout – I’ve been searching for something like that for hours. FYI for anyone who reads this – very similar set up to run in Windows environment for that timeout just use “timeout /t 20” instead – and ensure you place that wkhtmltoimage thing in your Windows systems PATH – excellent blog post.

  • I am getting this issue. Any hints?

    billybear@ubuntu:~/Nmap-Tools/NSE$ sudo nmap -script-updatedb

    Starting Nmap 5.21 ( http://nmap.org ) at 2014-05-15 09:23 CDT
    NSE: Updating rule database.
    NSE: error while updating Script Database:
    [string “local nse = ……”]:17: error loading module ‘citrixxml’ from file ‘/usr/share/nmap/nselib/citrixxml.lua’:
    /usr/share/nmap/nselib/citrixxml.lua:87: unfinished string near ‘”
    z’
    stack traceback:
    [C]: in function ‘assert’
    [string “local nse = ……”]:17: in main chunk

    Nmap done: 0 IP addresses (0 hosts up) scanned in 0.03 seconds

    • zeknox

      This is an issue caused by the citrixml.lua script, not the http-screenshot script. Fix or remove the citrixxml.lua script first.

  • Any one have a working script with cutycapt willing to share? someone noted cutycapt works also and to me worked better but having issues getting it built into the nse script

  • forgot to add – Detection, notification, and deletion of 0-byte thumbnails.

    Hope this helps!

  • Picked this up right in the middle of an engagement… good stuff!
    Stuff i worked into the new verion:

    – Your(zenox) scope fixes and image quality speedhack.
    – PaulDotCom’s filenaming convention (changed the ‘:’ to ‘_’ so they’re cross-platform friendly)
    – A short bash script to alleviate the need for the 3rd party app ‘timeout’. Accepts the argument timeout, and default is 60 seconds.
    – Argument ‘arch=64’ makes it call the amd64 version.

    You can find it here – http://www.canhazcode.com/sc.html .

    Sometime very soon, I’ll release a web enumeration kit for taking this output and generating an elegant, searchable, easy-to-use front-end… it’s already making the process a LOT more efficient!

  • WordPress stripped some of that out. Shoot me an email, and I’ll respond there.

  • Wow, I recently came up with almost an identical solution and found this while searching for references for my own blog post (thanks for saving me the trouble :). I wish I would have stumbled across this sooner.

    The one thing I added was the ability to specify the timeout via a script argument. 15 seconds seem to be find on internal tests, but I felt I may want to increase that for external tests over slower connections. I have this in the “action” block:

    local timeout = 15
    if nmap.registry.args.timeout then
    timeout = nmap.registry.args.timeout
    end

    The actual command then looks like this (obviously substitute the 64-bit version of wkhtmltoimage if that’s your platform):

    local cmd = “timeout ” .. timeout .. ” wkhtmltoimage-i386 -n ” .. prefix .. “://” .. host.ip .. “:” .. port.number .. ” ” .. filename .. ” &> null”

    I also added this to my .bashrc because I’m lazy. It just creates an HTML file (which is unfortunately not W3C compliant) that includes the images and links them to the IP/Port they were found on. Just run it when you’re in the directory with the images.

    screen2html() {
    echo ‘img { border: solid black 1px; margin: 10px; }’ > screenshots.html

    for i in `ls *.png`
    do
    url=`echo $i | sed ‘s/screenshot-nmap-|.png//g’`
    echo ‘‘ >> screenshots.html
    done

    echo ” >> screenshots.html

    firefox screenshots.html
    }

    alias s2h=screen2html

  • These directions worked pretty good for me on Ubuntu 12.04 64bit.

    A few changes however.

    timeout package is already installed, so skip that part.

    Had to add local in front of 1 additional variable, and then I encountered the following error:
    /usr/local/bin/../share/nmap/scripts/http-screenshot.nse:60: variable ‘stdnse’ is not declared

    Looking at the code it was referencing the line:
    return stdnse.format_output(true, result)

    To correct this, just add local stdnse = require “stdnse” near the top of the file.

    Thanks Zeknox

  • did it work for https for you? didnt take any screens of https sites for me specially with self/client signed certs

    • zeknox

      Yes, HTTPS works for me, even with self signed certificates. I just tested this functionality against my Cisco ASA on the LAN and it worked great. Here is the syntax that I used:
      nmap -sV -p 443 10.0.0.1 --script=http-screenshot -v -d
      The -sV is critical to properly enumerate the type of service. Keep me posted if you still have issues.

    • I was having problems with https/self-signed certs as well. All I was getting was a blank screen shot and never could find a fix for wkhtmltoimage. As an alternative I use CutyCapt and that’s worked for me so far. Has a little bit better image resolution as well.

  • Leave a Reply

    Your email address will not be published. Required fields are marked *

    This site uses Akismet to reduce spam. Learn how your comment data is processed.

    Share This

    Recent Posts

    Subscribe To Our Mailing List

    The Ultimate Burp Suite Training Program

    Learn Network Penetration Testing

    Penetration Testing

    Categories

    Metasploit

    Web Application Hacking


    Brandon McCann

    Copyright 2024

    css.php

    Are You Using the Top 5 Pentest Tools?

    Enter your email address to download your copy of our FREE e-book and find out now!

    Thank you, now go check your email!!