Documentation for v3.0 training data

README.md

@@ -1,4 +1,8 @@
 # Eyeballer
+
+![Logo](/docs/eyeballer_logo.png)
+
+
 Give those screenshots of yours a quick eyeballing.
 
 Eyeballer is meant for large-scope network penetration tests where you need to find "interesting" targets from a huge set of web-based hosts. Go ahead and use your favorite screenshotting tool like normal (EyeWitness or GoWitness) and then run them through Eyeballer to tell you what's likely to contain vulnerabilities, and what isn't.
@@ -14,6 +18,10 @@ Eyeballer is meant for large-scope network penetration tests where you need to f
 | ------ |:-----:|
 | ![Sample Webapp](/docs/homepage.png) | ![Sample Custom 404](/docs/404.png) |
 
+| Parked Domains |
+| ------ |
+| ![Sample Webapp](/docs/parked.png) |
+
 ## What the Labels Mean
 
 **Old-Looking Sites**
@@ -28,6 +36,9 @@ This tells you that there is a larger group of pages and functionality available
 **Custom 404**
 Modern sites love to have cutesy custom 404 pages with pictures of broken robots or sad looking dogs. Unfortunately, they also love to return HTTP 200 response codes while they do it. More often, the "404" page doesn't even contain the text "404" in it. These pages are typically uninteresting, despite having a lot going on visually, and Eyeballer can help you sift them out.
 
+**Parked Domains**
+Parked domains are websites that look real, but aren't valid attack surface. They're stand-in pages, usually devoid of any real functionality, consist almost entirely of ads, and are usually not run by our actual target. It's what you get when the domain specified is wrong or lapsed. Finding these pages and removing them from scope is really valuable over time.
+
 ## Setup
 
 Download required packages on pip:
@@ -42,19 +53,26 @@ sudo pip3 install -r requirements-gpu.txt
 
 **NOTE**: Setting up a GPU for use with TensorFlow is way beyond the scope of this README. There's hardware compatibility to consider, drivers to install... There's a lot. So you're just going to have to figure this part out on your own if you want a GPU. But at least from a Python package perspective, the above requirements file has you covered.
 
+**Pretrained Weights**
+
+For the latest pretrained weights, check out the releases here on GitHub.
+
 **Training Data** You can find our training data here:
 
-https://www.dropbox.com/sh/7aouywaid7xptpq/AAD_-I4hAHrDeiosDAQksnBma?dl=1
+https://www.dropbox.com/s/rpylhiv2g0kokts/eyeballer-3.0.zip?dl=1
 
-Pretty soon, we're going to add this as a TensorFlow DataSet, so you don't need to download this separately like this. It'll also let us version the data a bit better. But for now, just deal with it. There's two things you need from the training data:
+There's two things you need from the training data:
 
-1. `images/` folder, containing all the screenshots (resized down to 224x140. We'll have the full-size images up soon)
+1. `images/` folder, containing all the screenshots (resized down to 224x224)
 2. `labels.csv` that has all the labels
-3. `bishop-fox-pretrained-v1.h5` A pretrained weights file you can use right out of the box without training.
+3. `bishop-fox-pretrained-v2.h5` A pretrained weights file you can use right out of the box without training.
 
 Copy all three into the root of the Eyeballer code tree.
 
 ## Predicting Labels
+
+NOTE: For best results, make sure you screenshot your websites in a native 1.6x aspect ratio. IE: 1440x900. Eyeballer will scale the image down automatically to the right size for you, but if it's the wrong aspect ratio then it will squish in a way that will affect prediction performance.
+
 To eyeball some screenshots, just run the "predict" mode:
 
 ```

eyeballer.py

@@ -54,7 +54,7 @@ def predict(ctx, screenshot, heatmap, threshold):
         print(results)
     else:
         with open("results.csv", "w", newline="") as csvfile:
-            fieldnames = ["filename", "custom404", "login", "webapp", "oldlooking"]
+            fieldnames = ["filename", "custom404", "login", "webapp", "oldlooking", "parked"]
             labelwriter = csv.DictWriter(csvfile, fieldnames=fieldnames)
             labelwriter.writeheader()
             labelwriter.writerows(results)

eyeballer/model.py

@@ -17,7 +17,7 @@ from tensorflow.keras.applications.mobilenet import preprocess_input
 from sklearn.metrics import classification_report, accuracy_score, hamming_loss
 from eyeballer.augmentation import EyeballerAugmentation
 
-DATA_LABELS = ["custom404", "login", "webapp", "oldlooking"]
+DATA_LABELS = ["custom404", "login", "webapp", "oldlooking", "parked"]
 
 
 class EyeballModel:
@@ -207,6 +207,7 @@ class EyeballModel:
         result["login"] = prediction[0][1]
         result["webapp"] = prediction[0][2]
         result["oldlooking"] = prediction[0][3]
+        result["parked"] = prediction[0][4]
         return result
 
     def predict(self, path, threshold=0.5):
@@ -251,6 +252,7 @@ class EyeballModel:
             result["login"] = prediction[0][1]
             result["webapp"] = prediction[0][2]
             result["oldlooking"] = prediction[0][3]
+            result["parked"] = prediction[0][4]
             results.append(result)
         return results
 

prediction_output_template.html

@@ -9,6 +9,7 @@
       <button class="btn" onclick="filterSelection('login')"> Login</button>
       <button class="btn" onclick="filterSelection('webapp')"> Web Application</button>
       <button class="btn" onclick="filterSelection('oldlooking')"> Old Looking</button>
+      <button class="btn" onclick="filterSelection('parked')"> Parked Domain</button>      
         <select onchange="numOnPage=this.value; filterSelection([]);" style="float:right; margin:5px; margin-right:30px; height:30px; width:auto;">
           <option value="25">25</option>
           <option value="50">50</option>

utils/resizer.sh

@@ -1,7 +1,8 @@
 #!/bin/bash
 
 # Converts full-size images to a given size (replacing the originals)
-for f in images_full_size/*
+for f in images/*
 do
-	convert -resize 224x224 $f $f
+	echo $f
+	convert -resize 224x224\! $f $f
 done

utils/verify.py

@@ -0,0 +1,28 @@
+#!/usr/bin/env python3
+
+import csv
+import random
+from os import listdir
+from os.path import isfile, join
+
+with open("labels.csv", newline="") as csvfile:
+    # Get the header labels
+    csvreader = csv.DictReader(csvfile)
+    # fieldnames = next(csvreader)
+    file_list = [f for f in listdir("images") if isfile(join("images", f))]
+
+    label_list = []
+    for row in csvreader:
+        label_list.append(row["filename"])
+
+    # Loop through the file
+    print("Rows in labels.csv, but no image exists:")
+    for label in label_list:
+        if label not in file_list:
+            print(row["filename"])
+
+    print("Images that exist, but don't have labels:")
+    i = 0
+    for image in file_list:
+        if image not in label_list:
+            print(image)