Removed warning about TF2.0 in the readme

- Sorta works
 - Frontend needs work, but functionality is there

.gitignore

@@ -104,9 +104,10 @@ venv.bak/
 .mypy_cache/
 
 #screenshot data
-*.png
-*.jpg
-*.tar
+images*
+
+#Graphs and Results
+graphs/*
 
 #model files
 *.h5
@@ -118,3 +119,7 @@ results.html
 !requirements-gpu.txt
 !tests/models/*
 !tests/data/*
+!docs/*
+
+#Leave the webui things alone for their own gitignore
+!webui/*

README.md

@@ -3,7 +3,45 @@ Give those screenshots of yours a quick eyeballing.
 
 Eyeballer is meant for large-scope network penetration tests where you need to find "interesting" targets from a huge set of web-based hosts. Go ahead and use your favorite screenshotting tool like normal (EyeWitness or GoWitness) and then run them through Eyeballer to tell you what's likely to contain vulnerabilities, and what isn't.
 
-## Setup
+
+### Example Labels
+
+| Old-Looking Sites | Login Pages |
+| ------ |:-----:|
+| ![Sample Old-looking Page](/docs/old-looking.png) | ![Sample Login Page](/docs/login.png) |
+
+| Homepages | Custom 404's |
+| ------ |:-----:|
+| ![Sample Homepage](/docs/homepage.png) | ![Sample Custom 404](/docs/404.png) |
+
+## Web UI Setup
+
+Eyeballer's Web UI uses TensorflowJS and React. It runs entirely in your browser locally, with no backend functionality whatsoever. So you can "upload" files to classify and get results without the files ever leaving your computer. To run the Eyeballer Web UI locally, you'll need Node. On Debian, you can install that with the npm package:
+
+```
+sudo apt install npm
+```
+
+Then you'll need to install some Node dependencies:
+
+```
+cd webui/
+npm install
+```
+
+Copy a pre-trained model file (in TensorflowJS format) into a jsmodel/ folder under public/, so that it's available in the web server.
+
+```
+cp YOUR_MODEL/ webui/public/
+```
+
+Then you can run the Web UI with:
+
+```
+npm start
+```
+
+## Python Setup
 
 Download required packages on pip:
 ```
@@ -15,19 +53,36 @@ Or if you want GPU support:
 sudo pip3 install -r requirements-gpu.txt
 ```
 
+**NOTE**: Setting up a GPU for use with TensorFlow is way beyond the scope of this README. There's hardware compatibility to consider, drivers to install... There's a lot. So you're just going to have to figure this part out on your own if you want a GPU. But at least from a Python package perspective, the above requirements file has you covered.
+
+**Training Data** You can find our training data here:
+
+https://www.dropbox.com/sh/7aouywaid7xptpq/AAD_-I4hAHrDeiosDAQksnBma?dl=1
+
+Pretty soon, we're going to add this as a TensorFlow DataSet, so you don't need to download this separately like this. It'll also let us version the data a bit better. But for now, just deal with it. There's two things you need from the training data:
+
+1. `images/` folder, containing all the screenshots (resized down to 224x140. We'll have the full-size images up soon)
+2. `labels.csv` that has all the labels
+3. `bishop-fox-pretrained-v1.h5` A pretrained weights file you can use right out of the box without training.
+
+Copy all three into the root of the Eyeballer code tree.
+
 ## Predicting Labels
 To eyeball some screenshots, just run the "predict" mode:
 
 ```
-eyeballer.py --weights YOUR_WEIGHTS.h5 predict YOUR_FILE.png
+eyeballer.py --model YOUR_MODEL.h5 predict YOUR_FILE.png
 ```
 
 Or for a whole directory of files:
 
 ```
-eyeballer.py --weights YOUR_WEIGHTS.h5 predict PATH_TO/YOUR_FILES/
+eyeballer.py --model YOUR_MODEL.h5 predict PATH_TO/YOUR_FILES/
 ```
 
+Eyeballer will spit the results back to you in human readable format (a `results.html` file so you can browse it easily) and machine readable format (a `results.csv` file).
+
+
 ## Training
 To train a new model, run:
 ```
@@ -36,12 +91,14 @@ eyeballer.py train
 
 You'll want a machine with a good GPU for this to run in a reasonable amount of time. Setting that up is outside the scope of this readme, however.
 
-This will output a new model file (weights.h5 by default).
+This will output a new model file (model.h5 by default).
 
 ## Evaluation
 
 You just trained a new model, cool! Let's see how well it performs against some images it's never seen before, across a variety of metrics:
 
 ```
-eyeballer.py --weights YOUR_WEIGHTS.h5 evaluate
+eyeballer.py --model YOUR_MODEL.h5 evaluate
 ```
+
+The output will describe the model's accuracy in both recall and precision for each of the program's labels. (Including "none of the above" as a pseudo-label)

eyeballer.py

@@ -9,12 +9,12 @@ from jinja2 import Template
 
 
 @click.group(invoke_without_command=True)
-@click.option('--weights', default=None, type=click.Path(), help="Weights file for input/output")
+@click.option('--model', default=None, type=click.Path(), help="Model file for input/output")
 @click.option('--summary/--no-summary', default=False, help="Print model summary at start")
 @click.option('--seed', default=None, type=int, help="RNG seed for data shuffling and transformations, defaults to random value")
 @click.pass_context
-def cli(ctx, weights, summary, seed):
-    model_kwargs = {"weights_file": weights,
+def cli(ctx, model, summary, seed):
+    model_kwargs = {"model_file": model,
                     "print_summary": summary,
                     "seed": seed}
 

eyeballer/augmentation.py

@@ -1,7 +1,7 @@
 import numpy as np
 
 from Augmentor.Operations import Operation
-from keras.applications.mobilenet import preprocess_input
+from tensorflow.keras.applications.mobilenet import preprocess_input
 
 
 class EyeballerAugmentation(Operation):

eyeballer/model.py

@@ -26,17 +26,17 @@ class EyeballModel:
     Contains high-level functions for training, evaluating, and predicting.
     """
     graphs_directory = "graphs/"
-    checkpoint_file = "weights.h5"
+    checkpoint_file = "model.h5"
     image_dir = "images/"
     image_width, image_height = 224, 224
     input_shape = (image_width, image_height, 3)
 
-    def __init__(self, weights_file, print_summary=False, seed=None):
+    def __init__(self, model_file, print_summary=False, seed=None, quiet=False):
         """Constructor for model class.
 
         Keyword arguments:
         print_summary -- Whether or not to print to stdout the keras model summary, containing a detailed description of every model layer
-        weights_file -- A filename for where to load the model's weights.
+        model_file -- A filename for where to load the model's model.
         seed -- PRNG seed, useful for repeating a previous run and using the same data. Training/Validation split is determined randomly.
         """
         # # Build the model
@@ -60,6 +60,8 @@ class EyeballModel:
         if print_summary:
             print(self.model.summary())
 
+        self.quiet = quiet
+
         # Pull out our labels for use in generators later
         data = pd.read_csv("labels.csv")
         self.training_labels = data.loc[data['evaluation'] == False]  # noqa: E712
@@ -76,15 +78,15 @@ class EyeballModel:
         random.seed(self.seed)
         self.training_labels = self.training_labels.sample(frac=1)
 
-        if weights_file is not None and os.path.isfile(weights_file):
+        if model_file is not None and os.path.isfile(model_file):
             try:
-                self.model.load_weights(weights_file)
+                self.model = tf.keras.models.load_model(model_file)
             except OSError:
-                print("ERROR: Unable to open weights file '{}'".foramt(weights_file))
+                print("ERROR: Unable to open model file '{}'".format(model_file))
                 sys.exit(-1)
             print("Loaded model from file.")
         else:
-            if weights_file is not None:
+            if model_file is not None:
                 raise FileNotFoundError
             print("WARN: No model loaded from file. Generating random model")
 
@@ -103,7 +105,7 @@ class EyeballModel:
         self.preprocess_training_function = augmentor.keras_preprocess_func()
 
     def train(self, epochs=20, batch_size=32, print_graphs=False):
-        """Train the model, making a new weights file at each successfull checkpoint. You'll probably need a GPU for this to realistically run.
+        """Train the model, making a new model file at each successfull checkpoint. You'll probably need a GPU for this to realistically run.
 
         Keyword arguments:
         epochs -- The number of epochs to train for. (An epoch is one pass-through of the dataset)
@@ -140,12 +142,12 @@ class EyeballModel:
             seed=self.seed,
             class_mode="other")
 
-        # Model checkpoint - Saves model weights when validation accuracy improves
+        # Model checkpoint - Saves model when validation accuracy improves
         callbacks = [tf.keras.callbacks.ModelCheckpoint(self.checkpoint_file,
                      monitor='val_loss',
                      verbose=1,
                      save_best_only=True,
-                     save_weights_only=True,
+                     save_weights_only=False,
                      mode='min')]
 
         history = self.model.fit_generator(
@@ -216,7 +218,10 @@ class EyeballModel:
             raise FileNotFoundError
 
         results = []
-        for screenshot in progressbar.progressbar(screenshots):
+        bar = progressbar.ProgressBar()
+        if self.quiet:
+            bar = progressbar.NullBar()
+        for screenshot in bar(screenshots):
             # Load the image into memory
             img = None
             try:
@@ -287,7 +292,6 @@ class EyeballModel:
             steps=len(evaluation_generator))
 
         self._save_prediction_histograms(predictions)
-        print("predictions: ", type(predictions))
         predictions = predictions > threshold
         ground_truth = self.evaluation_labels[DATA_LABELS].to_numpy()
         filenames = self.evaluation_labels[["filename"]].to_numpy()

eyeballer/visualization.py

@@ -1,6 +1,7 @@
 import numpy as np
 import math
-from keras.preprocessing import image
+import tensorflow as tf
+
 from copy import deepcopy
 from matplotlib import pyplot as plt
 from eyeballer.model import DATA_LABELS
@@ -10,8 +11,8 @@ class HeatMap():
     def __init__(self, screenshot_path, model, threshold=0.5, boxsize=28, step=7):
         self.model = model
 
-        img = image.load_img(screenshot_path, target_size=(self.model.image_width, self.model.image_height))
-        img = image.img_to_array(img)
+        img = tf.keras.preprocessing.image.load_img(screenshot_path, target_size=(self.model.image_width, self.model.image_height))
+        img = tf.keras.preprocessing.image.img_to_array(img)
 
         self.screenshot = img
         self.boxsize = boxsize
@@ -43,7 +44,7 @@ class HeatMap():
                         labels.append(label)
 
         plt.figure()
-        screenshot_image = image.load_img(self.screenshot_path, target_size=(self.model.image_width, self.model.image_width))
+        screenshot_image = tf.keras.preprocessing.image.load_img(self.screenshot_path, target_size=(self.model.image_width, self.model.image_width))
         _, subplots = plt.subplots(1, len(heatmaps))
         # This is a little janky, but matplotlib returns a list above if there's multiple subplots, and just a single subplot if there's only one
         #   it would have been easier if it was always a list. But alas

requirements-gpu.txt

@@ -5,6 +5,6 @@ numpy
 pandas
 pillow
 sklearn
-tensorflow-gpu==2.0.0-beta0
+tensorflow-gpu
 jinja2
 progressbar2

requirements.txt

@@ -5,6 +5,6 @@ numpy
 pandas
 pillow
 sklearn
-tensorflow==2.0.0-beta0
+tensorflow
 jinja2
 progressbar2

tests/EvaluateTest.py

@@ -1 +0,0 @@
-#TODO

tests/PredictTest.py

@@ -4,6 +4,7 @@ import unittest
 import sys
 
 from eyeballer.model import EyeballModel
+from eyeballer.visualization import HeatMap
 
 
 class PredictTest(unittest.TestCase):
@@ -14,26 +15,28 @@ class PredictTest(unittest.TestCase):
             def flush(self): pass
 
         sys.stdout = DummyFile()
-        weights_file = "tests/models/test_weights.h5"
-        if not os.path.isfile(weights_file):
-            print("Error: Symlink the latest weights file to " + weights_file)
+        model_file = "tests/models/model_file.h5"
+        if not os.path.isfile(model_file):
+            print("Error: Symlink the latest model file to " + model_file)
             raise FileNotFoundError(
                 errno.ENOENT,
                 os.strerror(errno.ENOENT),
-                weights_file)
+                model_file)
 
         model_kwargs = {
-            "weights_file": weights_file,
+            "model_file": model_file,
             "print_summary": False,
-            "seed": None
+            "seed": None,
+            "quiet": True
         }
         self.model = EyeballModel(**model_kwargs)
 
     def test_different_seed_predict(self):
         model_kwargs = {
-            "weights_file": None,
+            "model_file": None,
             "print_summary": False,
-            "seed": 12345678
+            "seed": 12345678,
+            "quiet": True
         }
         same_seed_model = EyeballModel(**model_kwargs)
 
@@ -46,9 +49,10 @@ class PredictTest(unittest.TestCase):
 
     def test_same_seed_predict(self):
         model_kwargs = {
-            "weights_file": None,
+            "model_file": None,
             "print_summary": False,
-            "seed": 12345678
+            "seed": 12345678,
+            "quiet": True
         }
         same_seed_model = EyeballModel(**model_kwargs)
 
@@ -126,3 +130,13 @@ class PredictTest(unittest.TestCase):
         """
         screenshot = "tests/data/invalid.png"
         self.model.predict(screenshot)
+
+    def test_heatmap(self):
+        screenshot = "tests/data/login.png"
+        HeatMap(screenshot, self.model, 0.5)
+
+        screenshot = "tests/data/404.png"
+        HeatMap(screenshot, self.model, 0.5)
+
+        screenshot = "tests/data/empty.png"
+        HeatMap(screenshot, self.model, 0.5)

tests/TrainTest.py

@@ -1 +0,0 @@
-#TODO

tests/__init__.py

@@ -1 +1 @@
-from tests.PredictTest import *
+from tests.PredictTest import *  # noqa

webui/.gitignore

@@ -0,0 +1,20 @@
+# See https://help.github.com/ignore-files/ for more about ignoring files.
+
+# dependencies
+/node_modules
+
+# testing
+/coverage
+
+# production
+/build
+
+# misc
+.DS_Store
+.env
+npm-debug.log*
+yarn-debug.log*
+yarn-error.log*
+
+#Model files
+public/jsmodel/*

webui/package.json

@@ -0,0 +1,20 @@
+{
+  "name": "eyeballer-webui",
+  "version": "0.1.0",
+  "private": true,
+  "dependencies": {
+    "react": "^16.11.0",
+    "react-dom": "^16.11.0",
+    "react-dropzone": "^10.1.10",
+    "react-scripts": "0.9.5",
+    "bootstrap": "^4.3.1",
+    "@tensorflow/tfjs": "^1.3.2"
+  },
+  "devDependencies": {},
+  "scripts": {
+    "start": "react-scripts start",
+    "build": "react-scripts build",
+    "test": "react-scripts test --env=jsdom",
+    "eject": "react-scripts eject"
+  }
+}

webui/public/index.html

@@ -0,0 +1,31 @@
+<!doctype html>
+<html lang="en">
+  <head>
+    <meta charset="utf-8">
+    <meta name="viewport" content="width=device-width, initial-scale=1">
+    <link rel="shortcut icon" href="%PUBLIC_URL%/favicon.png">
+    <!--
+      Notice the use of %PUBLIC_URL% in the tag above.
+      It will be replaced with the URL of the `public` folder during the build.
+      Only files inside the `public` folder can be referenced from the HTML.
+
+      Unlike "/favicon.ico" or "favicon.ico", "%PUBLIC_URL%/favicon.ico" will
+      work correctly both with client-side routing and a non-root public URL.
+      Learn how to configure a non-root public URL by running `npm run build`.
+    -->
+    <title>React App</title>
+  </head>
+  <body>
+    <div id="root"></div>
+    <!--
+      This HTML file is a template.
+      If you open it directly in the browser, you will see an empty page.
+
+      You can add webfonts, meta tags, or analytics to this file.
+      The build step will place the bundled scripts into the <body> tag.
+
+      To begin the development, run `npm start`.
+      To create a production bundle, use `npm run build`.
+    -->
+  </body>
+</html>

webui/src/App.css

@@ -0,0 +1,23 @@
+.App {
+  text-align: center;
+}
+
+.App-logo {
+  height: 100px;
+}
+
+.App-header {
+  background-color: #222;
+  height: 170px;
+  padding: 10px;
+  color: white;
+}
+
+.App-intro {
+  font-size: large;
+}
+
+.thumbnail {
+  height: 280px;
+  width: 480px;
+}

webui/src/App.js

@@ -0,0 +1,132 @@
+import React, { Component } from 'react';
+import logo from './eyeballer_logo.png';
+import './App.css';
+import 'bootstrap/dist/css/bootstrap.min.css';
+import * as tf from '@tensorflow/tfjs';
+
+class App extends Component {
+  constructor(props) {
+    super(props);
+    this.state = {
+      fileContents: "Try uploading a file",
+    }
+  }
+
+  render() {
+    return (
+      <div className="App">
+        <div className="App-header">
+          <img src={logo} className="App-logo" alt="logo" />
+          <h2>Eyeballer</h2>
+        </div>
+        <PredictButton />
+        <ImageFile />
+
+      </div>
+    );
+  }
+}
+
+export default App;
+
+class ImageFile extends React.Component {
+  constructor(props){
+    super(props);
+    this.state = {
+      id : "someUniqueId", // I would use this.props.id for a real world implementation
+      imageURI : null
+    }
+  }
+
+  buildImgTag(){
+    let imgTag = null;
+    if (this.state.imageURI !== null)
+      imgTag = (<div className="row">
+                  <div className="small-9 small-centered columns">
+                    <img className="thumbnail" id="thumbnail" alt="Screenshot to be classified" src={this.state.imageURI}></img>
+                  </div>
+                </div>);
+    return imgTag;
+  }
+
+  readURI(e){
+    if(e.target.files && e.target.files[0]){
+      let reader = new FileReader();
+      reader.onload = function(ev){
+        this.setState({imageURI:ev.target.result});
+      }.bind(this);
+      reader.readAsDataURL(e.target.files[0]);
+    }
+  }
+
+  handleChange(e){
+    this.readURI(e); // maybe call this with webworker or async library?
+    if (this.props.onChange !== undefined)
+      this.props.onChange(e); // propagate to parent component
+  }
+
+  render() {
+    const imgTag = this.buildImgTag();
+
+    return <div>
+            <label
+              htmlFor={this.state.id}
+              className="button">
+              Upload an image
+            </label>
+            <input
+              id={this.state.id}
+              type="file"
+              onChange={this.handleChange.bind(this)}
+              className="show-for-sr" />
+            {imgTag}
+          </div>;
+  }
+}
+
+class PredictButton extends Component {
+  constructor(props) {
+    super(props);
+    this.state = {
+      classification: ""
+    }
+  }
+
+  onpress = async () => {
+    const modelUrl = 'jsmodel/model.json';
+    const model = await tf.loadLayersModel(modelUrl);
+    const offset = tf.scalar(127.5);
+    const image = tf.browser.fromPixels(document.getElementById('thumbnail'))
+      .resizeNearestNeighbor([224,224])
+      .toFloat()
+      .sub(offset)
+      .div(offset)
+      .expandDims();
+
+    const predictions = Array.from(model.predict(image).dataSync());
+    this.setState({classification: ""});
+    if (predictions[0] > 0.5) {
+      this.setState({classification: this.state.classification + "Custom 404, "});
+    }
+    if (predictions[1] > 0.5) {
+      this.setState({classification: this.state.classification + "Login Page, "});
+    }
+    if (predictions[2] > 0.5) {
+      this.setState({classification: this.state.classification + "Homepage, "});
+    }
+    if (predictions[3] > 0.5) {
+      this.setState({classification: this.state.classification + "Old Looking "});
+    }
+
+  }
+
+  render() {
+    return (
+      <div className="custom-file">
+        <button className="default" type="button" id="predictButton" onClick={this.onpress}>Classify</button>
+        {this.state.classification}
+      </div>
+    );
+  }
+
+}

webui/src/App.test.js

@@ -0,0 +1,8 @@
+import React from 'react';
+import ReactDOM from 'react-dom';
+import App from './App';
+
+it('renders without crashing', () => {
+  const div = document.createElement('div');
+  ReactDOM.render(<App />, div);
+});

webui/src/index.css

@@ -0,0 +1,5 @@
+body {
+  margin: 0;
+  padding: 0;
+  font-family: sans-serif;
+}

webui/src/index.js

@@ -0,0 +1,9 @@
+import React from 'react';
+import ReactDOM from 'react-dom';
+import App from './App';
+import './index.css';
+
+ReactDOM.render(
+  <App />,
+  document.getElementById('root')
+);