Change name in HTML output file to "webapp"

- Update some minor API things to not use deprecated features

.gitignore

@@ -104,10 +104,9 @@ venv.bak/
 .mypy_cache/
 
 #screenshot data
-images*
-
-#Graphs and Results
-graphs/*
+*.png
+*.jpg
+*.tar
 
 #model files
 *.h5
@@ -120,6 +119,4 @@ results.html
 !tests/models/*
 !tests/data/*
 !docs/*
-
-#Leave the webui things alone for their own gitignore
-!webui/*
+logs/*

README.md

@@ -10,38 +10,25 @@ Eyeballer is meant for large-scope network penetration tests where you need to f
 | ------ |:-----:|
 | ![Sample Old-looking Page](/docs/old-looking.png) | ![Sample Login Page](/docs/login.png) |
 
-| Homepages | Custom 404's |
+| Webapp | Custom 404's |
 | ------ |:-----:|
-| ![Sample Homepage](/docs/homepage.png) | ![Sample Custom 404](/docs/404.png) |
+| ![Sample Webapp](/docs/homepage.png) | ![Sample Custom 404](/docs/404.png) |
 
-## Web UI Setup
+## What the Labels Mean
 
-Eyeballer's Web UI uses TensorflowJS and React. It runs entirely in your browser locally, with no backend functionality whatsoever. So you can "upload" files to classify and get results without the files ever leaving your computer. To run the Eyeballer Web UI locally, you'll need Node. On Debian, you can install that with the npm package:
+**Old-Looking Sites**
+Blocky frames, broken CSS, that certain "je ne sais quoi" of a website that looks like it was designed in the early 2000's. You know it when you see it. Old websites aren't just ugly, they're also typically super vulnerable. When you're looking to hack into something, these websites are a gold mine.
 
-```
-sudo apt install npm
-```
+**Login Pages**
+Login pages are valuable to pen testing, they indicate that there's additional functionality you don't currently have access to. It also means there's a simple follow-up process of credential enumeration attacks. You might think that you can set a simple heuristic to find login pages, but in practice it's really hard. Modern sites don't just use a simple input tag we can grep for.
 
-Then you'll need to install some Node dependencies:
+**Webapp**
+This tells you that there is a larger group of pages and functionality available here that can serve as surface area to attack. This is in contrast to a simple login page, with no other functionality. Or a default IIS landing page which has no other functionality. This label should indicate to you that there is a web application here to attack.
 
-```
-cd webui/
-npm install
-```
+**Custom 404**
+Modern sites love to have cutesy custom 404 pages with pictures of broken robots or sad looking dogs. Unfortunately, they also love to return HTTP 200 response codes while they do it. More often, the "404" page doesn't even contain the text "404" in it. These pages are typically uninteresting, despite having a lot going on visually, and Eyeballer can help you sift them out.
 
-Copy a pre-trained model file (in TensorflowJS format) into a jsmodel/ folder under public/, so that it's available in the web server.
-
-```
-cp YOUR_MODEL/ webui/public/
-```
-
-Then you can run the Web UI with:
-
-```
-npm start
-```
-
-## Python Setup
+## Setup
 
 Download required packages on pip:
 ```
@@ -71,13 +58,13 @@ Copy all three into the root of the Eyeballer code tree.
 To eyeball some screenshots, just run the "predict" mode:
 
 ```
-eyeballer.py --model YOUR_MODEL.h5 predict YOUR_FILE.png
+eyeballer.py --weights YOUR_WEIGHTS.h5 predict YOUR_FILE.png
 ```
 
 Or for a whole directory of files:
 
 ```
-eyeballer.py --model YOUR_MODEL.h5 predict PATH_TO/YOUR_FILES/
+eyeballer.py --weights YOUR_WEIGHTS.h5 predict PATH_TO/YOUR_FILES/
 ```
 
 Eyeballer will spit the results back to you in human readable format (a `results.html` file so you can browse it easily) and machine readable format (a `results.csv` file).
@@ -91,14 +78,14 @@ eyeballer.py train
 
 You'll want a machine with a good GPU for this to run in a reasonable amount of time. Setting that up is outside the scope of this readme, however.
 
-This will output a new model file (model.h5 by default).
+This will output a new model file (weights.h5 by default).
 
 ## Evaluation
 
 You just trained a new model, cool! Let's see how well it performs against some images it's never seen before, across a variety of metrics:
 
 ```
-eyeballer.py --model YOUR_MODEL.h5 evaluate
+eyeballer.py --weights YOUR_WEIGHTS.h5 evaluate
 ```
 
 The output will describe the model's accuracy in both recall and precision for each of the program's labels. (Including "none of the above" as a pseudo-label)

eyeballer.py

@@ -9,12 +9,12 @@ from jinja2 import Template
 
 
 @click.group(invoke_without_command=True)
-@click.option('--model', default=None, type=click.Path(), help="Model file for input/output")
+@click.option('--weights', default=None, type=click.Path(), help="Weights file for input/output")
 @click.option('--summary/--no-summary', default=False, help="Print model summary at start")
 @click.option('--seed', default=None, type=int, help="RNG seed for data shuffling and transformations, defaults to random value")
 @click.pass_context
-def cli(ctx, model, summary, seed):
-    model_kwargs = {"model_file": model,
+def cli(ctx, weights, summary, seed):
+    model_kwargs = {"weights_file": weights,
                     "print_summary": summary,
                     "seed": seed}
 
@@ -54,7 +54,7 @@ def predict(ctx, screenshot, heatmap, threshold):
         print(results)
     else:
         with open("results.csv", "w", newline="") as csvfile:
-            fieldnames = ["filename", "custom404", "login", "homepage", "oldlooking"]
+            fieldnames = ["filename", "custom404", "login", "webapp", "oldlooking"]
             labelwriter = csv.DictWriter(csvfile, fieldnames=fieldnames)
             labelwriter.writeheader()
             labelwriter.writerows(results)

eyeballer/model.py

@@ -17,7 +17,7 @@ from tensorflow.keras.applications.mobilenet import preprocess_input
 from sklearn.metrics import classification_report, accuracy_score, hamming_loss
 from eyeballer.augmentation import EyeballerAugmentation
 
-DATA_LABELS = ["custom404", "login", "homepage", "oldlooking"]
+DATA_LABELS = ["custom404", "login", "webapp", "oldlooking"]
 
 
 class EyeballModel:
@@ -26,17 +26,17 @@ class EyeballModel:
     Contains high-level functions for training, evaluating, and predicting.
     """
     graphs_directory = "graphs/"
-    checkpoint_file = "model.h5"
+    checkpoint_file = "weights.h5"
     image_dir = "images/"
     image_width, image_height = 224, 224
     input_shape = (image_width, image_height, 3)
 
-    def __init__(self, model_file, print_summary=False, seed=None, quiet=False):
+    def __init__(self, weights_file, print_summary=False, seed=None, quiet=False):
         """Constructor for model class.
 
         Keyword arguments:
         print_summary -- Whether or not to print to stdout the keras model summary, containing a detailed description of every model layer
-        model_file -- A filename for where to load the model's model.
+        weights_file -- A filename for where to load the model's weights.
         seed -- PRNG seed, useful for repeating a previous run and using the same data. Training/Validation split is determined randomly.
         """
         # # Build the model
@@ -54,14 +54,25 @@ class EyeballModel:
                            loss="binary_crossentropy",
                            metrics=["accuracy"])
 
-        # for layer in pretrained_layer.layers:
-        #     layer.trainable = False
+        if weights_file is not None and os.path.isfile(weights_file):
+            try:
+                self.model.load_weights(weights_file)
+            except OSError:
+                print("ERROR: Unable to open weights file '{}'".foramt(weights_file))
+                sys.exit(-1)
+            print("Loaded model from file.")
+        else:
+            if weights_file is not None:
+                raise FileNotFoundError
+            print("WARN: No model loaded from file. Generating random model")
 
         if print_summary:
             print(self.model.summary())
 
         self.quiet = quiet
+        self.seed = seed
 
+    def _init_labels(self):
         # Pull out our labels for use in generators later
         data = pd.read_csv("labels.csv")
         self.training_labels = data.loc[data['evaluation'] == False]  # noqa: E712
@@ -69,7 +80,6 @@ class EyeballModel:
 
         # Shuffle the training labels
         self.random_seed = False
-        self.seed = seed
         if self.seed is None:
             self.random_seed = True
             self.seed = random.randint(0, 999999)
@@ -78,18 +88,6 @@ class EyeballModel:
         random.seed(self.seed)
         self.training_labels = self.training_labels.sample(frac=1)
 
-        if model_file is not None and os.path.isfile(model_file):
-            try:
-                self.model = tf.keras.models.load_model(model_file)
-            except OSError:
-                print("ERROR: Unable to open model file '{}'".format(model_file))
-                sys.exit(-1)
-            print("Loaded model from file.")
-        else:
-            if model_file is not None:
-                raise FileNotFoundError
-            print("WARN: No model loaded from file. Generating random model")
-
         # Data augmentation
         augmentor = Augmentor.Pipeline()
         augmentor.set_seed(self.seed)
@@ -105,7 +103,7 @@ class EyeballModel:
         self.preprocess_training_function = augmentor.keras_preprocess_func()
 
     def train(self, epochs=20, batch_size=32, print_graphs=False):
-        """Train the model, making a new model file at each successfull checkpoint. You'll probably need a GPU for this to realistically run.
+        """Train the model, making a new weights file at each successfull checkpoint. You'll probably need a GPU for this to realistically run.
 
         Keyword arguments:
         epochs -- The number of epochs to train for. (An epoch is one pass-through of the dataset)
@@ -114,6 +112,7 @@ class EyeballModel:
         """
         print("Training with seed: " + str(self.seed))
 
+        self._init_labels()
         data_generator = tf.keras.preprocessing.image.ImageDataGenerator(
             preprocessing_function=self.preprocess_training_function,
             validation_split=0.2,
@@ -129,7 +128,7 @@ class EyeballModel:
             subset='training',
             shuffle=True,
             seed=self.seed,
-            class_mode="other")
+            class_mode="raw")
         validation_generator = data_generator.flow_from_dataframe(
             self.training_labels,
             directory=self.image_dir,
@@ -140,17 +139,26 @@ class EyeballModel:
             subset='validation',
             shuffle=False,
             seed=self.seed,
-            class_mode="other")
+            class_mode="raw")
 
-        # Model checkpoint - Saves model when validation accuracy improves
+        # Model checkpoint - Saves model weights when validation accuracy improves
         callbacks = [tf.keras.callbacks.ModelCheckpoint(self.checkpoint_file,
-                     monitor='val_loss',
-                     verbose=1,
-                     save_best_only=True,
-                     save_weights_only=False,
-                     mode='min')]
+                                                        monitor='val_loss',
+                                                        verbose=1,
+                                                        save_best_only=True,
+                                                        save_weights_only=True,
+                                                        mode='min'),
+                     tf.keras.callbacks.TensorBoard(log_dir='logs',
+                                                    histogram_freq=2,
+                                                    write_graph=True,
+                                                    write_images=False,
+                                                    update_freq='epoch',
+                                                    profile_batch=2,
+                                                    embeddings_freq=0,
+                                                    embeddings_metadata=None),
+                     ]
 
-        history = self.model.fit_generator(
+        history = self.model.fit(
             training_generator,
             steps_per_epoch=len(training_generator.filenames) // batch_size,
             epochs=epochs,
@@ -197,7 +205,7 @@ class EyeballModel:
         result["filename"] = "custom-image"
         result["custom404"] = prediction[0][0]
         result["login"] = prediction[0][1]
-        result["homepage"] = prediction[0][2]
+        result["webapp"] = prediction[0][2]
         result["oldlooking"] = prediction[0][3]
         return result
 
@@ -241,7 +249,7 @@ class EyeballModel:
             result["filename"] = screenshot
             result["custom404"] = prediction[0][0]
             result["login"] = prediction[0][1]
-            result["homepage"] = prediction[0][2]
+            result["webapp"] = prediction[0][2]
             result["oldlooking"] = prediction[0][3]
             results.append(result)
         return results
@@ -253,6 +261,7 @@ class EyeballModel:
         threshold -- Value between 0->1. The cutoff where the numerical prediction becomes boolean. Default: 0.5
         """
         # Prepare the data
+        self._init_labels()
         data_generator = tf.keras.preprocessing.image.ImageDataGenerator(
             preprocessing_function=preprocess_input)
         evaluation_generator = data_generator.flow_from_dataframe(
@@ -263,7 +272,7 @@ class EyeballModel:
             target_size=(self.image_width, self.image_height),
             shuffle=False,
             batch_size=1,
-            class_mode="other")
+            class_mode="raw")
         # If a seed was selected, then also evaluate on the validation set for that seed
         if not self.random_seed:
             print("Using validation set...")
@@ -282,11 +291,11 @@ class EyeballModel:
                 subset='validation',
                 shuffle=False,
                 seed=self.seed,
-                class_mode="other")
+                class_mode="raw")
         else:
             print("Using evaluation set...")
 
-        predictions = self.model.predict_generator(
+        predictions = self.model.predict(
             evaluation_generator,
             verbose=1,
             steps=len(evaluation_generator))

prediction_output_template.html

@@ -7,7 +7,7 @@
       <button class="btn all" onclick="filterSelection('all')"> Show all</button>
       <button class="btn" onclick="filterSelection('custom404')"> 404</button>
       <button class="btn" onclick="filterSelection('login')"> Login</button>
-      <button class="btn" onclick="filterSelection('homepage')"> Home Page</button>
+      <button class="btn" onclick="filterSelection('webapp')"> Web Application</button>
       <button class="btn" onclick="filterSelection('oldlooking')"> Old Looking</button>
         <select onchange="numOnPage=this.value; filterSelection([]);" style="float:right; margin:5px; margin-right:30px; height:30px; width:auto;">
           <option value="25">25</option>

tests/PredictTest.py

@@ -15,16 +15,16 @@ class PredictTest(unittest.TestCase):
             def flush(self): pass
 
         sys.stdout = DummyFile()
-        model_file = "tests/models/model_file.h5"
-        if not os.path.isfile(model_file):
-            print("Error: Symlink the latest model file to " + model_file)
+        weights_file = "tests/models/test_weights.h5"
+        if not os.path.isfile(weights_file):
+            print("Error: Symlink the latest weights file to " + weights_file)
             raise FileNotFoundError(
                 errno.ENOENT,
                 os.strerror(errno.ENOENT),
-                model_file)
+                weights_file)
 
         model_kwargs = {
-            "model_file": model_file,
+            "weights_file": weights_file,
             "print_summary": False,
             "seed": None,
             "quiet": True
@@ -33,7 +33,7 @@ class PredictTest(unittest.TestCase):
 
     def test_different_seed_predict(self):
         model_kwargs = {
-            "model_file": None,
+            "weights_file": None,
             "print_summary": False,
             "seed": 12345678,
             "quiet": True
@@ -49,7 +49,7 @@ class PredictTest(unittest.TestCase):
 
     def test_same_seed_predict(self):
         model_kwargs = {
-            "model_file": None,
+            "weights_file": None,
             "print_summary": False,
             "seed": 12345678,
             "quiet": True

webui/.gitignore

@@ -1,20 +0,0 @@
-# See https://help.github.com/ignore-files/ for more about ignoring files.
-
-# dependencies
-/node_modules
-
-# testing
-/coverage
-
-# production
-/build
-
-# misc
-.DS_Store
-.env
-npm-debug.log*
-yarn-debug.log*
-yarn-error.log*
-
-#Model files
-public/jsmodel/*

webui/package.json

@@ -1,20 +0,0 @@
-{
-  "name": "eyeballer-webui",
-  "version": "0.1.0",
-  "private": true,
-  "dependencies": {
-    "react": "^16.11.0",
-    "react-dom": "^16.11.0",
-    "react-dropzone": "^10.1.10",
-    "react-scripts": "0.9.5",
-    "bootstrap": "^4.3.1",
-    "@tensorflow/tfjs": "^1.3.2"
-  },
-  "devDependencies": {},
-  "scripts": {
-    "start": "react-scripts start",
-    "build": "react-scripts build",
-    "test": "react-scripts test --env=jsdom",
-    "eject": "react-scripts eject"
-  }
-}

webui/public/index.html

@@ -1,31 +0,0 @@
-<!doctype html>
-<html lang="en">
-  <head>
-    <meta charset="utf-8">
-    <meta name="viewport" content="width=device-width, initial-scale=1">
-    <link rel="shortcut icon" href="%PUBLIC_URL%/favicon.png">
-    <!--
-      Notice the use of %PUBLIC_URL% in the tag above.
-      It will be replaced with the URL of the `public` folder during the build.
-      Only files inside the `public` folder can be referenced from the HTML.
-
-      Unlike "/favicon.ico" or "favicon.ico", "%PUBLIC_URL%/favicon.ico" will
-      work correctly both with client-side routing and a non-root public URL.
-      Learn how to configure a non-root public URL by running `npm run build`.
-    -->
-    <title>React App</title>
-  </head>
-  <body>
-    <div id="root"></div>
-    <!--
-      This HTML file is a template.
-      If you open it directly in the browser, you will see an empty page.
-
-      You can add webfonts, meta tags, or analytics to this file.
-      The build step will place the bundled scripts into the <body> tag.
-
-      To begin the development, run `npm start`.
-      To create a production bundle, use `npm run build`.
-    -->
-  </body>
-</html>

webui/src/App.css

@@ -1,23 +0,0 @@
-.App {
-  text-align: center;
-}
-
-.App-logo {
-  height: 100px;
-}
-
-.App-header {
-  background-color: #222;
-  height: 170px;
-  padding: 10px;
-  color: white;
-}
-
-.App-intro {
-  font-size: large;
-}
-
-.thumbnail {
-  height: 280px;
-  width: 480px;
-}

webui/src/App.js

@@ -1,132 +0,0 @@
-import React, { Component } from 'react';
-import logo from './eyeballer_logo.png';
-import './App.css';
-import 'bootstrap/dist/css/bootstrap.min.css';
-import * as tf from '@tensorflow/tfjs';
-
-class App extends Component {
-  constructor(props) {
-    super(props);
-    this.state = {
-      fileContents: "Try uploading a file",
-    }
-  }
-
-  render() {
-    return (
-      <div className="App">
-        <div className="App-header">
-          <img src={logo} className="App-logo" alt="logo" />
-          <h2>Eyeballer</h2>
-        </div>
-        <PredictButton />
-        <ImageFile />
-
-      </div>
-    );
-  }
-}
-
-export default App;
-
-class ImageFile extends React.Component {
-  constructor(props){
-    super(props);
-    this.state = {
-      id : "someUniqueId", // I would use this.props.id for a real world implementation
-      imageURI : null
-    }
-  }
-
-  buildImgTag(){
-    let imgTag = null;
-    if (this.state.imageURI !== null)
-      imgTag = (<div className="row">
-                  <div className="small-9 small-centered columns">
-                    <img className="thumbnail" id="thumbnail" alt="Screenshot to be classified" src={this.state.imageURI}></img>
-                  </div>
-                </div>);
-    return imgTag;
-  }
-
-  readURI(e){
-    if(e.target.files && e.target.files[0]){
-      let reader = new FileReader();
-      reader.onload = function(ev){
-        this.setState({imageURI:ev.target.result});
-      }.bind(this);
-      reader.readAsDataURL(e.target.files[0]);
-    }
-  }
-
-  handleChange(e){
-    this.readURI(e); // maybe call this with webworker or async library?
-    if (this.props.onChange !== undefined)
-      this.props.onChange(e); // propagate to parent component
-  }
-
-  render() {
-    const imgTag = this.buildImgTag();
-
-    return <div>
-            <label
-              htmlFor={this.state.id}
-              className="button">
-              Upload an image
-            </label>
-            <input
-              id={this.state.id}
-              type="file"
-              onChange={this.handleChange.bind(this)}
-              className="show-for-sr" />
-            {imgTag}
-          </div>;
-  }
-}
-
-class PredictButton extends Component {
-  constructor(props) {
-    super(props);
-    this.state = {
-      classification: ""
-    }
-  }
-
-  onpress = async () => {
-    const modelUrl = 'jsmodel/model.json';
-    const model = await tf.loadLayersModel(modelUrl);
-    const offset = tf.scalar(127.5);
-    const image = tf.browser.fromPixels(document.getElementById('thumbnail'))
-      .resizeNearestNeighbor([224,224])
-      .toFloat()
-      .sub(offset)
-      .div(offset)
-      .expandDims();
-
-    const predictions = Array.from(model.predict(image).dataSync());
-    this.setState({classification: ""});
-    if (predictions[0] > 0.5) {
-      this.setState({classification: this.state.classification + "Custom 404, "});
-    }
-    if (predictions[1] > 0.5) {
-      this.setState({classification: this.state.classification + "Login Page, "});
-    }
-    if (predictions[2] > 0.5) {
-      this.setState({classification: this.state.classification + "Homepage, "});
-    }
-    if (predictions[3] > 0.5) {
-      this.setState({classification: this.state.classification + "Old Looking "});
-    }
-
-  }
-
-  render() {
-    return (
-      <div className="custom-file">
-        <button className="default" type="button" id="predictButton" onClick={this.onpress}>Classify</button>
-        {this.state.classification}
-      </div>
-    );
-  }
-
-}

webui/src/App.test.js

@@ -1,8 +0,0 @@
-import React from 'react';
-import ReactDOM from 'react-dom';
-import App from './App';
-
-it('renders without crashing', () => {
-  const div = document.createElement('div');
-  ReactDOM.render(<App />, div);
-});

webui/src/index.css

@@ -1,5 +0,0 @@
-body {
-  margin: 0;
-  padding: 0;
-  font-family: sans-serif;
-}

webui/src/index.js

@@ -1,9 +0,0 @@
-import React from 'react';
-import ReactDOM from 'react-dom';
-import App from './App';
-import './index.css';
-
-ReactDOM.render(
-  <App />,
-  document.getElementById('root')
-);