More drei updates

* Update checks.py

* Update checks.py

.gitattributes

@@ -3,6 +3,8 @@
 *.md5 text eol=lf
 *.py text eol=lf
 *.xml text eol=lf
+LICENSE text eol=lf
+COMMITMENT text eol=lf
 
 *_ binary
 *.dll binary

.gitignore

@@ -1,6 +1,8 @@
-*.py[cod]
 output/
+__pycache__/
+*.py[cod]
 .sqlmap_history
 traffic.txt
 *~
+req*.txt
 .idea/

README.md

@@ -1,6 +1,6 @@
 # sqlmap
 
-[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://api.travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![PyPI version](https://badge.fury.io/py/sqlmap.svg)](https://badge.fury.io/py/sqlmap) [![GitHub closed issues](https://img.shields.io/github/issues-closed-raw/sqlmapproject/sqlmap.svg?colorB=ff69b4)](https://github.com/sqlmapproject/sqlmap/issues?q=is%3Aissue+is%3Aclosed) [![Twitter](https://img.shields.io/badge/[email protected])](https://twitter.com/sqlmap)
+[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![PyPI version](https://badge.fury.io/py/sqlmap.svg)](https://badge.fury.io/py/sqlmap) [![GitHub closed issues](https://img.shields.io/github/issues-closed-raw/sqlmapproject/sqlmap.svg?colorB=ff69b4)](https://github.com/sqlmapproject/sqlmap/issues?q=is%3Aissue+is%3Aclosed) [![Twitter](https://img.shields.io/badge/[email protected])](https://twitter.com/sqlmap)
 
 sqlmap is an open source penetration testing tool that automates the process of detecting and exploiting SQL injection flaws and taking over of database servers. It comes with a powerful detection engine, many niche features for the ultimate penetration tester and a broad range of switches lasting from database fingerprinting, over data fetching from the database, to accessing the underlying file system and executing commands on the operating system via out-of-band connections.
 

doc/THANKS.md

@@ -764,6 +764,12 @@ ultramegaman, <seclists(at)ultramegaman.com>
 Vinicius, <viniciusmaxdaloop(at)gmail.com>
 * for reporting a minor bug
 
+virusdefender
+* for contributing WAF scripts safeline.py
+
+w8ay
+* for contributing an implementation for chunked transfer-encoding (switch --chunked)
+
 wanglei, <wanglei(at)17uxi.cn>
 * for reporting a minor bug
 

doc/THIRD-PARTY.md

@@ -15,11 +15,6 @@ This file lists bundled packages and their associated licensing terms.
   Copyright (C) 2013, Jonathan Hartley.
 * The Fcrypt library located under thirdparty/fcrypt/.
   Copyright (C) 2000, 2001, 2004 Carey Evans.
-* The Odict library located under thirdparty/odict/.
-  Copyright (C) 2005, Nicola Larosa, Michael Foord.
-* The Oset library located under thirdparty/oset/.
-  Copyright (C) 2010, BlueDynamics Alliance, Austria.
-  Copyright (C) 2009, Raymond Hettinger, and others.
 * The PrettyPrint library located under thirdparty/prettyprint/.
   Copyright (C) 2010, Chris Hall.
 * The SocksiPy library located under thirdparty/socks/.
@@ -281,6 +276,10 @@ be bound by the terms and conditions of this License Agreement.
 
 * The bottle web framework library located under thirdparty/bottle/.
   Copyright (C) 2012, Marcel Hellkamp.
+* The ordereddict library located under thirdparty/odict/.
+  Copyright (C) 2009, Raymond Hettinger.
+* The six Python 2 and 3 compatibility library located under thirdparty/six/.
+  Copyright (C) 2010-2018, Benjamin Peterson.
 * The Termcolor library located under thirdparty/termcolor/.
   Copyright (C) 2008-2011, Volvox Development Team.
 

doc/translations/README-bg-BG.md

@@ -1,6 +1,6 @@
 # sqlmap
 
-[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://api.travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![Лиценз](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![Twitter](https://img.shields.io/badge/[email protected])](https://twitter.com/sqlmap)
+[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![PyPI version](https://badge.fury.io/py/sqlmap.svg)](https://badge.fury.io/py/sqlmap) [![GitHub closed issues](https://img.shields.io/github/issues-closed-raw/sqlmapproject/sqlmap.svg?colorB=ff69b4)](https://github.com/sqlmapproject/sqlmap/issues?q=is%3Aissue+is%3Aclosed) [![Twitter](https://img.shields.io/badge/[email protected])](https://twitter.com/sqlmap)
 
 sqlmap e инструмент за тестване и проникване, с отворен код, който автоматизира процеса на откриване и използване на недостатъците на SQL база данните чрез SQL инжекция, която ги взима от сървъра. Снабден е с мощен детектор, множество специални функции за най-добрия тестер и широк спектър от функции, които могат да се използват за множество цели - извличане на данни от базата данни, достъп до основната файлова система и изпълняване на команди на операционната система.
 

doc/translations/README-es-MX.md

@@ -1,6 +1,6 @@
 # sqlmap
 
-[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://api.travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![Twitter](https://img.shields.io/badge/[email protected])](https://twitter.com/sqlmap)
+[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![PyPI version](https://badge.fury.io/py/sqlmap.svg)](https://badge.fury.io/py/sqlmap) [![GitHub closed issues](https://img.shields.io/github/issues-closed-raw/sqlmapproject/sqlmap.svg?colorB=ff69b4)](https://github.com/sqlmapproject/sqlmap/issues?q=is%3Aissue+is%3Aclosed) [![Twitter](https://img.shields.io/badge/[email protected])](https://twitter.com/sqlmap)
 
 sqlmap es una herramienta para pruebas de penetración "penetration testing" de software libre que automatiza el proceso de detección y explotación de fallos mediante inyección de SQL además de tomar el control de servidores de bases de datos. Contiene un poderoso motor de detección, así como muchas de las funcionalidades escenciales para el "pentester" y una amplia gama de opciones desde la recopilación de información para identificar el objetivo conocido como "fingerprinting" mediante la extracción de información de la base de datos, hasta el acceso al sistema de archivos subyacente para ejecutar comandos en el sistema operativo a través de conexiones alternativas conocidas como "Out-of-band".
 

doc/translations/README-fr-FR.md

@@ -1,6 +1,6 @@
 # sqlmap
 
-[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://api.travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![Twitter](https://img.shields.io/badge/[email protected])](https://twitter.com/sqlmap)
+[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![PyPI version](https://badge.fury.io/py/sqlmap.svg)](https://badge.fury.io/py/sqlmap) [![GitHub closed issues](https://img.shields.io/github/issues-closed-raw/sqlmapproject/sqlmap.svg?colorB=ff69b4)](https://github.com/sqlmapproject/sqlmap/issues?q=is%3Aissue+is%3Aclosed) [![Twitter](https://img.shields.io/badge/[email protected])](https://twitter.com/sqlmap)
 
 **sqlmap** est un outil Open Source de test d'intrusion. Cet outil permet d'automatiser le processus de détection et d'exploitation des failles d'injection SQL afin de prendre le contrôle des serveurs de base de données. __sqlmap__ dispose d'un puissant moteur de détection utilisant les techniques les plus récentes et les plus dévastatrices de tests d'intrusion comme L'Injection SQL, qui permet d'accéder à la base de données, au système de fichiers sous-jacent et permet aussi l'exécution des commandes sur le système d'exploitation.
 

doc/translations/README-gr-GR.md

@@ -1,6 +1,6 @@
 # sqlmap
 
-[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://api.travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![Twitter](https://img.shields.io/badge/[email protected])](https://twitter.com/sqlmap)
+[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![PyPI version](https://badge.fury.io/py/sqlmap.svg)](https://badge.fury.io/py/sqlmap) [![GitHub closed issues](https://img.shields.io/github/issues-closed-raw/sqlmapproject/sqlmap.svg?colorB=ff69b4)](https://github.com/sqlmapproject/sqlmap/issues?q=is%3Aissue+is%3Aclosed) [![Twitter](https://img.shields.io/badge/[email protected])](https://twitter.com/sqlmap)
 
 Το sqlmap είναι πρόγραμμα ανοιχτού κώδικα, που αυτοματοποιεί την εύρεση και εκμετάλλευση ευπαθειών τύπου SQL Injection σε βάσεις δεδομένων. Έρχεται με μια δυνατή μηχανή αναγνώρισης ευπαθειών, πολλά εξειδικευμένα χαρακτηριστικά για τον απόλυτο penetration tester όπως και με ένα μεγάλο εύρος επιλογών αρχίζοντας από την αναγνώριση της βάσης δεδομένων, κατέβασμα δεδομένων της βάσης, μέχρι και πρόσβαση στο βαθύτερο σύστημα αρχείων και εκτέλεση εντολών στο απευθείας στο λειτουργικό μέσω εκτός ζώνης συνδέσεων.
 

doc/translations/README-hr-HR.md

@@ -1,6 +1,6 @@
 # sqlmap
 
-[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://api.travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![Twitter](https://img.shields.io/badge/[email protected])](https://twitter.com/sqlmap)
+[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![PyPI version](https://badge.fury.io/py/sqlmap.svg)](https://badge.fury.io/py/sqlmap) [![GitHub closed issues](https://img.shields.io/github/issues-closed-raw/sqlmapproject/sqlmap.svg?colorB=ff69b4)](https://github.com/sqlmapproject/sqlmap/issues?q=is%3Aissue+is%3Aclosed) [![Twitter](https://img.shields.io/badge/[email protected])](https://twitter.com/sqlmap)
 
 sqlmap je alat namijenjen za penetracijsko testiranje koji automatizira proces detekcije i eksploatacije sigurnosnih propusta SQL injekcije te preuzimanje poslužitelja baze podataka. Dolazi s moćnim mehanizmom za detekciju, mnoštvom korisnih opcija za napredno penetracijsko testiranje te široki spektar opcija od onih za prepoznavanja baze podataka, preko dohvaćanja podataka iz baze, do pristupa zahvaćenom datotečnom sustavu i izvršavanja komandi na operacijskom sustavu korištenjem tzv. "out-of-band" veza.
 

doc/translations/README-id-ID.md

@@ -1,6 +1,6 @@
 # sqlmap
 
-[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://api.travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![Twitter](https://img.shields.io/badge/[email protected])](https://twitter.com/sqlmap)
+[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![PyPI version](https://badge.fury.io/py/sqlmap.svg)](https://badge.fury.io/py/sqlmap) [![GitHub closed issues](https://img.shields.io/github/issues-closed-raw/sqlmapproject/sqlmap.svg?colorB=ff69b4)](https://github.com/sqlmapproject/sqlmap/issues?q=is%3Aissue+is%3Aclosed) [![Twitter](https://img.shields.io/badge/[email protected])](https://twitter.com/sqlmap)
 
 sqlmap merupakan alat _(tool)_ bantu _open source_ dalam melakukan tes penetrasi yang mengotomasi proses deteksi dan eksploitasi kelemahan _SQL injection_ dan pengambil-alihan server basisdata. sqlmap dilengkapi dengan pendeteksi canggih, fitur-fitur hanal bagi _penetration tester_, beragam cara untuk mendeteksi basisdata, hingga mengakses _file system_ dan mengeksekusi perintah dalam sistem operasi melalui koneksi _out-of-band_. 
 

doc/translations/README-it-IT.md

@@ -1,6 +1,6 @@
 # sqlmap
 
-[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://api.travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![Twitter](https://img.shields.io/badge/[email protected])](https://twitter.com/sqlmap)
+[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![PyPI version](https://badge.fury.io/py/sqlmap.svg)](https://badge.fury.io/py/sqlmap) [![GitHub closed issues](https://img.shields.io/github/issues-closed-raw/sqlmapproject/sqlmap.svg?colorB=ff69b4)](https://github.com/sqlmapproject/sqlmap/issues?q=is%3Aissue+is%3Aclosed) [![Twitter](https://img.shields.io/badge/[email protected])](https://twitter.com/sqlmap)
 
 sqlmap è uno strumento open source per il penetration testing. Il suo scopo è quello di rendere automatico il processo di scoperta ed exploit di vulnerabilità di tipo SQL injection al fine di compromettere database online. Dispone di un potente motore per la ricerca di vulnerabilità, molti strumenti di nicchia anche per il più esperto penetration tester ed un'ampia gamma di controlli che vanno dal fingerprinting di database allo scaricamento di dati, fino all'accesso al file system sottostante e l'esecuzione di comandi nel sistema operativo attraverso connessioni out-of-band.
 

doc/translations/README-ja-JP.md

@@ -1,6 +1,6 @@
 # sqlmap
 
-[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://api.travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![Twitter](https://img.shields.io/badge/[email protected])](https://twitter.com/sqlmap)
+[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![PyPI version](https://badge.fury.io/py/sqlmap.svg)](https://badge.fury.io/py/sqlmap) [![GitHub closed issues](https://img.shields.io/github/issues-closed-raw/sqlmapproject/sqlmap.svg?colorB=ff69b4)](https://github.com/sqlmapproject/sqlmap/issues?q=is%3Aissue+is%3Aclosed) [![Twitter](https://img.shields.io/badge/[email protected])](https://twitter.com/sqlmap)
 
 sqlmapはオープンソースのペネトレーションテスティングツールです。SQLインジェクションの脆弱性の検出、活用、そしてデータベースサーバ奪取のプロセスを自動化します。
 強力な検出エンジン、ペネトレーションテスターのための多くのニッチ機能、持続的なデータベースのフィンガープリンティングから、データベースのデータ取得やアウトオブバンド接続を介したオペレーティング・システム上でのコマンド実行、ファイルシステムへのアクセスなどの広範囲に及ぶスイッチを提供します。

doc/translations/README-pl-PL.md

@@ -1,6 +1,6 @@
 # sqlmap
 
-[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://api.travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![Twitter](https://img.shields.io/badge/[email protected])](https://twitter.com/sqlmap)
+[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![PyPI version](https://badge.fury.io/py/sqlmap.svg)](https://badge.fury.io/py/sqlmap) [![GitHub closed issues](https://img.shields.io/github/issues-closed-raw/sqlmapproject/sqlmap.svg?colorB=ff69b4)](https://github.com/sqlmapproject/sqlmap/issues?q=is%3Aissue+is%3Aclosed) [![Twitter](https://img.shields.io/badge/[email protected])](https://twitter.com/sqlmap)
 
 sqlmap to open sourceowe narzędzie do testów penetracyjnych, które automatyzuje procesy detekcji, przejmowania i testowania odporności serwerów SQL na podatność na iniekcję niechcianego kodu. Zawiera potężny mechanizm detekcji, wiele niszowych funkcji dla zaawansowanych testów penetracyjnych oraz szeroki wachlarz opcji począwszy od identyfikacji bazy danych, poprzez wydobywanie z nich danych, a nawet pozwalającuch na dostęp do systemu plików o uruchamianie poleceń w systemie operacyjnym serwera poprzez niestandardowe połączenia.
 

doc/translations/README-pt-BR.md

@@ -1,6 +1,6 @@
 # sqlmap
 
-[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://api.travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![Twitter](https://img.shields.io/badge/[email protected])](https://twitter.com/sqlmap)
+[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![PyPI version](https://badge.fury.io/py/sqlmap.svg)](https://badge.fury.io/py/sqlmap) [![GitHub closed issues](https://img.shields.io/github/issues-closed-raw/sqlmapproject/sqlmap.svg?colorB=ff69b4)](https://github.com/sqlmapproject/sqlmap/issues?q=is%3Aissue+is%3Aclosed) [![Twitter](https://img.shields.io/badge/[email protected])](https://twitter.com/sqlmap)
 
 sqlmap é uma ferramenta de teste de penetração de código aberto que automatiza o processo de detecção e exploração de falhas de injeção SQL. Com essa ferramenta é possível assumir total controle de servidores de banco de dados em páginas web vulneráveis, inclusive de base de dados fora do sistema invadido. Ele possui um motor de detecção poderoso, empregando as últimas e mais devastadoras técnicas de teste de penetração por SQL Injection, que permite acessar a base de dados, o sistema de arquivos subjacente e executar comandos no sistema operacional.
 

doc/translations/README-ru-RUS.md

@@ -1,6 +1,6 @@
 # sqlmap
 
-[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://api.travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![Twitter](https://img.shields.io/badge/[email protected])](https://twitter.com/sqlmap)
+[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![PyPI version](https://badge.fury.io/py/sqlmap.svg)](https://badge.fury.io/py/sqlmap) [![GitHub closed issues](https://img.shields.io/github/issues-closed-raw/sqlmapproject/sqlmap.svg?colorB=ff69b4)](https://github.com/sqlmapproject/sqlmap/issues?q=is%3Aissue+is%3Aclosed) [![Twitter](https://img.shields.io/badge/[email protected])](https://twitter.com/sqlmap)
 
 sqlmap - это инструмент для тестирования уязвимостей с открытым исходным кодом, который автоматизирует процесс обнаружения и использования ошибок SQL-инъекций и захвата серверов баз данных. Он оснащен мощным механизмом обнаружения, множеством приятных функций для профессионального тестера уязвимостей и широким спектром скриптов, которые упрощают работу с базами данных, от сбора данных из базы данных, до доступа к базовой файловой системе и выполнения команд в операционной системе через out-of-band соединение.
 

doc/translations/README-tr-TR.md

@@ -1,6 +1,6 @@
 # sqlmap
 
-[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://api.travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![Twitter](https://img.shields.io/badge/[email protected])](https://twitter.com/sqlmap)
+[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![PyPI version](https://badge.fury.io/py/sqlmap.svg)](https://badge.fury.io/py/sqlmap) [![GitHub closed issues](https://img.shields.io/github/issues-closed-raw/sqlmapproject/sqlmap.svg?colorB=ff69b4)](https://github.com/sqlmapproject/sqlmap/issues?q=is%3Aissue+is%3Aclosed) [![Twitter](https://img.shields.io/badge/[email protected])](https://twitter.com/sqlmap)
 
 sqlmap sql injection açıklarını otomatik olarak tespit ve istismar etmeye yarayan açık kaynak bir penetrasyon aracıdır. sqlmap gelişmiş tespit özelliğinin yanı sıra penetrasyon testleri sırasında gerekli olabilecek bir çok aracı, -uzak veritabınınından, veri indirmek, dosya sistemine erişmek, dosya çalıştırmak gibi - işlevleri de barındırmaktadır.
 

doc/translations/README-uk-UA.md

@@ -1,6 +1,6 @@
 # sqlmap
 
-[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://api.travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![PyPI version](https://badge.fury.io/py/sqlmap.svg)](https://badge.fury.io/py/sqlmap) [![GitHub closed issues](https://img.shields.io/github/issues-closed-raw/sqlmapproject/sqlmap.svg?colorB=ff69b4)](https://github.com/sqlmapproject/sqlmap/issues?q=is%3Aissue+is%3Aclosed) [![Twitter](https://img.shields.io/badge/[email protected])](https://twitter.com/sqlmap)
+[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![PyPI version](https://badge.fury.io/py/sqlmap.svg)](https://badge.fury.io/py/sqlmap) [![GitHub closed issues](https://img.shields.io/github/issues-closed-raw/sqlmapproject/sqlmap.svg?colorB=ff69b4)](https://github.com/sqlmapproject/sqlmap/issues?q=is%3Aissue+is%3Aclosed) [![Twitter](https://img.shields.io/badge/[email protected])](https://twitter.com/sqlmap)
 
 sqlmap - це інструмент для тестування вразливостей з відкритим сирцевим кодом, який автоматизує процес виявлення і використання дефектів SQL-ін'єкцій, а також захоплення серверів баз даних. Він оснащений потужним механізмом виявлення, безліччю приємних функцій для професійного тестувальника вразливостей і широким спектром скриптів, які спрощують роботу з базами даних - від відбитка бази даних до доступу до базової файлової системи та виконання команд в операційній системі через out-of-band з'єднання.
 

doc/translations/README-zh-CN.md

@@ -1,6 +1,6 @@
 # sqlmap
 
-[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://api.travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![Twitter](https://img.shields.io/badge/[email protected])](https://twitter.com/sqlmap)
+[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![PyPI version](https://badge.fury.io/py/sqlmap.svg)](https://badge.fury.io/py/sqlmap) [![GitHub closed issues](https://img.shields.io/github/issues-closed-raw/sqlmapproject/sqlmap.svg?colorB=ff69b4)](https://github.com/sqlmapproject/sqlmap/issues?q=is%3Aissue+is%3Aclosed) [![Twitter](https://img.shields.io/badge/[email protected])](https://twitter.com/sqlmap)
 
 sqlmap 是一个开源的渗透测试工具，可以用来自动化的检测，利用SQL注入漏洞，获取数据库服务器的权限。它具有功能强大的检测引擎,针对各种不同类型数据库的渗透测试的功能选项，包括获取数据库中存储的数据，访问操作系统文件甚至可以通过外带数据连接的方式执行操作系统命令。
 

extra/__init__.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)

extra/beep/beep.py

@@ -8,7 +8,6 @@ See the file 'LICENSE' for copying permission
 """
 
 import os
-import subprocess
 import sys
 import wave
 
@@ -16,11 +15,11 @@ BEEP_WAV_FILENAME = os.path.join(os.path.dirname(__file__), "beep.wav")
 
 def beep():
     try:
-        if subprocess.mswindows:
+        if sys.platform == "nt":
             _win_wav_play(BEEP_WAV_FILENAME)
         elif sys.platform == "darwin":
             _mac_beep()
-        elif sys.platform == "linux2":
+        elif sys.platform.startswith("linux"):
             _linux_wav_play(BEEP_WAV_FILENAME)
         else:
             _speaker_beep()

extra/cloak/cloak.py

@@ -10,19 +10,21 @@ See the file 'LICENSE' for copying permission
 from __future__ import print_function
 
 import os
+import struct
 import sys
 import zlib
 
 from optparse import OptionError
 from optparse import OptionParser
 
+if sys.version_info >= (3, 0):
+    xrange = range
+
 def hideAscii(data):
-    retVal = ""
+    retVal = b""
     for i in xrange(len(data)):
-        if ord(data[i]) < 128:
+        value = data[i] if isinstance(data[i], int) else ord(data[i])
-            retVal += chr(ord(data[i]) ^ 127)
+        retVal += struct.pack('B', value ^ (127 if value < 128 else 0))
-        else:
-            retVal += data[i]
 
     return retVal
 
@@ -39,7 +41,8 @@ def decloak(inputFile=None, data=None):
             data = f.read()
     try:
         data = zlib.decompress(hideAscii(data))
-    except:
+    except Exception as ex:
+        print(ex)
         print('ERROR: the provided input file \'%s\' does not contain valid cloaked content' % inputFile)
         sys.exit(1)
     finally:

extra/dbgtool/dbgtool.py

@@ -11,7 +11,6 @@ from __future__ import print_function
 
 import os
 import sys
-import struct
 
 from optparse import OptionError
 from optparse import OptionParser
@@ -34,7 +33,7 @@ def convert(inputFile):
     fileContent = fp.read()
 
     for fileChar in fileContent:
-        unsignedFileChar = struct.unpack("B", fileChar)[0]
+        unsignedFileChar = fileChar if sys.version_info >= (3, 0) else ord(fileChar)
 
         if unsignedFileChar != 0:
             counter2 += 1

extra/icmpsh/icmpsh_m.py

@@ -22,7 +22,6 @@
 import os
 import select
 import socket
-import subprocess
 import sys
 
 def setNonBlocking(fd):
@@ -37,7 +36,7 @@ def setNonBlocking(fd):
     fcntl.fcntl(fd, fcntl.F_SETFL, flags)
 
 def main(src, dst):
-    if subprocess.mswindows:
+    if sys.platform == "nt":
         sys.stderr.write('icmpsh master can only run on Posix systems\n')
         sys.exit(255)
 

extra/safe2bin/__init__.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)

extra/safe2bin/safe2bin.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 safe2bin.py - Simple safe(hex) to binary format converter
@@ -18,6 +18,14 @@ import sys
 from optparse import OptionError
 from optparse import OptionParser
 
+if sys.version_info >= (3, 0):
+    xrange = range
+    text_type = str
+    string_types = (str,)
+else:
+    text_type = unicode
+    string_types = (basestring,)
+
 # Regex used for recognition of hex encoded characters
 HEX_ENCODED_CHAR_REGEX = r"(?P<result>\\x[0-9A-Fa-f]{2})"
 
@@ -48,15 +56,15 @@ def safecharencode(value):
 
     retVal = value
 
-    if isinstance(value, basestring):
+    if isinstance(value, string_types):
-        if any([_ not in SAFE_CHARS for _ in value]):
+        if any(_ not in SAFE_CHARS for _ in value):
             retVal = retVal.replace(HEX_ENCODED_PREFIX, HEX_ENCODED_PREFIX_MARKER)
             retVal = retVal.replace('\\', SLASH_MARKER)
 
             for char in SAFE_ENCODE_SLASH_REPLACEMENTS:
                 retVal = retVal.replace(char, repr(char).strip('\''))
 
-            retVal = reduce(lambda x, y: x + (y if (y in string.printable or isinstance(value, unicode) and ord(y) >= 160) else '\\x%02x' % ord(y)), retVal, (unicode if isinstance(value, unicode) else str)())
+            retVal = reduce(lambda x, y: x + (y if (y in string.printable or isinstance(value, text_type) and ord(y) >= 160) else '\\x%02x' % ord(y)), retVal, type(value)())
 
             retVal = retVal.replace(SLASH_MARKER, "\\\\")
             retVal = retVal.replace(HEX_ENCODED_PREFIX_MARKER, HEX_ENCODED_PREFIX)
@@ -72,13 +80,13 @@ def safechardecode(value, binary=False):
     """
 
     retVal = value
-    if isinstance(value, basestring):
+    if isinstance(value, string_types):
         retVal = retVal.replace('\\\\', SLASH_MARKER)
 
         while True:
             match = re.search(HEX_ENCODED_CHAR_REGEX, retVal)
             if match:
-                retVal = retVal.replace(match.group("result"), (unichr if isinstance(value, unicode) else chr)(ord(binascii.unhexlify(match.group("result").lstrip("\\x")))))
+                retVal = retVal.replace(match.group("result"), (unichr if isinstance(value, text_type) else chr)(ord(binascii.unhexlify(match.group("result").lstrip("\\x")))))
             else:
                 break
 
@@ -88,7 +96,7 @@ def safechardecode(value, binary=False):
         retVal = retVal.replace(SLASH_MARKER, '\\')
 
         if binary:
-            if isinstance(retVal, unicode):
+            if isinstance(retVal, text_type):
                 retVal = retVal.encode("utf8")
 
     elif isinstance(value, (list, tuple)):

extra/shutils/blanks.sh

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-# Copyright (c) 2006-2013 sqlmap developers (http://sqlmap.org/)
+# Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 # See the file 'LICENSE' for copying permission
 
 # Removes trailing spaces from blank lines inside project files

extra/shutils/drei.sh

@@ -0,0 +1,11 @@
+#!/bin/bash
+
+# Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
+# See the file 'LICENSE' for copying permission
+
+# Stress test against Python3
+
+export SQLMAP_DREI=1
+for i in $(find . -iname "*.py" | grep -v __init__); do python3 -c 'import '`echo $i | cut -d '.' -f 2 | cut -d '/' -f 2- | sed 's/\//./g'`''; done
+unset SQLMAP_DREI
+source `dirname "$0"`"/junk.sh"

extra/shutils/duplicates.py

@@ -9,21 +9,22 @@ from __future__ import print_function
 
 import sys
 
-if len(sys.argv) > 0:
+if __name__ == "__main__":
-    items = list()
+    if len(sys.argv) > 1:
+        items = list()
 
-    with open(sys.argv[1], 'r') as f:
+        with open(sys.argv[1], 'r') as f:
-        for item in f.readlines():
+            for item in f.readlines():
-            item = item.strip()
+                item = item.strip()
-            try:
+                try:
-                str.encode(item)
+                    str.encode(item)
-                if item in items:
+                    if item in items:
-                    if item:
+                        if item:
-                        print(item)
+                            print(item)
-                else:
+                    else:
-                    items.append(item)
+                        items.append(item)
-            except:
+                except:
-                pass
+                    pass
 
-    with open(sys.argv[1], 'w+') as f:
+        with open(sys.argv[1], 'w+') as f:
-        f.writelines("\n".join(items))
+            f.writelines("\n".join(items))

extra/shutils/junk.sh

@@ -0,0 +1,7 @@
+#!/bin/bash
+
+# Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
+# See the file 'LICENSE' for copying permission
+
+find . -type d -name "__pycache__" -exec rm -rf {} \; &>/dev/null
+find . -name "*.pyc" -exec rm -f {} \; &>/dev/null

extra/shutils/newlines.py

@@ -1,8 +1,5 @@
 #! /usr/bin/env python
 
-# Runs pylint on all python scripts found in a directory tree
-# Reference: http://rowinggolfer.blogspot.com/2009/08/pylint-recursively.html
-
 from __future__ import print_function
 
 import os
@@ -11,9 +8,10 @@ import sys
 def check(filepath):
     if filepath.endswith(".py"):
         content = open(filepath, "rb").read()
+        pattern = "\n\n\n".encode("ascii")
 
-        if "\n\n\n" in content:
+        if pattern in content:
-            index = content.find("\n\n\n")
+            index = content.find(pattern)
             print(filepath, repr(content[index - 30:index + 30]))
 
 if __name__ == "__main__":

extra/shutils/postcommit-hook.sh

@@ -11,6 +11,7 @@ chmod +x .git/hooks/post-commit
 '
 
 SETTINGS="../../lib/core/settings.py"
+PYPI="../../extra/shutils/pypi.sh"
 
 declare -x SCRIPTPATH="${0}"
 
@@ -28,6 +29,6 @@ then
         git tag $NEW_TAG
         git push origin $NEW_TAG
         echo "Going to push PyPI package"
-        /bin/bash ${SCRIPTPATH%/*}/pypi.sh
+        /bin/bash ${SCRIPTPATH%/*}/$PYPI
     fi
 fi

extra/shutils/precommit-hook.sh

@@ -12,13 +12,11 @@ chmod +x .git/hooks/pre-commit
 
 PROJECT="../../"
 SETTINGS="../../lib/core/settings.py"
-CHECKSUM="../../txt/checksum.md5"
 
 declare -x SCRIPTPATH="${0}"
 
 PROJECT_FULLPATH=${SCRIPTPATH%/*}/$PROJECT
 SETTINGS_FULLPATH=${SCRIPTPATH%/*}/$SETTINGS
-CHECKSUM_FULLPATH=${SCRIPTPATH%/*}/$CHECKSUM
 
 git diff $SETTINGS_FULLPATH | grep "VERSION =" > /dev/null && exit 0
 
@@ -37,6 +35,3 @@ then
     fi
     git add "$SETTINGS_FULLPATH"
 fi
-
-truncate -s 0 "$CHECKSUM_FULLPATH"
-cd $PROJECT_FULLPATH && for i in $(find . -name "*.py" -o -name "*.xml" -o -name "*_" -o -type f -regex "./[^./]*" | sort); do git ls-files $i --error-unmatch &>/dev/null && md5sum $i | stdbuf -i0 -o0 -e0 sed 's/\.\///' >> "$CHECKSUM_FULLPATH"; git add "$CHECKSUM_FULLPATH"; done

extra/shutils/pydiatra.sh

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-# Copyright (c) 2006-2013 sqlmap developers (http://sqlmap.org/)
+# Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 # See the file 'LICENSE' for copying permission
 
 # Runs py2diatra on all python files (prerequisite: pip install pydiatra)

extra/shutils/pyflakes.sh

@@ -1,7 +1,7 @@
 #!/bin/bash
 
-# Copyright (c) 2006-2013 sqlmap developers (http://sqlmap.org/)
+# Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 # See the file 'LICENSE' for copying permission
 
 # Runs pyflakes on all python files (prerequisite: apt-get install pyflakes)
-find . -wholename "./thirdparty" -prune -o -type f -iname "*.py" -exec pyflakes '{}' \;
+find . -wholename "./thirdparty" -prune -o -type f -iname "*.py" -exec pyflakes '{}' \; | grep -v "redefines '_'"

extra/shutils/pypi.sh

@@ -26,7 +26,8 @@ setup(
     name='sqlmap',
     version='$VERSION',
     description='Automatic SQL injection and database takeover tool',
-    long_description='sqlmap is an open source penetration testing tool that automates the process of detecting and exploiting SQL injection flaws and taking over of database servers. It comes with a powerful detection engine, many niche features for the ultimate penetration tester and a broad range of switches lasting from database fingerprinting, over data fetching from the database, to accessing the underlying file system and executing commands on the operating system via out-of-band connections.',
+    long_description=open('README.rst').read(),
+    long_description_content_type='text/x-rst',
     author='Bernardo Damele Assumpcao Guimaraes, Miroslav Stampar',
     author_email='[email protected], [email protected]',
     url='http://sqlmap.org',
@@ -158,7 +159,7 @@ Links
 -  User's manual: https://github.com/sqlmapproject/sqlmap/wiki
 -  Frequently Asked Questions (FAQ):
    https://github.com/sqlmapproject/sqlmap/wiki/FAQ
--  Twitter: [@sqlmap](https://twitter.com/sqlmap)
+-  Twitter: https://twitter.com/sqlmap
 -  Demos: http://www.youtube.com/user/inquisb/videos
 -  Screenshots: https://github.com/sqlmapproject/sqlmap/wiki/Screenshots
 
@@ -176,7 +177,6 @@ Links
 EOF
 sed -i "s/^VERSION =.*/VERSION = \"$VERSION\"/g" sqlmap/lib/core/settings.py
 sed -i "s/^TYPE =.*/TYPE = \"$TYPE\"/g" sqlmap/lib/core/settings.py
-sed -i "s/.*lib\/core\/settings\.py/`md5sum sqlmap/lib/core/settings.py | cut -d ' ' -f 1`  lib\/core\/settings\.py/g" sqlmap/txt/checksum.md5
 for file in $(find sqlmap -type f | grep -v -E "\.(git|yml)"); do echo include $file >> MANIFEST.in; done
 python setup.py sdist upload
 rm -rf $TMP_DIR

extra/shutils/regressiontest.py

@@ -1,166 +0,0 @@
-#!/usr/bin/env python
-
-# Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
-# See the file 'LICENSE' for copying permission
-
-from __future__ import print_function
-
-import codecs
-import inspect
-import os
-import re
-import smtplib
-import subprocess
-import sys
-import time
-import traceback
-
-from email.mime.multipart import MIMEMultipart
-from email.mime.text import MIMEText
-
-sys.path.append(os.path.normpath("%s/../../" % os.path.dirname(inspect.getfile(inspect.currentframe()))))
-
-from lib.core.revision import getRevisionNumber
-
-START_TIME = time.strftime("%H:%M:%S %d-%m-%Y", time.gmtime())
-SQLMAP_HOME = "/opt/sqlmap"
-
-SMTP_SERVER = "127.0.0.1"
-SMTP_PORT = 25
-SMTP_TIMEOUT = 30
-FROM = "[email protected]"
-# TO = "[email protected]"
-TO = ["[email protected]", "[email protected]"]
-SUBJECT = "regression test started on %s using revision %s" % (START_TIME, getRevisionNumber())
-TARGET = "debian"
-
-def prepare_email(content):
-    global FROM
-    global TO
-    global SUBJECT
-
-    msg = MIMEMultipart()
-    msg["Subject"] = SUBJECT
-    msg["From"] = FROM
-    msg["To"] = TO if isinstance(TO, basestring) else ','.join(TO)
-
-    msg.attach(MIMEText(content))
-
-    return msg
-
-def send_email(msg):
-    global SMTP_SERVER
-    global SMTP_PORT
-    global SMTP_TIMEOUT
-
-    try:
-        s = smtplib.SMTP(host=SMTP_SERVER, port=SMTP_PORT, timeout=SMTP_TIMEOUT)
-        s.sendmail(FROM, TO, msg.as_string())
-        s.quit()
-    # Catch all for SMTP exceptions
-    except smtplib.SMTPException as ex:
-        print("Failure to send email: '%s" % ex)
-
-def failure_email(msg):
-    msg = prepare_email(msg)
-    send_email(msg)
-    sys.exit(1)
-
-def main():
-    global SUBJECT
-
-    content = ""
-    test_counts = []
-    attachments = {}
-
-    updateproc = subprocess.Popen("cd /opt/sqlmap/ ; python /opt/sqlmap/sqlmap.py --update", shell=True, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
-    stdout, stderr = updateproc.communicate()
-
-    if stderr:
-        failure_email("Update of sqlmap failed with error:\n\n%s" % stderr)
-
-    regressionproc = subprocess.Popen("python /opt/sqlmap/sqlmap.py --live-test", shell=True, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, close_fds=False)
-    stdout, stderr = regressionproc.communicate()
-
-    if stderr:
-        failure_email("Execution of regression test failed with error:\n\n%s" % stderr)
-
-    failed_tests = re.findall(r"running live test case: (.+?) \((\d+)\/\d+\)[\r]*\n.+test failed (at parsing items: (.+))?\s*\- scan folder: (\/.+) \- traceback: (.*?)( - SQL injection not detected)?[\r]*\n", stdout)
-
-    for failed_test in failed_tests:
-        title = failed_test[0]
-        test_count = int(failed_test[1])
-        parse = failed_test[3] if failed_test[3] else None
-        output_folder = failed_test[4]
-        traceback = False if failed_test[5] == "False" else bool(failed_test[5])
-        detected = False if failed_test[6] else True
-
-        test_counts.append(test_count)
-
-        console_output_file = os.path.join(output_folder, "console_output")
-        log_file = os.path.join(output_folder, TARGET, "log")
-        traceback_file = os.path.join(output_folder, "traceback")
-
-        if os.path.exists(console_output_file):
-            console_output_fd = codecs.open(console_output_file, "rb", "utf8")
-            console_output = console_output_fd.read()
-            console_output_fd.close()
-            attachments[test_count] = str(console_output)
-
-        if os.path.exists(log_file):
-            log_fd = codecs.open(log_file, "rb", "utf8")
-            log = log_fd.read()
-            log_fd.close()
-
-        if os.path.exists(traceback_file):
-            traceback_fd = codecs.open(traceback_file, "rb", "utf8")
-            traceback = traceback_fd.read()
-            traceback_fd.close()
-
-        content += "Failed test case '%s' (#%d)" % (title, test_count)
-
-        if parse:
-            content += " at parsing: %s:\n\n" % parse
-            content += "### Log file:\n\n"
-            content += "%s\n\n" % log
-        elif not detected:
-            content += " - SQL injection not detected\n\n"
-        else:
-            content += "\n\n"
-
-        if traceback:
-            content += "### Traceback:\n\n"
-            content += "%s\n\n" % str(traceback)
-
-        content += "#######################################################################\n\n"
-
-    end_string = "Regression test finished at %s" % time.strftime("%H:%M:%S %d-%m-%Y", time.gmtime())
-
-    if content:
-        content += end_string
-        SUBJECT = "Failed %s (%s)" % (SUBJECT, ", ".join("#%d" % count for count in test_counts))
-
-        msg = prepare_email(content)
-
-        for test_count, attachment in attachments.items():
-            attachment = MIMEText(attachment)
-            attachment.add_header("Content-Disposition", "attachment", filename="test_case_%d_console_output.txt" % test_count)
-            msg.attach(attachment)
-
-        send_email(msg)
-    else:
-        SUBJECT = "Successful %s" % SUBJECT
-        msg = prepare_email("All test cases were successful\n\n%s" % end_string)
-        send_email(msg)
-
-if __name__ == "__main__":
-    log_fd = open("/tmp/sqlmapregressiontest.log", "wb")
-    log_fd.write("Regression test started at %s\n" % START_TIME)
-
-    try:
-        main()
-    except Exception:
-        log_fd.write("An exception has occurred:\n%s" % str(traceback.format_exc()))
-
-    log_fd.write("Regression test finished at %s\n\n" % time.strftime("%H:%M:%S %d-%m-%Y", time.gmtime()))
-    log_fd.close()

extra/sqlharvest/sqlharvest.py

@@ -1,143 +0,0 @@
-#!/usr/bin/env python
-
-"""
-Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
-See the file 'LICENSE' for copying permission
-"""
-
-from __future__ import print_function
-
-import cookielib
-import re
-import socket
-import sys
-import urllib
-import urllib2
-import ConfigParser
-
-from operator import itemgetter
-
-TIMEOUT = 10
-CONFIG_FILE = 'sqlharvest.cfg'
-TABLES_FILE = 'tables.txt'
-USER_AGENT = 'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; AskTB5.3)'
-SEARCH_URL = 'http://www.google.com/m?source=mobileproducts&dc=gorganic'
-MAX_FILE_SIZE = 2 * 1024 * 1024  # if a result (.sql) file for downloading is more than 2MB in size just skip it
-QUERY = 'CREATE TABLE ext:sql'
-REGEX_URLS = r';u=([^"]+?)&q='
-REGEX_RESULT = r'(?i)CREATE TABLE\s*(/\*.*\*/)?\s*(IF NOT EXISTS)?\s*(?P<result>[^\(;]+)'
-
-def main():
-    tables = dict()
-    cookies = cookielib.CookieJar()
-    cookie_processor = urllib2.HTTPCookieProcessor(cookies)
-    opener = urllib2.build_opener(cookie_processor)
-    opener.addheaders = [("User-Agent", USER_AGENT)]
-
-    conn = opener.open(SEARCH_URL)
-    page = conn.read()  # set initial cookie values
-
-    config = ConfigParser.ConfigParser()
-    config.read(CONFIG_FILE)
-
-    if not config.has_section("options"):
-        config.add_section("options")
-    if not config.has_option("options", "index"):
-        config.set("options", "index", "0")
-
-    i = int(config.get("options", "index"))
-
-    try:
-        with open(TABLES_FILE, 'r') as f:
-            for line in f.xreadlines():
-                if len(line) > 0 and ',' in line:
-                    temp = line.split(',')
-                    tables[temp[0]] = int(temp[1])
-    except:
-        pass
-
-    socket.setdefaulttimeout(TIMEOUT)
-
-    files, old_files = None, None
-    try:
-        while True:
-            abort = False
-            old_files = files
-            files = []
-
-            try:
-                conn = opener.open("%s&q=%s&start=%d&sa=N" % (SEARCH_URL, QUERY.replace(' ', '+'), i * 10))
-                page = conn.read()
-                for match in re.finditer(REGEX_URLS, page):
-                    files.append(urllib.unquote(match.group(1)))
-                    if len(files) >= 10:
-                        break
-                abort = (files == old_files)
-
-            except KeyboardInterrupt:
-                raise
-
-            except Exception as ex:
-                print(ex)
-
-            if abort:
-                break
-
-            sys.stdout.write("\n---------------\n")
-            sys.stdout.write("Result page #%d\n" % (i + 1))
-            sys.stdout.write("---------------\n")
-
-            for sqlfile in files:
-                print(sqlfile)
-
-                try:
-                    req = urllib2.Request(sqlfile)
-                    response = urllib2.urlopen(req)
-
-                    if "Content-Length" in response.headers:
-                        if int(response.headers.get("Content-Length")) > MAX_FILE_SIZE:
-                            continue
-
-                    page = response.read()
-                    found = False
-                    counter = 0
-
-                    for match in re.finditer(REGEX_RESULT, page):
-                        counter += 1
-                        table = match.group("result").strip().strip("`\"'").replace('"."', ".").replace("].[", ".").strip('[]')
-
-                        if table and not any(_ in table for _ in ('>', '<', '--', ' ')):
-                            found = True
-                            sys.stdout.write('*')
-
-                            if table in tables:
-                                tables[table] += 1
-                            else:
-                                tables[table] = 1
-                    if found:
-                        sys.stdout.write("\n")
-
-                except KeyboardInterrupt:
-                    raise
-
-                except Exception as ex:
-                    print(ex)
-
-            else:
-                i += 1
-
-    except KeyboardInterrupt:
-        pass
-
-    finally:
-        with open(TABLES_FILE, 'w+') as f:
-            tables = sorted(tables.items(), key=itemgetter(1), reverse=True)
-            for table, count in tables:
-                f.write("%s,%d\n" % (table, count))
-
-        config.set("options", "index", str(i + 1))
-        with open(CONFIG_FILE, 'w+') as f:
-            config.write(f)
-
-if __name__ == "__main__":
-    main()

extra/vulnserver/vulnserver.py

@@ -0,0 +1,174 @@
+#!/usr/bin/env python
+
+"""
+vulnserver.py - Trivial SQLi vulnerable HTTP server (Note: for testing purposes)
+
+Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
+See the file 'LICENSE' for copying permission
+"""
+
+from __future__ import print_function
+
+import re
+import sqlite3
+import sys
+import threading
+import traceback
+
+if sys.version_info >= (3, 0):
+    from http.client import FOUND
+    from http.client import INTERNAL_SERVER_ERROR
+    from http.client import NOT_FOUND
+    from http.client import OK
+    from http.server import BaseHTTPRequestHandler
+    from http.server import HTTPServer
+    from socketserver import ThreadingMixIn
+    from urllib.parse import parse_qs
+    from urllib.parse import unquote_plus
+else:
+    from BaseHTTPServer import BaseHTTPRequestHandler
+    from BaseHTTPServer import HTTPServer
+    from httplib import FOUND
+    from httplib import INTERNAL_SERVER_ERROR
+    from httplib import NOT_FOUND
+    from httplib import OK
+    from SocketServer import ThreadingMixIn
+    from urlparse import parse_qs
+    from urllib import unquote_plus
+
+SCHEMA = """
+    CREATE TABLE users (
+        id INTEGER,
+        name TEXT,
+        surname TEXT
+    );
+    INSERT INTO users (id, name, surname) VALUES (1, 'luther', 'blisset');
+    INSERT INTO users (id, name, surname) VALUES (2, 'fluffy', 'bunny');
+    INSERT INTO users (id, name, surname) VALUES (3, 'wu', '179ad45c6ce2cb97cf1029e212046e81');
+    INSERT INTO users (id, name, surname) VALUES (4, 'sqlmap/1.0-dev (http://sqlmap.org)', 'user agent header');
+    INSERT INTO users (id, name, surname) VALUES (5, NULL, 'nameisnull');
+"""
+
+LISTEN_ADDRESS = "localhost"
+LISTEN_PORT = 8440
+
+_conn = None
+_cursor = None
+_lock = None
+_server = None
+
+def init(quiet=False):
+    global _conn
+    global _cursor
+    global _lock
+
+    _conn = sqlite3.connect(":memory:", isolation_level=None, check_same_thread=False)
+    _cursor = _conn.cursor()
+    _lock = threading.Lock()
+
+    _cursor.executescript(SCHEMA)
+
+    if quiet:
+        global print
+
+        def _(*args, **kwargs):
+            pass
+
+        print = _
+
+class ThreadingServer(ThreadingMixIn, HTTPServer):
+    def finish_request(self, *args, **kwargs):
+        try:
+            HTTPServer.finish_request(self, *args, **kwargs)
+        except Exception:
+            traceback.print_exc()
+
+class ReqHandler(BaseHTTPRequestHandler):
+    def do_REQUEST(self):
+        path, query = self.path.split('?', 1) if '?' in self.path else (self.path, "")
+        params = {}
+
+        if query:
+            params.update(parse_qs(query))
+
+            if "<script>" in unquote_plus(query):
+                self.send_response(INTERNAL_SERVER_ERROR)
+                self.send_header("Connection", "close")
+                self.end_headers()
+                self.wfile.write("CLOUDFLARE_ERROR_500S_BOX".encode("utf8"))
+                return
+
+        if hasattr(self, "data"):
+            params.update(parse_qs(self.data))
+
+        for key in params:
+            if params[key]:
+                params[key] = params[key][-1]
+
+        self.url, self.params = path, params
+
+        if self.url == '/':
+            if "id" not in params:
+                self.send_response(FOUND)
+                self.send_header("Connection", "close")
+                self.send_header("Location", "/?id=1")
+                self.end_headers()
+            else:
+                self.send_response(OK)
+                self.send_header("Content-type", "text/html")
+                self.send_header("Connection", "close")
+                self.end_headers()
+
+                try:
+                    with _lock:
+                        _cursor.execute("SELECT * FROM users WHERE id=%s LIMIT 0, 1" % self.params.get("id", ""))
+                        results = _cursor.fetchall()
+
+                    output = "<b>SQL results:</b>\n"
+                    output += "<table border=\"1\">\n"
+                    for row in results:
+                        output += "<tr>"
+                        for value in row:
+                            output += "<td>%s</td>" % value
+                        output += "</tr>\n"
+                    output += "</table>\n"
+                    output += "</body></html>"
+                except Exception as ex:
+                    output = "%s: %s" % (re.search(r"'([^']+)'", str(type(ex))).group(1), ex)
+
+                self.wfile.write(output.encode("utf8"))
+        else:
+            self.send_response(NOT_FOUND)
+            self.send_header("Connection", "close")
+            self.end_headers()
+
+    def do_GET(self):
+        self.do_REQUEST()
+
+    def do_POST(self):
+        length = int(self.headers.get("Content-length", 0))
+        if length:
+            data = self.rfile.read(length)
+            data = unquote_plus(data.decode("utf8"))
+            self.data = data
+        self.do_REQUEST()
+
+    def log_message(self, format, *args):
+        return
+
+def run(address=LISTEN_ADDRESS, port=LISTEN_PORT):
+    global _server
+    try:
+        _server = ThreadingServer((address, port), ReqHandler)
+        print("[i] running HTTP server at '%s:%d'" % (address, port))
+        _server.serve_forever()
+    except KeyboardInterrupt:
+        _server.socket.close()
+        raise
+
+if __name__ == "__main__":
+    try:
+        init()
+        run(sys.argv[1] if len(sys.argv) > 1 else LISTEN_ADDRESS, int(sys.argv[2] if len(sys.argv) > 2 else LISTEN_PORT))
+    except KeyboardInterrupt:
+        print("\r[x] Ctrl-C received")

extra/wafdetectify/__init__.py

@@ -1,8 +0,0 @@
-#!/usr/bin/env python
-
-"""
-Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
-See the file 'LICENSE' for copying permission
-"""
-
-pass

extra/wafdetectify/wafdetectify.py

@@ -1,134 +0,0 @@
-#!/usr/bin/env python
-
-"""
-Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
-See the file 'LICENSE' for copying permission
-"""
-
-from __future__ import print_function
-
-import cookielib
-import glob
-import httplib
-import inspect
-import os
-import re
-import socket
-import ssl
-import subprocess
-import sys
-import urllib2
-
-sys.dont_write_bytecode = True
-
-if hasattr(ssl, "_create_unverified_context"):
-    ssl._create_default_https_context = ssl._create_unverified_context
-
-NAME, VERSION, AUTHOR = "WAF Detectify", "0.1", "sqlmap developers (@sqlmap)"
-TIMEOUT = 10
-HEADERS = {"User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:61.0) Gecko/20100101 Firefox/61.0", "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8", "Accept-Language": "en-US,en;q=0.5", "Accept-Encoding": "identity", "Cache-Control": "max-age=0"}
-SQLMAP_DIR = os.path.abspath(os.path.join(os.path.dirname(__file__), "..", ".."))
-SCRIPTS_DIR = os.path.join(SQLMAP_DIR, "waf")
-LEVEL_COLORS = {"o": "\033[00;94m", "x": "\033[00;91m", "!": "\033[00;93m", "i": "\033[00;92m"}
-CACHE = {}
-WAF_FUNCTIONS = []
-
-def get_page(get=None, url=None, host=None, data=None):
-    key = (get, url, host, data)
-
-    if key in CACHE:
-        return CACHE[key]
-
-    page, headers, code = None, {}, httplib.OK
-
-    url = url or ("%s%s%s" % (sys.argv[1], '?' if '?' not in sys.argv[1] else '&', get) if get else sys.argv[1])
-    if not url.startswith("http"):
-        url = "http://%s" % url
-
-    try:
-        req = urllib2.Request("".join(url[_].replace(' ', "%20") if _ > url.find('?') else url[_] for _ in xrange(len(url))), data, HEADERS)
-        conn = urllib2.urlopen(req, timeout=TIMEOUT)
-        page = conn.read()
-        headers = conn.info()
-    except Exception as ex:
-        code = getattr(ex, "code", None)
-        page = ex.read() if hasattr(ex, "read") else getattr(ex, "msg", "")
-        headers = ex.info() if hasattr(ex, "info") else {}
-
-    result = CACHE[key] = page, headers, code
-
-    return result
-
-def colorize(message):
-    if not subprocess.mswindows and sys.stdout.isatty():
-        message = re.sub(r"\[(.)\]", lambda match: "[%s%s\033[00;49m]" % (LEVEL_COLORS[match.group(1)], match.group(1)), message)
-        message = message.replace("@sqlmap", "\033[00;96m@sqlmap\033[00;49m")
-        message = message.replace(NAME, "\033[00;93m%s\033[00;49m" % NAME)
-
-    return message
-
-def main():
-    global WAF_FUNCTIONS
-
-    print(colorize("%s #v%s\n by: %s\n" % (NAME, VERSION, AUTHOR)))
-
-    if len(sys.argv) < 2:
-        exit(colorize("[x] usage: python %s <hostname>" % os.path.split(__file__)[-1]))
-
-    cookie_jar = cookielib.CookieJar()
-    opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cookie_jar))
-    urllib2.install_opener(opener)
-
-    sys.path.insert(0, SQLMAP_DIR)
-
-    for found in glob.glob(os.path.join(SCRIPTS_DIR, "*.py")):
-        dirname, filename = os.path.split(found)
-        dirname = os.path.abspath(dirname)
-
-        if filename == "__init__.py":
-            continue
-
-        if dirname not in sys.path:
-            sys.path.insert(0, dirname)
-
-        try:
-            if filename[:-3] in sys.modules:
-                del sys.modules[filename[:-3]]
-            module = __import__(filename[:-3].encode(sys.getfilesystemencoding() or "utf8"))
-        except ImportError as ex:
-            exit(colorize("[x] cannot import WAF script '%s' (%s)" % (filename[:-3], ex)))
-
-        _ = dict(inspect.getmembers(module))
-        if "detect" not in _:
-            exit(colorize("[x] missing function 'detect(get_page)' in WAF script '%s'" % found))
-        else:
-            WAF_FUNCTIONS.append((_["detect"], _.get("__product__", filename[:-3])))
-
-    WAF_FUNCTIONS = sorted(WAF_FUNCTIONS, key=lambda _: "generic" in _[1].lower())
-
-    print(colorize("[i] checking '%s'..." % sys.argv[1]))
-
-    hostname = sys.argv[1].split("//")[-1].split('/')[0]
-    try:
-        socket.getaddrinfo(hostname, None)
-    except socket.gaierror:
-        print(colorize("[x] host '%s' does not exist" % hostname))
-        exit(1)
-
-    found = False
-    for function, product in WAF_FUNCTIONS:
-        if found and "unknown" in product.lower():
-            continue
-
-        if function(get_page):
-            exit(colorize("[!] WAF/IPS identified as '%s'" % product))
-
-    if not found:
-        print(colorize("[o] nothing found"))
-
-    print()
-
-    exit(int(not found))
-
-if __name__ == "__main__":
-    main()

lib/__init__.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)

lib/controller/__init__.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)

lib/controller/action.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
@@ -97,37 +97,87 @@ def action():
             raise
 
     if conf.getDbs:
-        conf.dumper.dbs(conf.dbmsHandler.getDbs())
+        try:
+            conf.dumper.dbs(conf.dbmsHandler.getDbs())
+        except SqlmapNoneDataException as ex:
+            logger.critical(ex)
+        except:
+            raise
 
     if conf.getTables:
-        conf.dumper.dbTables(conf.dbmsHandler.getTables())
+        try:
+            conf.dumper.dbTables(conf.dbmsHandler.getTables())
+        except SqlmapNoneDataException as ex:
+            logger.critical(ex)
+        except:
+            raise
 
     if conf.commonTables:
-        conf.dumper.dbTables(tableExists(paths.COMMON_TABLES))
+        try:
+            conf.dumper.dbTables(tableExists(paths.COMMON_TABLES))
+        except SqlmapNoneDataException as ex:
+            logger.critical(ex)
+        except:
+            raise
 
     if conf.getSchema:
-        conf.dumper.dbTableColumns(conf.dbmsHandler.getSchema(), CONTENT_TYPE.SCHEMA)
+        try:
+            conf.dumper.dbTableColumns(conf.dbmsHandler.getSchema(), CONTENT_TYPE.SCHEMA)
+        except SqlmapNoneDataException as ex:
+            logger.critical(ex)
+        except:
+            raise
 
     if conf.getColumns:
-        conf.dumper.dbTableColumns(conf.dbmsHandler.getColumns(), CONTENT_TYPE.COLUMNS)
+        try:
+            conf.dumper.dbTableColumns(conf.dbmsHandler.getColumns(), CONTENT_TYPE.COLUMNS)
+        except SqlmapNoneDataException as ex:
+            logger.critical(ex)
+        except:
+            raise
 
     if conf.getCount:
-        conf.dumper.dbTablesCount(conf.dbmsHandler.getCount())
+        try:
+            conf.dumper.dbTablesCount(conf.dbmsHandler.getCount())
+        except SqlmapNoneDataException as ex:
+            logger.critical(ex)
+        except:
+            raise
 
     if conf.commonColumns:
-        conf.dumper.dbTableColumns(columnExists(paths.COMMON_COLUMNS))
+        try:
+            conf.dumper.dbTableColumns(columnExists(paths.COMMON_COLUMNS))
+        except SqlmapNoneDataException as ex:
+            logger.critical(ex)
+        except:
+            raise
 
     if conf.dumpTable:
-        conf.dbmsHandler.dumpTable()
+        try:
+            conf.dbmsHandler.dumpTable()
+        except SqlmapNoneDataException as ex:
+            logger.critical(ex)
+        except:
+            raise
 
     if conf.dumpAll:
-        conf.dbmsHandler.dumpAll()
+        try:
+            conf.dbmsHandler.dumpAll()
+        except SqlmapNoneDataException as ex:
+            logger.critical(ex)
+        except:
+            raise
 
     if conf.search:
-        conf.dbmsHandler.search()
+        try:
+            conf.dbmsHandler.search()
+        except SqlmapNoneDataException as ex:
+            logger.critical(ex)
+        except:
+            raise
 
-    if conf.query:
+    if conf.sqlQuery:
-        conf.dumper.query(conf.query, conf.dbmsHandler.sqlQuery(conf.query))
+        conf.dumper.sqlQuery(conf.sqlQuery, conf.dbmsHandler.sqlQuery(conf.sqlQuery))
 
     if conf.sqlShell:
         conf.dbmsHandler.sqlShell()

lib/controller/checks.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
@@ -6,7 +6,6 @@ See the file 'LICENSE' for copying permission
 """
 
 import copy
-import httplib
 import logging
 import os
 import random
@@ -22,6 +21,7 @@ from lib.core.agent import agent
 from lib.core.common import Backend
 from lib.core.common import extractRegexResult
 from lib.core.common import extractTextTagContent
+from lib.core.common import filterNone
 from lib.core.common import findDynamicContent
 from lib.core.common import Format
 from lib.core.common import getFilteredPageContent
@@ -48,7 +48,7 @@ from lib.core.common import unArrayizeValue
 from lib.core.common import urlencode
 from lib.core.common import wasLastResponseDBMSError
 from lib.core.common import wasLastResponseHTTPError
-from lib.core.convert import unicodeencode
+from lib.core.compat import xrange
 from lib.core.defaults import defaults
 from lib.core.data import conf
 from lib.core.data import kb
@@ -99,6 +99,7 @@ from lib.core.settings import SUPPORTED_DBMS
 from lib.core.settings import UNICODE_ENCODING
 from lib.core.settings import URI_HTTP_HEADER
 from lib.core.settings import UPPER_RATIO_BOUND
+from lib.core.settings import INFERENCE_EQUALS_CHAR
 from lib.core.threads import getCurrentThreadData
 from lib.request.connect import Connect as Request
 from lib.request.comparison import comparison
@@ -106,6 +107,8 @@ from lib.request.inject import checkBooleanExpression
 from lib.request.templates import getPageTemplate
 from lib.techniques.union.test import unionTest
 from lib.techniques.union.use import configUnion
+from thirdparty import six
+from thirdparty.six.moves import http_client as _http_client
 
 def checkSqlInjection(place, parameter, value):
     # Store here the details about boundaries and payload used to
@@ -579,7 +582,7 @@ def checkSqlInjection(place, parameter, value):
                                         else:
                                             errorSet = set()
 
-                                        candidates = filter(None, (_.strip() if _.strip() in trueRawResponse and _.strip() not in falseRawResponse else None for _ in (trueSet - falseSet - errorSet)))
+                                        candidates = filterNone(_.strip() if _.strip() in trueRawResponse and _.strip() not in falseRawResponse else None for _ in (trueSet - falseSet - errorSet))
 
                                         if candidates:
                                             candidates = sorted(candidates, key=lambda _: len(_))
@@ -593,7 +596,7 @@ def checkSqlInjection(place, parameter, value):
                                             logger.info(infoMsg)
 
                                         if not any((conf.string, conf.notString)):
-                                            candidates = filter(None, (_.strip() if _.strip() in falseRawResponse and _.strip() not in trueRawResponse else None for _ in (falseSet - trueSet)))
+                                            candidates = filterNone(_.strip() if _.strip() in falseRawResponse and _.strip() not in trueRawResponse else None for _ in (falseSet - trueSet))
 
                                             if candidates:
                                                 candidates = sorted(candidates, key=lambda _: len(_))
@@ -692,7 +695,7 @@ def checkSqlInjection(place, parameter, value):
                             # Test for UNION query SQL injection
                             reqPayload, vector = unionTest(comment, place, parameter, value, prefix, suffix)
 
-                            if isinstance(reqPayload, basestring):
+                            if isinstance(reqPayload, six.string_types):
                                 infoMsg = "%s parameter '%s' is '%s' injectable" % (paramType, parameter, title)
                                 logger.info(infoMsg)
 
@@ -913,27 +916,26 @@ def checkFalsePositives(injection):
                 if randInt3 > randInt2 > randInt1:
                     break
 
-            if not checkBooleanExpression("%d=%d" % (randInt1, randInt1)):
+            if not checkBooleanExpression("%d%s%d" % (randInt1, INFERENCE_EQUALS_CHAR, randInt1)):
                 retVal = False
                 break
 
-            # Just in case if DBMS hasn't properly recovered from previous delayed request
             if PAYLOAD.TECHNIQUE.BOOLEAN not in injection.data:
-                checkBooleanExpression("%d=%d" % (randInt1, randInt2))
+                checkBooleanExpression("%d%s%d" % (randInt1, INFERENCE_EQUALS_CHAR, randInt2))          # just in case if DBMS hasn't properly recovered from previous delayed request
 
-            if checkBooleanExpression("%d=%d" % (randInt1, randInt3)):          # this must not be evaluated to True
+            if checkBooleanExpression("%d%s%d" % (randInt1, INFERENCE_EQUALS_CHAR, randInt3)):          # this must not be evaluated to True
                 retVal = False
                 break
 
-            elif checkBooleanExpression("%d=%d" % (randInt3, randInt2)):        # this must not be evaluated to True
+            elif checkBooleanExpression("%d%s%d" % (randInt3, INFERENCE_EQUALS_CHAR, randInt2)):        # this must not be evaluated to True
                 retVal = False
                 break
 
-            elif not checkBooleanExpression("%d=%d" % (randInt2, randInt2)):    # this must be evaluated to True
+            elif not checkBooleanExpression("%d%s%d" % (randInt2, INFERENCE_EQUALS_CHAR, randInt2)):    # this must be evaluated to True
                 retVal = False
                 break
 
-            elif checkBooleanExpression("%d %d" % (randInt3, randInt2)):        # this must not be evaluated to True (invalid statement)
+            elif checkBooleanExpression("%d %d" % (randInt3, randInt2)):                                # this must not be evaluated to True (invalid statement)
                 retVal = False
                 break
 
@@ -1337,6 +1339,9 @@ def checkWaf():
     if any((conf.string, conf.notString, conf.regexp, conf.dummy, conf.offline, conf.skipWaf)):
         return None
 
+    if kb.originalCode == _http_client.NOT_FOUND:
+        return None
+
     _ = hashDBRetrieve(HASHDB_KEYS.CHECK_WAF_RESULT, True)
     if _ is not None:
         if _:
@@ -1421,17 +1426,24 @@ def identifyWaf():
         page, headers, code = None, None, None
         try:
             pushValue(kb.redirectChoice)
+            pushValue(kb.resendPostOnRedirect)
+
             kb.redirectChoice = REDIRECTION.YES
+            kb.resendPostOnRedirect = True
+
             if kwargs.get("get"):
                 kwargs["get"] = urlencode(kwargs["get"])
             kwargs["raise404"] = False
             kwargs["silent"] = True
             kwargs["finalCode"] = True
+
             page, headers, code = Request.getPage(*args, **kwargs)
         except Exception:
             pass
         finally:
+            kb.resendPostOnRedirect = popValue()
             kb.redirectChoice = popValue()
+
         return page or "", headers or {}, code
 
     retVal = []
@@ -1497,46 +1509,59 @@ def checkNullConnection():
     if conf.data:
         return False
 
-    infoMsg = "testing NULL connection to the target URL"
+    _ = hashDBRetrieve(HASHDB_KEYS.CHECK_NULL_CONNECTION_RESULT, True)
-    logger.info(infoMsg)
+    if _ is not None:
+        kb.nullConnection = _
 
-    pushValue(kb.pageCompress)
+        if _:
-    kb.pageCompress = False
+            dbgMsg = "resuming NULL connection method '%s'" % _
+            logger.debug(dbgMsg)
 
-    try:
+    else:
-        page, headers, _ = Request.getPage(method=HTTPMETHOD.HEAD, raise404=False)
+        infoMsg = "testing NULL connection to the target URL"
+        logger.info(infoMsg)
 
-        if not page and HTTP_HEADER.CONTENT_LENGTH in (headers or {}):
+        pushValue(kb.pageCompress)
-            kb.nullConnection = NULLCONNECTION.HEAD
+        kb.pageCompress = False
 
-            infoMsg = "NULL connection is supported with HEAD method ('Content-Length')"
+        try:
-            logger.info(infoMsg)
+            page, headers, _ = Request.getPage(method=HTTPMETHOD.HEAD, raise404=False)
-        else:
-            page, headers, _ = Request.getPage(auxHeaders={HTTP_HEADER.RANGE: "bytes=-1"})
 
-            if page and len(page) == 1 and HTTP_HEADER.CONTENT_RANGE in (headers or {}):
+            if not page and HTTP_HEADER.CONTENT_LENGTH in (headers or {}):
-                kb.nullConnection = NULLCONNECTION.RANGE
+                kb.nullConnection = NULLCONNECTION.HEAD
 
-                infoMsg = "NULL connection is supported with GET method ('Range')"
+                infoMsg = "NULL connection is supported with HEAD method ('Content-Length')"
                 logger.info(infoMsg)
             else:
-                _, headers, _ = Request.getPage(skipRead=True)
+                page, headers, _ = Request.getPage(auxHeaders={HTTP_HEADER.RANGE: "bytes=-1"})
 
-                if HTTP_HEADER.CONTENT_LENGTH in (headers or {}):
+                if page and len(page) == 1 and HTTP_HEADER.CONTENT_RANGE in (headers or {}):
-                    kb.nullConnection = NULLCONNECTION.SKIP_READ
+                    kb.nullConnection = NULLCONNECTION.RANGE
 
-                    infoMsg = "NULL connection is supported with 'skip-read' method"
+                    infoMsg = "NULL connection is supported with GET method ('Range')"
                     logger.info(infoMsg)
+                else:
+                    _, headers, _ = Request.getPage(skipRead=True)
 
-    except SqlmapConnectionException:
+                    if HTTP_HEADER.CONTENT_LENGTH in (headers or {}):
-        pass
+                        kb.nullConnection = NULLCONNECTION.SKIP_READ
 
-    finally:
+                        infoMsg = "NULL connection is supported with 'skip-read' method"
-        kb.pageCompress = popValue()
+                        logger.info(infoMsg)
 
-    return kb.nullConnection is not None
+        except SqlmapConnectionException:
+            pass
+
+        finally:
+            kb.pageCompress = popValue()
+            kb.nullConnection = False if kb.nullConnection is None else kb.nullConnection
+            hashDBWrite(HASHDB_KEYS.CHECK_NULL_CONNECTION_RESULT, kb.nullConnection, True)
+
+    return kb.nullConnection in getPublicTypeMembers(NULLCONNECTION, True)
 
 def checkConnection(suppressOutput=False):
+    threadData = getCurrentThreadData()
+
     if not re.search(r"\A\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\Z", conf.hostname):
         if not any((conf.proxy, conf.tor, conf.dummy, conf.offline)):
             try:
@@ -1561,8 +1586,7 @@ def checkConnection(suppressOutput=False):
 
     try:
         kb.originalPageTime = time.time()
-        page, headers, _ = Request.queryPage(content=True, noteResponseTime=False)
+        Request.queryPage(content=True, noteResponseTime=False)
-        kb.originalPage = kb.pageTemplate = page
 
         kb.errorIsNone = False
 
@@ -1585,10 +1609,8 @@ def checkConnection(suppressOutput=False):
         else:
             kb.errorIsNone = True
 
-        threadData = getCurrentThreadData()
-
         if kb.redirectChoice == REDIRECTION.YES and threadData.lastRedirectURL and threadData.lastRedirectURL[0] == threadData.lastRequestUID:
-            if (threadData.lastRedirectURL[1] or "").startswith("https://") and unicodeencode(conf.hostname) in threadData.lastRedirectURL[1]:
+            if (threadData.lastRedirectURL[1] or "").startswith("https://") and conf.hostname in getUnicode(threadData.lastRedirectURL[1]):
                 conf.url = re.sub(r"https?://", "https://", conf.url)
                 match = re.search(r":(\d+)", threadData.lastRedirectURL[1])
                 port = match.group(1) if match else 443
@@ -1603,7 +1625,7 @@ def checkConnection(suppressOutput=False):
             warnMsg += "any addressing issues"
             singleTimeWarnMessage(warnMsg)
 
-        if any(code in kb.httpErrorCodes for code in (httplib.NOT_FOUND, )):
+        if any(code in kb.httpErrorCodes for code in (_http_client.NOT_FOUND, )):
             errMsg = getSafeExString(ex)
             logger.critical(errMsg)
 
@@ -1617,6 +1639,9 @@ def checkConnection(suppressOutput=False):
                 kb.ignoreNotFound = True
         else:
             raise
+    finally:
+        kb.originalPage = kb.pageTemplate = threadData.lastPage
+        kb.originalCode = threadData.lastCode
 
     return True
 

lib/controller/controller.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
@@ -40,6 +40,7 @@ from lib.core.common import safeCSValue
 from lib.core.common import showHttpErrorCodes
 from lib.core.common import urlencode
 from lib.core.common import urldecode
+from lib.core.compat import xrange
 from lib.core.data import conf
 from lib.core.data import kb
 from lib.core.data import logger
@@ -47,6 +48,7 @@ from lib.core.decorators import stackedmethod
 from lib.core.enums import CONTENT_TYPE
 from lib.core.enums import HASHDB_KEYS
 from lib.core.enums import HEURISTIC_TEST
+from lib.core.enums import HTTP_HEADER
 from lib.core.enums import HTTPMETHOD
 from lib.core.enums import NOTE
 from lib.core.enums import PAYLOAD
@@ -317,6 +319,13 @@ def start():
             conf.cookie = targetCookie
             conf.httpHeaders = list(initialHeaders)
             conf.httpHeaders.extend(targetHeaders or [])
+
+            if conf.randomAgent or conf.mobile:
+                for header, value in initialHeaders:
+                    if header.upper() == HTTP_HEADER.USER_AGENT.upper():
+                        conf.httpHeaders.append((header, value))
+                        break
+
             conf.httpHeaders = [conf.httpHeaders[i] for i in xrange(len(conf.httpHeaders)) if conf.httpHeaders[i][0].upper() not in (__[0].upper() for __ in conf.httpHeaders[i + 1:])]
 
             initTargetEnv()
@@ -324,7 +333,7 @@ def start():
 
             testSqlInj = False
 
-            if PLACE.GET in conf.parameters and not any([conf.data, conf.testParameter]):
+            if PLACE.GET in conf.parameters and not any((conf.data, conf.testParameter)):
                 for parameter in re.findall(r"([^=]+)=([^%s]+%s?|\Z)" % (re.escape(conf.paramDel or "") or DEFAULT_GET_POST_DELIMITER, re.escape(conf.paramDel or "") or DEFAULT_GET_POST_DELIMITER), conf.parameters[PLACE.GET]):
                     paramKey = (conf.hostname, conf.path, PLACE.GET, parameter[0])
 
@@ -491,7 +500,7 @@ def start():
                         elif parameter in conf.testParameter:
                             pass
 
-                        elif parameter == conf.rParam:
+                        elif parameter in conf.rParam:
                             testSqlInj = False
 
                             infoMsg = "skipping randomizing %s parameter '%s'" % (paramType, parameter)

lib/controller/handler.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
@@ -75,7 +75,7 @@ def setHandler():
         (DBMS.INFORMIX, INFORMIX_ALIASES, InformixMap, InformixConn),
     ]
 
-    _ = max(_ if (conf.get("dbms") or Backend.getIdentifiedDbms() or kb.heuristicExtendedDbms or "").lower() in _[1] else None for _ in items)
+    _ = max(_ if (conf.get("dbms") or Backend.getIdentifiedDbms() or kb.heuristicExtendedDbms or "").lower() in _[1] else "" for _ in items) or None
     if _:
         items.remove(_)
         items.insert(0, _)

lib/core/__init__.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)

lib/core/agent.py

@@ -1,14 +1,16 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """
 
+import base64
 import re
 
 from lib.core.common import Backend
 from lib.core.common import extractRegexResult
+from lib.core.common import filterNone
 from lib.core.common import getSQLSnippet
 from lib.core.common import getUnicode
 from lib.core.common import isDBMSVersionAtLeast
@@ -23,6 +25,7 @@ from lib.core.common import splitFields
 from lib.core.common import unArrayizeValue
 from lib.core.common import urlencode
 from lib.core.common import zeroDepthSearch
+from lib.core.compat import xrange
 from lib.core.data import conf
 from lib.core.data import kb
 from lib.core.data import queries
@@ -105,7 +108,7 @@ class Agent(object):
             if place == PLACE.URI:
                 origValue = origValue.split(kb.customInjectionMark)[0]
             else:
-                origValue = filter(None, (re.search(_, origValue.split(BOUNDED_INJECTION_MARKER)[0]) for _ in (r"\w+\Z", r"[^\"'><]+\Z", r"[^ ]+\Z")))[0].group(0)
+                origValue = filterNone(re.search(_, origValue.split(BOUNDED_INJECTION_MARKER)[0]) for _ in (r"\w+\Z", r"[^\"'><]+\Z", r"[^ ]+\Z"))[0].group(0)
             origValue = origValue[origValue.rfind('/') + 1:]
             for char in ('?', '=', ':', ',', '&'):
                 if char in origValue:
@@ -162,6 +165,11 @@ class Agent(object):
 
         newValue = self.cleanupPayload(newValue, origValue)
 
+        if re.sub(r" \(.+", "", parameter) in conf.base64Parameter:
+            # TODO: support for POST_HINT
+            newValue = base64.b64encode(newValue)
+            origValue = base64.b64encode(origValue)
+
         if place in (PLACE.URI, PLACE.CUSTOM_POST, PLACE.CUSTOM_HEADER):
             _ = "%s%s" % (origValue, kb.customInjectionMark)
             if kb.postHint == POST_HINT.JSON and not isNumber(newValue) and not '"%s"' % _ in paramString:
@@ -298,16 +306,19 @@ class Agent(object):
         if payload is None:
             return
 
-        replacements = (
+        replacements = {
-            ("[DELIMITER_START]", kb.chars.start),
+            "[DELIMITER_START]": kb.chars.start,
-            ("[DELIMITER_STOP]", kb.chars.stop),
+            "[DELIMITER_STOP]": kb.chars.stop,
-            ("[AT_REPLACE]", kb.chars.at),
+            "[AT_REPLACE]": kb.chars.at,
-            ("[SPACE_REPLACE]", kb.chars.space),
+            "[SPACE_REPLACE]": kb.chars.space,
-            ("[DOLLAR_REPLACE]", kb.chars.dollar),
+            "[DOLLAR_REPLACE]": kb.chars.dollar,
-            ("[HASH_REPLACE]", kb.chars.hash_),
+            "[HASH_REPLACE]": kb.chars.hash_,
-            ("[GENERIC_SQL_COMMENT]", GENERIC_SQL_COMMENT)
+            "[GENERIC_SQL_COMMENT]": GENERIC_SQL_COMMENT
-        )
+        }
-        payload = reduce(lambda x, y: x.replace(y[0], y[1]), replacements, payload)
+
+        for value in re.findall(r"\[[A-Z_]+\]", payload):
+            if value in replacements:
+                payload = payload.replace(value, replacements[value])
 
         for _ in set(re.findall(r"(?i)\[RANDNUM(?:\d+)?\]", payload)):
             payload = payload.replace(_, str(randomInt()))
@@ -978,8 +989,7 @@ class Agent(object):
                     limitedQuery = limitedQuery.replace(" (SELECT TOP %s" % startTopNums, " (SELECT TOP %d" % num)
                     forgeNotIn = False
                 else:
-                    topNum = re.search(r"TOP\s+([\d]+)\s+", limitedQuery, re.I).group(1)
+                    limitedQuery = re.sub(r"\bTOP\s+\d+\s*", "", limitedQuery, flags=re.I)
-                    limitedQuery = limitedQuery.replace("TOP %s " % topNum, "")
 
             if forgeNotIn:
                 limitedQuery = limitedQuery.replace("SELECT ", (limitStr % 1), 1)

lib/core/bigarray.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
@@ -16,6 +16,7 @@ import os
 import sys
 import tempfile
 
+from lib.core.compat import xrange
 from lib.core.enums import MKSTEMP_PREFIX
 from lib.core.exception import SqlmapSystemException
 from lib.core.settings import BIGARRAY_CHUNK_SIZE

lib/core/compat.py

@@ -0,0 +1,172 @@
+#!/usr/bin/env python2
+
+"""
+Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
+See the file 'LICENSE' for copying permission
+"""
+
+import binascii
+import os
+import random
+import uuid
+import sys
+
+class WichmannHill(random.Random):
+    """
+    Reference: https://svn.python.org/projects/python/trunk/Lib/random.py
+    """
+
+    VERSION = 1     # used by getstate/setstate
+
+    def seed(self, a=None):
+        """Initialize internal state from hashable object.
+
+        None or no argument seeds from current time or from an operating
+        system specific randomness source if available.
+
+        If a is not None or an int or long, hash(a) is used instead.
+
+        If a is an int or long, a is used directly.  Distinct values between
+        0 and 27814431486575L inclusive are guaranteed to yield distinct
+        internal states (this guarantee is specific to the default
+        Wichmann-Hill generator).
+        """
+
+        if a is None:
+            try:
+                a = int(binascii.hexlify(os.urandom(16)), 16)
+            except NotImplementedError:
+                import time
+                a = int(time.time() * 256)  # use fractional seconds
+
+        if not isinstance(a, int):
+            a = hash(a)
+
+        a, x = divmod(a, 30268)
+        a, y = divmod(a, 30306)
+        a, z = divmod(a, 30322)
+        self._seed = int(x) + 1, int(y) + 1, int(z) + 1
+
+        self.gauss_next = None
+
+    def random(self):
+        """Get the next random number in the range [0.0, 1.0)."""
+
+        # Wichman-Hill random number generator.
+        #
+        # Wichmann, B. A. & Hill, I. D. (1982)
+        # Algorithm AS 183:
+        # An efficient and portable pseudo-random number generator
+        # Applied Statistics 31 (1982) 188-190
+        #
+        # see also:
+        #        Correction to Algorithm AS 183
+        #        Applied Statistics 33 (1984) 123
+        #
+        #        McLeod, A. I. (1985)
+        #        A remark on Algorithm AS 183
+        #        Applied Statistics 34 (1985),198-200
+
+        # This part is thread-unsafe:
+        # BEGIN CRITICAL SECTION
+        x, y, z = self._seed
+        x = (171 * x) % 30269
+        y = (172 * y) % 30307
+        z = (170 * z) % 30323
+        self._seed = x, y, z
+        # END CRITICAL SECTION
+
+        # Note:  on a platform using IEEE-754 double arithmetic, this can
+        # never return 0.0 (asserted by Tim; proof too long for a comment).
+        return (x / 30269.0 + y / 30307.0 + z / 30323.0) % 1.0
+
+    def getstate(self):
+        """Return internal state; can be passed to setstate() later."""
+        return self.VERSION, self._seed, self.gauss_next
+
+    def setstate(self, state):
+        """Restore internal state from object returned by getstate()."""
+        version = state[0]
+        if version == 1:
+            version, self._seed, self.gauss_next = state
+        else:
+            raise ValueError("state with version %s passed to "
+                             "Random.setstate() of version %s" %
+                             (version, self.VERSION))
+
+    def jumpahead(self, n):
+        """Act as if n calls to random() were made, but quickly.
+
+        n is an int, greater than or equal to 0.
+
+        Example use:  If you have 2 threads and know that each will
+        consume no more than a million random numbers, create two Random
+        objects r1 and r2, then do
+            r2.setstate(r1.getstate())
+            r2.jumpahead(1000000)
+        Then r1 and r2 will use guaranteed-disjoint segments of the full
+        period.
+        """
+
+        if not n >= 0:
+            raise ValueError("n must be >= 0")
+        x, y, z = self._seed
+        x = int(x * pow(171, n, 30269)) % 30269
+        y = int(y * pow(172, n, 30307)) % 30307
+        z = int(z * pow(170, n, 30323)) % 30323
+        self._seed = x, y, z
+
+    def __whseed(self, x=0, y=0, z=0):
+        """Set the Wichmann-Hill seed from (x, y, z).
+
+        These must be integers in the range [0, 256).
+        """
+
+        if not type(x) == type(y) == type(z) == int:
+            raise TypeError('seeds must be integers')
+        if not (0 <= x < 256 and 0 <= y < 256 and 0 <= z < 256):
+            raise ValueError('seeds must be in range(0, 256)')
+        if 0 == x == y == z:
+            # Initialize from current time
+            import time
+            t = int(time.time() * 256)
+            t = int((t & 0xffffff) ^ (t >> 24))
+            t, x = divmod(t, 256)
+            t, y = divmod(t, 256)
+            t, z = divmod(t, 256)
+        # Zero is a poor seed, so substitute 1
+        self._seed = (x or 1, y or 1, z or 1)
+
+        self.gauss_next = None
+
+    def whseed(self, a=None):
+        """Seed from hashable object's hash code.
+
+        None or no argument seeds from current time.  It is not guaranteed
+        that objects with distinct hash codes lead to distinct internal
+        states.
+
+        This is obsolete, provided for compatibility with the seed routine
+        used prior to Python 2.1.  Use the .seed() method instead.
+        """
+
+        if a is None:
+            self.__whseed()
+            return
+        a = hash(a)
+        a, x = divmod(a, 256)
+        a, y = divmod(a, 256)
+        a, z = divmod(a, 256)
+        x = (x + a) % 256 or 1
+        y = (y + a) % 256 or 1
+        z = (z + a) % 256 or 1
+        self.__whseed(x, y, z)
+
+# Reference: https://github.com/urllib3/urllib3/blob/master/src/urllib3/filepost.py
+def choose_boundary():
+    return uuid.uuid4().hex
+
+if sys.version_info >= (3, 0):
+    xrange = range
+else:
+    xrange = xrange

lib/core/convert.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
@@ -9,18 +9,15 @@ try:
     import cPickle as pickle
 except:
     import pickle
-finally:
-    import pickle as picklePy
 
 import base64
 import json
 import re
-import StringIO
 import sys
 
 from lib.core.settings import IS_WIN
 from lib.core.settings import UNICODE_ENCODING
-from lib.core.settings import PICKLE_REDUCE_WHITELIST
+from thirdparty import six
 
 def base64decode(value):
     """
@@ -30,7 +27,7 @@ def base64decode(value):
     'foobar'
     """
 
-    return base64.b64decode(value)
+    return base64.b64decode(unicodeencode(value))
 
 def base64encode(value):
     """
@@ -40,7 +37,7 @@ def base64encode(value):
     'Zm9vYmFy'
     """
 
-    return base64.b64encode(value)
+    return base64.b64encode(unicodeencode(value))
 
 def base64pickle(value):
     """
@@ -66,7 +63,7 @@ def base64pickle(value):
 
     return retVal
 
-def base64unpickle(value, unsafe=False):
+def base64unpickle(value):
     """
     Decodes value from Base64 to plain format and deserializes (with pickle) its content
 
@@ -76,26 +73,10 @@ def base64unpickle(value, unsafe=False):
 
     retVal = None
 
-    def _(self):
-        if len(self.stack) > 1:
-            func = self.stack[-2]
-            if func not in PICKLE_REDUCE_WHITELIST:
-                raise Exception("abusing reduce() is bad, Mkay!")
-        self.load_reduce()
-
-    def loads(str):
-        f = StringIO.StringIO(str)
-        if unsafe:
-            unpickler = picklePy.Unpickler(f)
-            unpickler.dispatch[picklePy.REDUCE] = _
-        else:
-            unpickler = pickle.Unpickler(f)
-        return unpickler.load()
-
     try:
-        retVal = loads(base64decode(value))
+        retVal = pickle.loads(base64decode(value))
     except TypeError:
-        retVal = loads(base64decode(bytes(value)))
+        retVal = pickle.loads(base64decode(bytes(value)))
 
     return retVal
 
@@ -108,7 +89,14 @@ def hexdecode(value):
     """
 
     value = value.lower()
-    return (value[2:] if value.startswith("0x") else value).decode("hex")
+    value = value[2:] if value.startswith("0x") else value
+
+    if six.PY2:
+        retVal = value.decode("hex")
+    else:
+        retVal = bytes.fromhex(value)
+
+    return retVal
 
 def hexencode(value, encoding=None):
     """
@@ -118,7 +106,14 @@ def hexencode(value, encoding=None):
     '666f6f626172'
     """
 
-    return unicodeencode(value, encoding).encode("hex")
+    retVal = unicodeencode(value, encoding)
+
+    if six.PY2:
+        retVal = retVal.encode("hex")
+    else:
+        retVal = retVal.hex()
+
+    return retVal
 
 def unicodeencode(value, encoding=None):
     """
@@ -129,11 +124,13 @@ def unicodeencode(value, encoding=None):
     """
 
     retVal = value
-    if isinstance(value, unicode):
+
+    if isinstance(value, six.text_type):
         try:
             retVal = value.encode(encoding or UNICODE_ENCODING)
         except UnicodeEncodeError:
-            retVal = value.encode(UNICODE_ENCODING, "replace")
+            retVal = value.encode(encoding or UNICODE_ENCODING, "replace")
+
     return retVal
 
 def utf8encode(value):
@@ -165,9 +162,11 @@ def htmlunescape(value):
     """
 
     retVal = value
-    if value and isinstance(value, basestring):
+    if value and isinstance(value, six.string_types):
-        codes = (("&lt;", '<'), ("&gt;", '>'), ("&quot;", '"'), ("&nbsp;", ' '), ("&amp;", '&'), ("&apos;", "'"))
+        replacements = (("&lt;", '<'), ("&gt;", '>'), ("&quot;", '"'), ("&nbsp;", ' '), ("&amp;", '&'), ("&apos;", "'"))
-        retVal = reduce(lambda x, y: x.replace(y[0], y[1]), codes, retVal)
+        for code, value in replacements:
+            retVal = retVal.replace(code, value)
+
         try:
             retVal = re.sub(r"&#x([^ ;]+);", lambda match: unichr(int(match.group(1), 16)), retVal)
         except ValueError:
@@ -180,30 +179,26 @@ def singleTimeWarnMessage(message):  # Cross-referenced function
     sys.stdout.flush()
 
 def stdoutencode(data):
-    retVal = None
+    retVal = data
 
-    try:
+    if six.PY2:
-        data = data or ""
+        try:
+            retVal = unicodeencode(data or "", sys.stdout.encoding)
 
-        # Reference: http://bugs.python.org/issue1602
+            # Reference: http://bugs.python.org/issue1602
-        if IS_WIN:
+            if IS_WIN:
-            output = data.encode(sys.stdout.encoding, "replace")
+                if '?' in retVal and '?' not in retVal:
+                    warnMsg = "cannot properly display Unicode characters "
+                    warnMsg += "inside Windows OS command prompt "
+                    warnMsg += "(http://bugs.python.org/issue1602). All "
+                    warnMsg += "unhandled occurrences will result in "
+                    warnMsg += "replacement with '?' character. Please, find "
+                    warnMsg += "proper character representation inside "
+                    warnMsg += "corresponding output files. "
+                    singleTimeWarnMessage(warnMsg)
 
-            if '?' in output and '?' not in data:
+        except:
-                warnMsg = "cannot properly display Unicode characters "
+            retVal = unicodeencode(data or "")
-                warnMsg += "inside Windows OS command prompt "
-                warnMsg += "(http://bugs.python.org/issue1602). All "
-                warnMsg += "unhandled occurrences will result in "
-                warnMsg += "replacement with '?' character. Please, find "
-                warnMsg += "proper character representation inside "
-                warnMsg += "corresponding output files. "
-                singleTimeWarnMessage(warnMsg)
-
-            retVal = output
-        else:
-            retVal = data.encode(sys.stdout.encoding)
-    except:
-        retVal = data.encode(UNICODE_ENCODING) if isinstance(data, unicode) else data
 
     return retVal
 

lib/core/data.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)

lib/core/datatype.py

@@ -1,14 +1,15 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """
 
+import collections
 import copy
 import types
 
-from thirdparty.odict.odict import OrderedDict
+from thirdparty.odict import OrderedDict
 
 class AttribDict(dict):
     """
@@ -120,15 +121,12 @@ class LRUDict(object):
         return key in self.cache
 
     def __getitem__(self, key):
-        try:
+        value = self.cache.pop(key)
-            value = self.cache.pop(key)
+        self.cache[key] = value
-            self.cache[key] = value
+        return value
-            return value
-        except KeyError:
-            return -1
 
     def get(self, key):
-        return self.__getitem__(self, key)
+        return self.__getitem__(key)
 
     def __setitem__(self, key, value):
         try:
@@ -143,3 +141,61 @@ class LRUDict(object):
 
     def keys(self):
         return self.cache.keys()
+
+# Reference: https://code.activestate.com/recipes/576694/
+class OrderedSet(collections.MutableSet):
+    def __init__(self, iterable=None):
+        self.end = end = []
+        end += [None, end, end]         # sentinel node for doubly linked list
+        self.map = {}                   # key --> [key, prev, next]
+        if iterable is not None:
+            self |= iterable
+
+    def __len__(self):
+        return len(self.map)
+
+    def __contains__(self, key):
+        return key in self.map
+
+    def add(self, key):
+        if key not in self.map:
+            end = self.end
+            curr = end[1]
+            curr[2] = end[1] = self.map[key] = [key, curr, end]
+
+    def discard(self, key):
+        if key in self.map:
+            key, prev, next = self.map.pop(key)
+            prev[2] = next
+            next[1] = prev
+
+    def __iter__(self):
+        end = self.end
+        curr = end[2]
+        while curr is not end:
+            yield curr[0]
+            curr = curr[2]
+
+    def __reversed__(self):
+        end = self.end
+        curr = end[1]
+        while curr is not end:
+            yield curr[0]
+            curr = curr[1]
+
+    def pop(self, last=True):
+        if not self:
+            raise KeyError('set is empty')
+        key = self.end[1][0] if last else self.end[2][0]
+        self.discard(key)
+        return key
+
+    def __repr__(self):
+        if not self:
+            return '%s()' % (self.__class__.__name__,)
+        return '%s(%r)' % (self.__class__.__name__, list(self))
+
+    def __eq__(self, other):
+        if isinstance(other, OrderedSet):
+            return len(self) == len(other) and list(self) == list(other)
+        return set(self) == set(other)

lib/core/decorators.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
@@ -10,6 +10,7 @@ import hashlib
 import threading
 
 from lib.core.settings import MAX_CACHE_ITEMS
+from lib.core.settings import UNICODE_ENCODING
 from lib.core.datatype import LRUDict
 from lib.core.threads import getCurrentThreadData
 
@@ -24,12 +25,18 @@ def cachedmethod(f, cache=LRUDict(capacity=MAX_CACHE_ITEMS)):
 
     @functools.wraps(f)
     def _(*args, **kwargs):
-        with _lock:
+        key = int(hashlib.md5("|".join(str(_) for _ in (f, args, kwargs)).encode(UNICODE_ENCODING)).hexdigest(), 16) & 0x7fffffffffffffff
-            key = int(hashlib.md5("|".join(str(_) for _ in (f, args, kwargs))).hexdigest(), 16) & 0x7fffffffffffffff
-            if key not in cache:
-                cache[key] = f(*args, **kwargs)
 
-            return cache[key]
+        try:
+            with _lock:
+                result = cache[key]
+        except KeyError:
+            result = f(*args, **kwargs)
+
+            with _lock:
+                cache[key] = result
+
+        return result
 
     return _
 

lib/core/defaults.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)

lib/core/dicts.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)

lib/core/dump.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
@@ -17,6 +17,7 @@ from lib.core.common import Backend
 from lib.core.common import checkFile
 from lib.core.common import dataToDumpFile
 from lib.core.common import dataToStdout
+from lib.core.common import getBytes
 from lib.core.common import getSafeExString
 from lib.core.common import getUnicode
 from lib.core.common import isListLike
@@ -26,8 +27,8 @@ from lib.core.common import openFile
 from lib.core.common import prioritySortColumns
 from lib.core.common import randomInt
 from lib.core.common import safeCSValue
-from lib.core.common import unicodeencode
 from lib.core.common import unsafeSQLIdentificatorNaming
+from lib.core.compat import xrange
 from lib.core.data import conf
 from lib.core.data import kb
 from lib.core.data import logger
@@ -50,6 +51,7 @@ from lib.core.settings import UNICODE_ENCODING
 from lib.core.settings import UNSAFE_DUMP_FILEPATH_REPLACEMENT
 from lib.core.settings import VERSION_STRING
 from lib.core.settings import WINDOWS_RESERVED_NAMES
+from thirdparty import six
 from thirdparty.magic import magic
 
 from extra.safe2bin.safe2bin import safechardecode
@@ -133,7 +135,7 @@ class Dump(object):
             if "\n" in _:
                 self._write("%s:\n---\n%s\n---" % (header, _))
             else:
-                self._write("%s:    %s" % (header, ("'%s'" % _) if isinstance(data, basestring) else _))
+                self._write("%s: %s" % (header, ("'%s'" % _) if isinstance(data, six.string_types) else _))
         else:
             self._write("%s:\tNone" % header)
 
@@ -142,7 +144,7 @@ class Dump(object):
             try:
                 elements = set(elements)
                 elements = list(elements)
-                elements.sort(key=lambda _: _.lower() if isinstance(_, basestring) else _)
+                elements.sort(key=lambda _: _.lower() if hasattr(_, "lower") else _)
             except:
                 pass
 
@@ -154,7 +156,7 @@ class Dump(object):
             self._write("%s [%d]:" % (header, len(elements)))
 
         for element in elements:
-            if isinstance(element, basestring):
+            if isinstance(element, six.string_types):
                 self._write("[*] %s" % element)
             elif isListLike(element):
                 self._write("[*] " + ", ".join(getUnicode(e) for e in element))
@@ -193,7 +195,7 @@ class Dump(object):
             userSettings = userSettings[0]
 
         users = userSettings.keys()
-        users.sort(key=lambda _: _.lower() if isinstance(_, basestring) else _)
+        users.sort(key=lambda _: _.lower() if hasattr(_, "lower") else _)
 
         if conf.api:
             self._write(userSettings, content_type=content_type)
@@ -240,7 +242,7 @@ class Dump(object):
                     if table and isListLike(table):
                         table = table[0]
 
-                    maxlength = max(maxlength, len(unsafeSQLIdentificatorNaming(normalizeUnicode(table) or unicode(table))))
+                    maxlength = max(maxlength, len(unsafeSQLIdentificatorNaming(normalizeUnicode(table) or getUnicode(table))))
 
             lines = "-" * (int(maxlength) + 2)
 
@@ -261,7 +263,7 @@ class Dump(object):
                         table = table[0]
 
                     table = unsafeSQLIdentificatorNaming(table)
-                    blank = " " * (maxlength - len(normalizeUnicode(table) or unicode(table)))
+                    blank = " " * (maxlength - len(normalizeUnicode(table) or getUnicode(table)))
                     self._write("| %s%s |" % (table, blank))
 
                 self._write("+%s+\n" % lines)
@@ -287,7 +289,7 @@ class Dump(object):
                     colType = None
 
                     colList = columns.keys()
-                    colList.sort(key=lambda _: _.lower() if isinstance(_, basestring) else _)
+                    colList.sort(key=lambda _: _.lower() if hasattr(_, "lower") else _)
 
                     for column in colList:
                         colType = columns[column]
@@ -356,7 +358,7 @@ class Dump(object):
             for ctables in dbTables.values():
                 for tables in ctables.values():
                     for table in tables:
-                        maxlength1 = max(maxlength1, len(normalizeUnicode(table) or unicode(table)))
+                        maxlength1 = max(maxlength1, len(normalizeUnicode(table) or getUnicode(table)))
 
             for db, counts in dbTables.items():
                 self._write("Database: %s" % unsafeSQLIdentificatorNaming(db) if db else "Current database")
@@ -379,10 +381,10 @@ class Dump(object):
                     if count is None:
                         count = "Unknown"
 
-                    tables.sort(key=lambda _: _.lower() if isinstance(_, basestring) else _)
+                    tables.sort(key=lambda _: _.lower() if hasattr(_, "lower") else _)
 
                     for table in tables:
-                        blank1 = " " * (maxlength1 - len(normalizeUnicode(table) or unicode(table)))
+                        blank1 = " " * (maxlength1 - len(normalizeUnicode(table) or getUnicode(table)))
                         blank2 = " " * (maxlength2 - len(str(count)))
                         self._write("| %s%s | %d%s |" % (table, blank1, count, blank2))
 
@@ -420,8 +422,8 @@ class Dump(object):
                 except:
                     warnFile = True
 
-                    _ = unicodeencode(re.sub(r"[^\w]", UNSAFE_DUMP_FILEPATH_REPLACEMENT, unsafeSQLIdentificatorNaming(db)))
+                    _ = re.sub(r"[^\w]", UNSAFE_DUMP_FILEPATH_REPLACEMENT, unsafeSQLIdentificatorNaming(db))
-                    dumpDbPath = os.path.join(conf.dumpPath, "%s-%s" % (_, hashlib.md5(unicodeencode(db)).hexdigest()[:8]))
+                    dumpDbPath = os.path.join(conf.dumpPath, "%s-%s" % (_, hashlib.md5(getBytes(db)).hexdigest()[:8]))
 
                     if not os.path.isdir(dumpDbPath):
                         try:
@@ -454,8 +456,8 @@ class Dump(object):
 
                     _ = re.sub(r"[^\w]", UNSAFE_DUMP_FILEPATH_REPLACEMENT, normalizeUnicode(unsafeSQLIdentificatorNaming(table)))
                     if len(_) < len(table) or IS_WIN and table.upper() in WINDOWS_RESERVED_NAMES:
-                        _ = unicodeencode(re.sub(r"[^\w]", UNSAFE_DUMP_FILEPATH_REPLACEMENT, unsafeSQLIdentificatorNaming(table)))
+                        _ = re.sub(r"[^\w]", UNSAFE_DUMP_FILEPATH_REPLACEMENT, unsafeSQLIdentificatorNaming(table))
-                        dumpFileName = os.path.join(dumpDbPath, "%s-%s.%s" % (_, hashlib.md5(unicodeencode(table)).hexdigest()[:8], conf.dumpFormat.lower()))
+                        dumpFileName = os.path.join(dumpDbPath, "%s-%s.%s" % (_, hashlib.md5(getBytes(table)).hexdigest()[:8], conf.dumpFormat.lower()))
                     else:
                         dumpFileName = os.path.join(dumpDbPath, "%s.%s" % (_, conf.dumpFormat.lower()))
             else:
@@ -702,7 +704,7 @@ class Dump(object):
 
             self.dbTableColumns(_)
 
-    def query(self, query, queryRes):
+    def sqlQuery(self, query, queryRes):
         self.string(query, queryRes, content_type=CONTENT_TYPE.SQL_QUERY)
 
     def rFile(self, fileData):

lib/core/enums.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
@@ -156,13 +156,17 @@ class HASH:
 
 # Reference: http://www.zytrax.com/tech/web/mobile_ids.html
 class MOBILES:
-    BLACKBERRY = ("BlackBerry 9900", "Mozilla/5.0 (BlackBerry; U; BlackBerry 9900; en) AppleWebKit/534.11+ (KHTML, like Gecko) Version/7.1.0.346 Mobile Safari/534.11+")
+    BLACKBERRY = ("BlackBerry Z10", "Mozilla/5.0 (BB10; Kbd) AppleWebKit/537.35+ (KHTML, like Gecko) Version/10.3.3.2205 Mobile Safari/537.35+")
-    GALAXY = ("Samsung Galaxy S", "Mozilla/5.0 (Linux; U; Android 2.2; en-US; SGH-T959D Build/FROYO) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1")
+    GALAXY = ("Samsung Galaxy S7", "Mozilla/5.0 (Linux; Android 7.0; SM-G930V Build/NRD90M) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.125 Mobile Safari/537.36")
     HP = ("HP iPAQ 6365", "Mozilla/4.0 (compatible; MSIE 4.01; Windows CE; PPC; 240x320; HP iPAQ h6300)")
-    HTC = ("HTC Sensation", "Mozilla/5.0 (Linux; U; Android 4.0.3; de-ch; HTC Sensation Build/IML74K) AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 Mobile Safari/534.30")
+    HTC = ("HTC 10", "Mozilla/5.0 (Linux; Android 8.0.0; HTC 10 Build/OPR1.170623.027) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Mobile Safari/537.36")
-    IPHONE = ("Apple iPhone 4s", "Mozilla/5.0 (iPhone; CPU iPhone OS 5_1 like Mac OS X) AppleWebKit/534.46 (KHTML, like Gecko) Version/5.1 Mobile/9B179 Safari/7534.48.3")
+    HUAWEI = ("Huawei P8", "Mozilla/5.0 (Linux; Android 4.4.4; HUAWEI H891L Build/HuaweiH891L) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/33.0.0.0 Mobile Safari/537.36")
+    IPHONE = ("Apple iPhone 8", "Mozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X) AppleWebKit/604.1.38 (KHTML, like Gecko) Version/11.0 Mobile/15A372 Safari/604.1")
+    LUMIA = ("Microsoft Lumia 950", "Mozilla/5.0 (Windows Phone 10.0; Android 6.0.1; Microsoft; Lumia 950) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Mobile Safari/537.36 Edge/15.14977")
     NEXUS = ("Google Nexus 7", "Mozilla/5.0 (Linux; Android 4.1.1; Nexus 7 Build/JRO03D) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.166 Safari/535.19")
     NOKIA = ("Nokia N97", "Mozilla/5.0 (SymbianOS/9.4; Series60/5.0 NokiaN97-1/10.0.012; Profile/MIDP-2.1 Configuration/CLDC-1.1; en-us) AppleWebKit/525 (KHTML, like Gecko) WicKed/7.1.12344")
+    PIXEL = ("Google Pixel", "Mozilla/5.0 (Linux; Android 8.0.0; Pixel Build/OPR3.170623.013) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.111 Mobile Safari/537.36")
+    XIAOMI = ("Xiaomi Mi 3", "Mozilla/5.0 (Linux; U; Android 4.4.4; en-gb; MI 3W Build/KTU84P) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/39.0.0.0 Mobile Safari/537.36 XiaoMi/MiuiBrowser/2.1.1")
 
 class PROXY_TYPE:
     HTTP = "HTTP"
@@ -227,6 +231,7 @@ class HASHDB_KEYS:
     DBMS = "DBMS"
     DBMS_FORK = "DBMS_FORK"
     CHECK_WAF_RESULT = "CHECK_WAF_RESULT"
+    CHECK_NULL_CONNECTION_RESULT = "CHECK_NULL_CONNECTION_RESULT"
     CONF_TMP_PATH = "CONF_TMP_PATH"
     KB_ABS_FILE_PATHS = "KB_ABS_FILE_PATHS"
     KB_BRUTE_COLUMNS = "KB_BRUTE_COLUMNS"
@@ -248,7 +253,7 @@ class PAYLOAD:
         2: "error-based",
         3: "inline query",
         4: "stacked queries",
-        5: "AND/OR time-based blind",
+        5: "time-based blind",
         6: "UNION query",
     }
 
@@ -373,6 +378,7 @@ class MKSTEMP_PREFIX:
     COOKIE_JAR = "sqlmapcookiejar-"
     BIG_ARRAY = "sqlmapbigarray-"
     SPECIFIC_RESPONSE = "sqlmapresponse-"
+    PREPROCESS = "sqlmappreprocess-"
 
 class TIMEOUT_STATE:
     NORMAL = 0
@@ -381,4 +387,4 @@ class TIMEOUT_STATE:
 
 class HINT:
     PREPEND = 0
-    APPEND = 1
+    APPEND = 1

lib/core/exception.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)

lib/core/log.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)

lib/core/option.py

@@ -1,11 +1,10 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """
 
-import cookielib
 import glob
 import inspect
 import logging
@@ -13,13 +12,10 @@ import os
 import random
 import re
 import socket
-import string
 import sys
 import tempfile
 import threading
 import time
-import urllib2
-import urlparse
 
 import lib.controller.checks
 import lib.core.common
@@ -36,11 +32,14 @@ from lib.core.common import dataToStdout
 from lib.core.common import decodeStringEscape
 from lib.core.common import getPublicTypeMembers
 from lib.core.common import getSafeExString
+from lib.core.common import getUnicode
+from lib.core.common import filterNone
 from lib.core.common import findLocalPort
 from lib.core.common import findPageForms
 from lib.core.common import getConsoleWidth
 from lib.core.common import getFileItems
 from lib.core.common import getFileType
+from lib.core.common import intersect
 from lib.core.common import normalizePath
 from lib.core.common import ntToPosixSlashes
 from lib.core.common import openFile
@@ -53,12 +52,14 @@ from lib.core.common import readInput
 from lib.core.common import resetCookieJar
 from lib.core.common import runningAsAdmin
 from lib.core.common import safeExpandUser
+from lib.core.common import safeFilepathEncode
 from lib.core.common import saveConfig
 from lib.core.common import setColor
 from lib.core.common import setOptimize
 from lib.core.common import setPaths
 from lib.core.common import singleTimeWarnMessage
 from lib.core.common import urldecode
+from lib.core.compat import xrange
 from lib.core.data import conf
 from lib.core.data import kb
 from lib.core.data import logger
@@ -66,6 +67,7 @@ from lib.core.data import mergedOptions
 from lib.core.data import queries
 from lib.core.datatype import AttribDict
 from lib.core.datatype import InjectionDict
+from lib.core.datatype import OrderedSet
 from lib.core.defaults import defaults
 from lib.core.dicts import DBMS_DICT
 from lib.core.dicts import DUMP_REPLACEMENTS
@@ -75,6 +77,7 @@ from lib.core.enums import CUSTOM_LOGGING
 from lib.core.enums import DUMP_FORMAT
 from lib.core.enums import HTTP_HEADER
 from lib.core.enums import HTTPMETHOD
+from lib.core.enums import MKSTEMP_PREFIX
 from lib.core.enums import MOBILES
 from lib.core.enums import OPTION_TYPE
 from lib.core.enums import PAYLOAD
@@ -121,7 +124,6 @@ from lib.core.settings import SQLMAP_ENVIRONMENT_PREFIX
 from lib.core.settings import SUPPORTED_DBMS
 from lib.core.settings import SUPPORTED_OS
 from lib.core.settings import TIME_DELAY_CANDIDATES
-from lib.core.settings import UNICODE_ENCODING
 from lib.core.settings import UNION_CHAR_REGEX
 from lib.core.settings import UNKNOWN_DBMS_VERSION
 from lib.core.settings import URI_INJECTABLE_REGEX
@@ -136,6 +138,7 @@ from lib.request.basic import checkCharEncoding
 from lib.request.connect import Connect as Request
 from lib.request.dns import DNSServer
 from lib.request.basicauthhandler import SmartHTTPBasicAuthHandler
+from lib.request.chunkedhandler import ChunkedHandler
 from lib.request.httpshandler import HTTPSHandler
 from lib.request.pkihandler import HTTPSPKIAuthHandler
 from lib.request.rangehandler import HTTPRangeHandler
@@ -146,16 +149,20 @@ from lib.utils.crawler import crawl
 from lib.utils.deps import checkDependencies
 from lib.utils.search import search
 from lib.utils.purge import purge
+from thirdparty import six
 from thirdparty.keepalive import keepalive
 from thirdparty.multipart import multipartpost
-from thirdparty.oset.pyoset import oset
+from thirdparty.six.moves import http_client as _http_client
+from thirdparty.six.moves import http_cookiejar as _http_cookiejar
+from thirdparty.six.moves import urllib as _urllib
 from thirdparty.socks import socks
 from xml.etree.ElementTree import ElementTree
 
-authHandler = urllib2.BaseHandler()
+authHandler = _urllib.request.BaseHandler()
+chunkedHandler = ChunkedHandler()
 httpsHandler = HTTPSHandler()
 keepAliveHandler = keepalive.HTTPHandler()
-proxyHandler = urllib2.ProxyHandler()
+proxyHandler = _urllib.request.ProxyHandler()
 redirectHandler = SmartRedirectHandler()
 rangeHandler = HTTPRangeHandler()
 multipartPostHandler = multipartpost.MultipartPostHandler()
@@ -223,7 +230,7 @@ def _setMultipleTargets():
         errMsg = "the specified list of targets does not exist"
         raise SqlmapFilePathException(errMsg)
 
-    if os.path.isfile(conf.logFile):
+    if checkFile(conf.logFile, False):
         for target in parseRequestFile(conf.logFile):
             url, _, data, _, _ = target
             key = re.sub(r"(\w+=)[^%s ]*" % (conf.paramDel or DEFAULT_GET_POST_DELIMITER), r"\g<1>", "%s %s" % (url, data))
@@ -286,27 +293,30 @@ def _setRequestFromFile():
     """
 
     if conf.requestFile:
-        conf.requestFile = safeExpandUser(conf.requestFile)
+        for requestFile in re.split(PARAMETER_SPLITTING_REGEX, conf.requestFile):
-        seen = set()
+            requestFile = safeExpandUser(requestFile)
+            seen = set()
 
-        if not os.path.isfile(conf.requestFile):
+            if not checkFile(requestFile, False):
-            errMsg = "specified HTTP request file '%s' " % conf.requestFile
+                errMsg = "specified HTTP request file '%s' " % requestFile
-            errMsg += "does not exist"
+                errMsg += "does not exist"
-            raise SqlmapFilePathException(errMsg)
+                raise SqlmapFilePathException(errMsg)
 
-        infoMsg = "parsing HTTP request from '%s'" % conf.requestFile
+            infoMsg = "parsing HTTP request from '%s'" % requestFile
-        logger.info(infoMsg)
+            logger.info(infoMsg)
 
-        for target in parseRequestFile(conf.requestFile):
+            for target in parseRequestFile(requestFile):
-            url = target[0]
+                url = target[0]
-            if url not in seen:
+                if url not in seen:
-                kb.targets.add(target)
+                    kb.targets.add(target)
-                seen.add(url)
+                    if len(kb.targets) > 1:
+                        conf.multipleTargets = True
+                    seen.add(url)
 
     if conf.secondReq:
         conf.secondReq = safeExpandUser(conf.secondReq)
 
-        if not os.path.isfile(conf.secondReq):
+        if not checkFile(conf.secondReq, False):
             errMsg = "specified second-order HTTP request file '%s' " % conf.secondReq
             errMsg += "does not exist"
             raise SqlmapFilePathException(errMsg)
@@ -408,7 +418,7 @@ def _setBulkMultipleTargets():
     infoMsg = "parsing multiple targets list from '%s'" % conf.bulkFile
     logger.info(infoMsg)
 
-    if not os.path.isfile(conf.bulkFile):
+    if not checkFile(conf.bulkFile, False):
         errMsg = "the specified bulk file "
         errMsg += "does not exist"
         raise SqlmapFilePathException(errMsg)
@@ -547,11 +557,11 @@ def _setMetasploit():
 
     if conf.msfPath:
         for path in (conf.msfPath, os.path.join(conf.msfPath, "bin")):
-            if any(os.path.exists(normalizePath(os.path.join(path, _))) for _ in ("msfcli", "msfconsole")):
+            if any(os.path.exists(normalizePath(os.path.join(path, "%s%s" % (_, ".bat" if IS_WIN else "")))) for _ in ("msfcli", "msfconsole")):
                 msfEnvPathExists = True
-                if all(os.path.exists(normalizePath(os.path.join(path, _))) for _ in ("msfvenom",)):
+                if all(os.path.exists(normalizePath(os.path.join(path, "%s%s" % (_, ".bat" if IS_WIN else "")))) for _ in ("msfvenom",)):
                     kb.oldMsf = False
-                elif all(os.path.exists(normalizePath(os.path.join(path, _))) for _ in ("msfencode", "msfpayload")):
+                elif all(os.path.exists(normalizePath(os.path.join(path, "%s%s" % (_, ".bat" if IS_WIN else "")))) for _ in ("msfencode", "msfpayload")):
                     kb.oldMsf = True
                 else:
                     msfEnvPathExists = False
@@ -586,11 +596,11 @@ def _setMetasploit():
         for envPath in envPaths:
             envPath = envPath.replace(";", "")
 
-            if any(os.path.exists(normalizePath(os.path.join(envPath, _))) for _ in ("msfcli", "msfconsole")):
+            if any(os.path.exists(normalizePath(os.path.join(envPath, "%s%s" % (_, ".bat" if IS_WIN else "")))) for _ in ("msfcli", "msfconsole")):
                 msfEnvPathExists = True
-                if all(os.path.exists(normalizePath(os.path.join(envPath, _))) for _ in ("msfvenom",)):
+                if all(os.path.exists(normalizePath(os.path.join(envPath, "%s%s" % (_, ".bat" if IS_WIN else "")))) for _ in ("msfvenom",)):
                     kb.oldMsf = False
-                elif all(os.path.exists(normalizePath(os.path.join(envPath, _))) for _ in ("msfencode", "msfpayload")):
+                elif all(os.path.exists(normalizePath(os.path.join(envPath, "%s%s" % (_, ".bat" if IS_WIN else "")))) for _ in ("msfencode", "msfpayload")):
                     kb.oldMsf = True
                 else:
                     msfEnvPathExists = False
@@ -654,7 +664,7 @@ def _setTechnique():
     validTechniques = sorted(getPublicTypeMembers(PAYLOAD.TECHNIQUE), key=lambda x: x[1])
     validLetters = [_[0][0].upper() for _ in validTechniques]
 
-    if conf.tech and isinstance(conf.tech, basestring):
+    if conf.tech and isinstance(conf.tech, six.string_types):
         _ = []
 
         for letter in conf.tech.upper():
@@ -732,8 +742,8 @@ def _setTamperingFunctions():
         for script in re.split(PARAMETER_SPLITTING_REGEX, conf.tamper):
             found = False
 
-            path = paths.SQLMAP_TAMPER_PATH.encode(sys.getfilesystemencoding() or UNICODE_ENCODING)
+            path = safeFilepathEncode(paths.SQLMAP_TAMPER_PATH)
-            script = script.strip().encode(sys.getfilesystemencoding() or UNICODE_ENCODING)
+            script = safeFilepathEncode(script.strip())
 
             try:
                 if not script:
@@ -768,9 +778,9 @@ def _setTamperingFunctions():
                 sys.path.insert(0, dirname)
 
             try:
-                module = __import__(filename[:-3].encode(sys.getfilesystemencoding() or UNICODE_ENCODING))
+                module = __import__(safeFilepathEncode(filename[:-3]))
             except Exception as ex:
-                raise SqlmapSyntaxException("cannot import tamper module '%s' (%s)" % (filename[:-3], getSafeExString(ex)))
+                raise SqlmapSyntaxException("cannot import tamper module '%s' (%s)" % (getUnicode(filename[:-3]), getSafeExString(ex)))
 
             priority = PRIORITY.NORMAL if not hasattr(module, "__priority__") else module.__priority__
 
@@ -778,7 +788,7 @@ def _setTamperingFunctions():
                 if name == "tamper" and inspect.getargspec(function).args and inspect.getargspec(function).keywords == "kwargs":
                     found = True
                     kb.tamperFunctions.append(function)
-                    function.func_name = module.__name__
+                    function.__name__ = module.__name__
 
                     if check_priority and priority > last_priority:
                         message = "it appears that you might have mixed "
@@ -804,7 +814,7 @@ def _setTamperingFunctions():
                         function()
                     except Exception as ex:
                         errMsg = "error occurred while checking dependencies "
-                        errMsg += "for tamper module '%s' ('%s')" % (filename[:-3], getSafeExString(ex))
+                        errMsg += "for tamper module '%s' ('%s')" % (getUnicode(filename[:-3]), getSafeExString(ex))
                         raise SqlmapGenericException(errMsg)
 
             if not found:
@@ -824,6 +834,80 @@ def _setTamperingFunctions():
             for _, function in priorities:
                 kb.tamperFunctions.append(function)
 
+def _setPreprocessFunctions():
+    """
+    Loads preprocess functions from given script(s)
+    """
+
+    if conf.preprocess:
+        for script in re.split(PARAMETER_SPLITTING_REGEX, conf.preprocess):
+            found = False
+
+            script = safeFilepathEncode(script.strip())
+
+            try:
+                if not script:
+                    continue
+
+                if not os.path.exists(script):
+                    errMsg = "preprocess script '%s' does not exist" % script
+                    raise SqlmapFilePathException(errMsg)
+
+                elif not script.endswith(".py"):
+                    errMsg = "preprocess script '%s' should have an extension '.py'" % script
+                    raise SqlmapSyntaxException(errMsg)
+            except UnicodeDecodeError:
+                errMsg = "invalid character provided in option '--preprocess'"
+                raise SqlmapSyntaxException(errMsg)
+
+            dirname, filename = os.path.split(script)
+            dirname = os.path.abspath(dirname)
+
+            infoMsg = "loading preprocess module '%s'" % filename[:-3]
+            logger.info(infoMsg)
+
+            if not os.path.exists(os.path.join(dirname, "__init__.py")):
+                errMsg = "make sure that there is an empty file '__init__.py' "
+                errMsg += "inside of preprocess scripts directory '%s'" % dirname
+                raise SqlmapGenericException(errMsg)
+
+            if dirname not in sys.path:
+                sys.path.insert(0, dirname)
+
+            try:
+                module = __import__(safeFilepathEncode(filename[:-3]))
+            except Exception as ex:
+                raise SqlmapSyntaxException("cannot import preprocess module '%s' (%s)" % (getUnicode(filename[:-3]), getSafeExString(ex)))
+
+            for name, function in inspect.getmembers(module, inspect.isfunction):
+                if name == "preprocess" and inspect.getargspec(function).args and all(_ in inspect.getargspec(function).args for _ in ("page", "headers", "code")):
+                    found = True
+
+                    kb.preprocessFunctions.append(function)
+                    function.__name__ = module.__name__
+
+                    break
+
+            if not found:
+                errMsg = "missing function 'preprocess(page, headers=None, code=None)' "
+                errMsg += "in preprocess script '%s'" % script
+                raise SqlmapGenericException(errMsg)
+            else:
+                try:
+                    _, _, _ = function("", {}, None)
+                except:
+                    handle, filename = tempfile.mkstemp(prefix=MKSTEMP_PREFIX.PREPROCESS, suffix=".py")
+                    os.close(handle)
+
+                    open(filename, "w+b").write("#!/usr/bin/env\n\ndef preprocess(page, headers=None, code=None):\n    return page, headers, code\n")
+                    open(os.path.join(os.path.dirname(filename), "__init__.py"), "w+b").write("pass")
+
+                    errMsg = "function 'preprocess(page, headers=None, code=None)' "
+                    errMsg += "in preprocess script '%s' " % script
+                    errMsg += "should return a tuple '(page, headers, code)' "
+                    errMsg += "(Note: find template script at '%s')" % filename
+                    raise SqlmapGenericException(errMsg)
+
 def _setWafFunctions():
     """
     Loads WAF/IPS detecting functions from script(s)
@@ -846,9 +930,9 @@ def _setWafFunctions():
             try:
                 if filename[:-3] in sys.modules:
                     del sys.modules[filename[:-3]]
-                module = __import__(filename[:-3].encode(sys.getfilesystemencoding() or UNICODE_ENCODING))
+                module = __import__(safeFilepathEncode(filename[:-3]))
             except ImportError as ex:
-                raise SqlmapSyntaxException("cannot import WAF script '%s' (%s)" % (filename[:-3], getSafeExString(ex)))
+                raise SqlmapSyntaxException("cannot import WAF script '%s' (%s)" % (getUnicode(filename[:-3]), getSafeExString(ex)))
 
             _ = dict(inspect.getmembers(module))
             if "detect" not in _:
@@ -974,7 +1058,7 @@ def _setHTTPHandlers():
         logger.debug(debugMsg)
 
         try:
-            _ = urlparse.urlsplit(conf.proxy)
+            _ = _urllib.parse.urlsplit(conf.proxy)
         except Exception as ex:
             errMsg = "invalid proxy address '%s' ('%s')" % (conf.proxy, getSafeExString(ex))
             raise SqlmapSyntaxException(errMsg)
@@ -1010,10 +1094,14 @@ def _setHTTPHandlers():
         if scheme in (PROXY_TYPE.SOCKS4, PROXY_TYPE.SOCKS5):
             proxyHandler.proxies = {}
 
+            if scheme == PROXY_TYPE.SOCKS4:
+                warnMsg = "SOCKS4 does not support resolving (DNS) names (i.e. causing DNS leakage)"
+                singleTimeWarnMessage(warnMsg)
+
             socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS5 if scheme == PROXY_TYPE.SOCKS5 else socks.PROXY_TYPE_SOCKS4, hostname, port, username=username, password=password)
-            socks.wrapmodule(urllib2)
+            socks.wrapmodule(_http_client)
         else:
-            socks.unwrapmodule(urllib2)
+            socks.unwrapmodule(_http_client)
 
             if conf.proxyCred:
                 # Reference: http://stackoverflow.com/questions/34079/how-to-specify-an-authenticated-proxy-for-a-python-http-connection
@@ -1029,16 +1117,16 @@ def _setHTTPHandlers():
     debugMsg = "creating HTTP requests opener object"
     logger.debug(debugMsg)
 
-    handlers = filter(None, [multipartPostHandler, proxyHandler if proxyHandler.proxies else None, authHandler, redirectHandler, rangeHandler, httpsHandler])
+    handlers = filterNone([multipartPostHandler, proxyHandler if proxyHandler.proxies else None, authHandler, redirectHandler, rangeHandler, chunkedHandler if conf.chunked else None, httpsHandler])
 
     if not conf.dropSetCookie:
         if not conf.loadCookies:
-            conf.cj = cookielib.CookieJar()
+            conf.cj = _http_cookiejar.CookieJar()
         else:
-            conf.cj = cookielib.MozillaCookieJar()
+            conf.cj = _http_cookiejar.MozillaCookieJar()
             resetCookieJar(conf.cj)
 
-        handlers.append(urllib2.HTTPCookieProcessor(conf.cj))
+        handlers.append(_urllib.request.HTTPCookieProcessor(conf.cj))
 
     # Reference: http://www.w3.org/Protocols/rfc2616/rfc2616-sec8.html
     if conf.keepAlive:
@@ -1054,8 +1142,9 @@ def _setHTTPHandlers():
         else:
             handlers.append(keepAliveHandler)
 
-    opener = urllib2.build_opener(*handlers)
+    opener = _urllib.request.build_opener(*handlers)
-    urllib2.install_opener(opener)
+    opener.addheaders = []  # Note: clearing default "User-Agent: Python-urllib/X.Y"
+    _urllib.request.install_opener(opener)
 
 def _setSafeVisit():
     """
@@ -1068,14 +1157,14 @@ def _setSafeVisit():
         checkFile(conf.safeReqFile)
 
         raw = readCachedFileContent(conf.safeReqFile)
-        match = re.search(r"\A([A-Z]+) ([^ ]+) HTTP/[0-9.]+\Z", raw[:raw.find('\n')])
+        match = re.search(r"\A([A-Z]+) ([^ ]+) HTTP/[0-9.]+\Z", raw.split('\n')[0].strip())
 
         if match:
             kb.safeReq.method = match.group(1)
             kb.safeReq.url = match.group(2)
             kb.safeReq.headers = {}
 
-            for line in raw[raw.find('\n') + 1:].split('\n'):
+            for line in raw.split('\n')[1:]:
                 line = line.strip()
                 if line and ':' in line:
                     key, value = line.split(':', 1)
@@ -1087,7 +1176,7 @@ def _setSafeVisit():
                             if value.endswith(":443"):
                                 scheme = "https"
                             value = "%s://%s" % (scheme, value)
-                        kb.safeReq.url = urlparse.urljoin(value, kb.safeReq.url)
+                        kb.safeReq.url = _urllib.parse.urljoin(value, kb.safeReq.url)
                 else:
                     break
 
@@ -1210,7 +1299,7 @@ def _setHTTPAuthentication():
         conf.authUsername = aCredRegExp.group(1)
         conf.authPassword = aCredRegExp.group(2)
 
-        kb.passwordMgr = urllib2.HTTPPasswordMgrWithDefaultRealm()
+        kb.passwordMgr = _urllib.request.HTTPPasswordMgrWithDefaultRealm()
 
         _setAuthCred()
 
@@ -1218,7 +1307,7 @@ def _setHTTPAuthentication():
             authHandler = SmartHTTPBasicAuthHandler(kb.passwordMgr)
 
         elif authType == AUTH_TYPE.DIGEST:
-            authHandler = urllib2.HTTPDigestAuthHandler(kb.passwordMgr)
+            authHandler = _urllib.request.HTTPDigestAuthHandler(kb.passwordMgr)
 
         elif authType == AUTH_TYPE.NTLM:
             try:
@@ -1277,28 +1366,32 @@ def _setHTTPUserAgent():
           file choosed as user option
     """
 
+    debugMsg = "setting the HTTP User-Agent header"
+    logger.debug(debugMsg)
+
     if conf.mobile:
-        message = "which smartphone do you want sqlmap to imitate "
+        if conf.randomAgent:
-        message += "through HTTP User-Agent header?\n"
+            _ = random.sample([_[1] for _ in getPublicTypeMembers(MOBILES, True)], 1)[0]
-        items = sorted(getPublicTypeMembers(MOBILES, True))
+            conf.httpHeaders.append((HTTP_HEADER.USER_AGENT, _))
+        else:
+            message = "which smartphone do you want sqlmap to imitate "
+            message += "through HTTP User-Agent header?\n"
+            items = sorted(getPublicTypeMembers(MOBILES, True))
 
-        for count in xrange(len(items)):
+            for count in xrange(len(items)):
-            item = items[count]
+                item = items[count]
-            message += "[%d] %s%s\n" % (count + 1, item[0], " (default)" if item == MOBILES.IPHONE else "")
+                message += "[%d] %s%s\n" % (count + 1, item[0], " (default)" if item == MOBILES.IPHONE else "")
 
-        test = readInput(message.rstrip('\n'), default=items.index(MOBILES.IPHONE) + 1)
+            test = readInput(message.rstrip('\n'), default=items.index(MOBILES.IPHONE) + 1)
 
-        try:
+            try:
-            item = items[int(test) - 1]
+                item = items[int(test) - 1]
-        except:
+            except:
-            item = MOBILES.IPHONE
+                item = MOBILES.IPHONE
 
-        conf.httpHeaders.append((HTTP_HEADER.USER_AGENT, item[1]))
+            conf.httpHeaders.append((HTTP_HEADER.USER_AGENT, item[1]))
 
     elif conf.agent:
-        debugMsg = "setting the HTTP User-Agent header"
-        logger.debug(debugMsg)
-
         conf.httpHeaders.append((HTTP_HEADER.USER_AGENT, conf.agent))
 
     elif not conf.randomAgent:
@@ -1376,7 +1469,7 @@ def _setHostname():
 
     if conf.url:
         try:
-            conf.hostname = urlparse.urlsplit(conf.url).netloc.split(':')[0]
+            conf.hostname = _urllib.parse.urlsplit(conf.url).netloc.split(':')[0]
         except ValueError as ex:
             errMsg = "problem occurred while "
             errMsg += "parsing an URL '%s' ('%s')" % (conf.url, getSafeExString(ex))
@@ -1412,6 +1505,41 @@ def _checkDependencies():
     if conf.dependencies:
         checkDependencies()
 
+def _createHomeDirectories():
+    """
+    Creates directories inside sqlmap's home directory
+    """
+
+    for context in "output", "history":
+        directory = paths["SQLMAP_%s_PATH" % context.upper()]
+        try:
+            if not os.path.isdir(directory):
+                os.makedirs(directory)
+
+            _ = os.path.join(directory, randomStr())
+            open(_, "w+b").close()
+            os.remove(_)
+
+            if conf.outputDir and context == "output":
+                warnMsg = "using '%s' as the %s directory" % (directory, context)
+                logger.warn(warnMsg)
+        except (OSError, IOError) as ex:
+            try:
+                tempDir = tempfile.mkdtemp(prefix="sqlmap%s" % context)
+            except Exception as _:
+                errMsg = "unable to write to the temporary directory ('%s'). " % _
+                errMsg += "Please make sure that your disk is not full and "
+                errMsg += "that you have sufficient write permissions to "
+                errMsg += "create temporary files and/or directories"
+                raise SqlmapSystemException(errMsg)
+
+            warnMsg = "unable to %s %s directory " % ("create" if not os.path.isdir(directory) else "write to the", context)
+            warnMsg += "'%s' (%s). " % (directory, getUnicode(ex))
+            warnMsg += "Using temporary directory '%s' instead" % getUnicode(tempDir)
+            logger.warn(warnMsg)
+
+            paths["SQLMAP_%s_PATH" % context.upper()] = tempDir
+
 def _createTemporaryDirectory():
     """
     Creates temporary directory for this run.
@@ -1489,6 +1617,13 @@ def _cleanupOptions():
     else:
         conf.testParameter = []
 
+    if conf.base64Parameter:
+        conf.base64Parameter = urldecode(conf.base64Parameter)
+        conf.base64Parameter = conf.base64Parameter.replace(" ", "")
+        conf.base64Parameter = re.split(PARAMETER_SPLITTING_REGEX, conf.base64Parameter)
+    else:
+        conf.base64Parameter = []
+
     if conf.agent:
         conf.agent = re.sub(r"[\r\n]", "", conf.agent)
 
@@ -1575,13 +1710,13 @@ def _cleanupOptions():
             re.compile(conf.csrfToken)
 
             if re.escape(conf.csrfToken) != conf.csrfToken:
-                message = "provided value for option '--csrf-token' is a regular expression? [Y/n] "
+                message = "provided value for option '--csrf-token' is a regular expression? [y/N] "
-                if not readInput(message, default='Y', boolean=True):
+                if not readInput(message, default='N', boolean=True):
                     conf.csrfToken = re.escape(conf.csrfToken)
         except re.error:
             conf.csrfToken = re.escape(conf.csrfToken)
         finally:
-            class _(unicode):
+            class _(six.text_type):
                 pass
             conf.csrfToken = _(conf.csrfToken)
             conf.csrfToken._original = original
@@ -1616,7 +1751,7 @@ def _cleanupOptions():
     if conf.csvDel:
         conf.csvDel = decodeStringEscape(conf.csvDel)
 
-    if conf.torPort and isinstance(conf.torPort, basestring) and conf.torPort.isdigit():
+    if conf.torPort and hasattr(conf.torPort, "isdigit") and conf.torPort.isdigit():
         conf.torPort = int(conf.torPort)
 
     if conf.torType:
@@ -1665,8 +1800,8 @@ def _cleanupEnvironment():
     Cleanup environment (e.g. from leftovers after --sqlmap-shell).
     """
 
-    if issubclass(urllib2.socket.socket, socks.socksocket):
+    if issubclass(_http_client.socket.socket, socks.socksocket):
-        socks.unwrapmodule(urllib2)
+        socks.unwrapmodule(_http_client)
 
     if hasattr(socket, "_ready"):
         socket._ready.clear()
@@ -1763,6 +1898,7 @@ def _setKnowledgeBaseAttributes(flushAll=True):
     kb.connErrorChoice = None
     kb.connErrorCounter = 0
     kb.cookieEncodeChoice = None
+    kb.copyExecTest = None
     kb.counters = {}
     kb.customInjectionMark = CUSTOM_INJECTION_MARK_CHAR
     kb.data = AttribDict()
@@ -1897,9 +2033,10 @@ def _setKnowledgeBaseAttributes(flushAll=True):
         kb.headerPaths = {}
         kb.keywords = set(getFileItems(paths.SQL_KEYWORDS))
         kb.passwordMgr = None
+        kb.preprocessFunctions = []
         kb.skipVulnHost = None
         kb.tamperFunctions = []
-        kb.targets = oset()
+        kb.targets = OrderedSet()
         kb.testedParams = set()
         kb.userAgents = None
         kb.vainRun = True
@@ -2192,7 +2329,11 @@ def _setTorSocksProxySettings():
 
     # SOCKS5 to prevent DNS leaks (http://en.wikipedia.org/wiki/Tor_%28anonymity_network%29)
     socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS5 if conf.torType == PROXY_TYPE.SOCKS5 else socks.PROXY_TYPE_SOCKS4, LOCALHOST, port)
-    socks.wrapmodule(urllib2)
+    socks.wrapmodule(_http_client)
+
+def _setHttpChunked():
+    if conf.chunked and conf.data:
+        _http_client.HTTPConnection._set_content_length = lambda self, a, b: None
 
 def _checkWebSocket():
     if conf.url and (conf.url.startswith("ws:/") or conf.url.startswith("wss:/")):
@@ -2281,6 +2422,10 @@ def _basicOptionValidation():
         errMsg = "switch '--dump' is incompatible with switch '--search'"
         raise SqlmapSyntaxException(errMsg)
 
+    if conf.chunked and not any((conf.data, conf.requestFile, conf.forms)):
+        errMsg = "switch '--chunked' requires usage of (POST) options/switches '--data', '-r' or '--forms'"
+        raise SqlmapSyntaxException(errMsg)
+
     if conf.api and not conf.configFile:
         errMsg = "switch '--api' requires usage of option '-c'"
         raise SqlmapSyntaxException(errMsg)
@@ -2416,8 +2561,14 @@ def _basicOptionValidation():
         raise SqlmapSyntaxException(errMsg)
 
     if conf.skip and conf.testParameter:
-        errMsg = "option '--skip' is incompatible with option '-p'"
+        if intersect(conf.skip, conf.testParameter):
-        raise SqlmapSyntaxException(errMsg)
+            errMsg = "option '--skip' is incompatible with option '-p'"
+            raise SqlmapSyntaxException(errMsg)
+
+    if conf.rParam and conf.testParameter:
+        if intersect(conf.rParam, conf.testParameter):
+            errMsg = "option '--randomize' is incompatible with option '-p'"
+            raise SqlmapSyntaxException(errMsg)
 
     if conf.mobile and conf.agent:
         errMsg = "switch '--mobile' is incompatible with option '--user-agent'"
@@ -2439,7 +2590,7 @@ def _basicOptionValidation():
         errMsg = "option '--crack' should be used as a standalone"
         raise SqlmapSyntaxException(errMsg)
 
-    if isinstance(conf.uCols, basestring):
+    if isinstance(conf.uCols, six.string_types):
         if not conf.uCols.isdigit() and ("-" not in conf.uCols or len(conf.uCols.split("-")) != 2):
             errMsg = "value for option '--union-cols' must be a range with hyphon "
             errMsg += "(e.g. 1-10) or integer value (e.g. 5)"
@@ -2493,6 +2644,7 @@ def init():
     _cleanupEnvironment()
     _purge()
     _checkDependencies()
+    _createHomeDirectories()
     _createTemporaryDirectory()
     _basicOptionValidation()
     _setProxyList()
@@ -2502,10 +2654,12 @@ def init():
     _setMultipleTargets()
     _listTamperingFunctions()
     _setTamperingFunctions()
+    _setPreprocessFunctions()
     _setWafFunctions()
     _setTrafficOutputFP()
     _setupHTTPCollector()
     _resolveCrossReferences()
+    _setHttpChunked()
     _checkWebSocket()
 
     parseTargetDirect()

lib/core/optiondict.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
@@ -61,6 +61,7 @@ optDict = {
         "csrfToken": "string",
         "csrfUrl": "string",
         "forceSSL": "boolean",
+        "chunked": "boolean",
         "hpp": "boolean",
         "evalCode": "string",
     },
@@ -149,7 +150,7 @@ optDict = {
         "limitStop": "integer",
         "firstChar": "integer",
         "lastChar": "integer",
-        "query": "string",
+        "sqlQuery": "string",
         "sqlShell": "boolean",
         "sqlFile": "string",
     },
@@ -192,7 +193,6 @@ optDict = {
     },
 
     "General": {
-        # "xmlFile": "string",
         "trafficFile": "string",
         "batch": "boolean",
         "binaryFields": "string",
@@ -211,6 +211,8 @@ optDict = {
         "hexConvert": "boolean",
         "outputDir": "string",
         "parseErrors": "boolean",
+        "preprocess": "string",
+        "repair": "boolean",
         "saveConfig": "string",
         "scope": "string",
         "testFilter": "string",

lib/core/patch.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
@@ -6,9 +6,9 @@ See the file 'LICENSE' for copying permission
 """
 
 import codecs
-import httplib
 
 from lib.core.settings import IS_WIN
+from thirdparty.six.moves import http_client as _http_client
 
 def dirtyPatches():
     """
@@ -16,7 +16,7 @@ def dirtyPatches():
     """
 
     # accept overly long result lines (e.g. SQLi results in HTTP header responses)
-    httplib._MAXLINE = 1 * 1024 * 1024
+    _http_client._MAXLINE = 1 * 1024 * 1024
 
     # add support for inet_pton() on Windows OS
     if IS_WIN:
@@ -24,3 +24,11 @@ def dirtyPatches():
 
     # Reference: https://github.com/nodejs/node/issues/12786#issuecomment-298652440
     codecs.register(lambda name: codecs.lookup("utf-8") if name == "cp65001" else None)
+
+    # Reference: http://bugs.python.org/issue17849
+    if hasattr(_http_client, "LineAndFileWrapper"):
+        def _(self, *args):
+            return self._readline()
+
+        _http_client.LineAndFileWrapper._readline = _http_client.LineAndFileWrapper.readline
+        _http_client.LineAndFileWrapper.readline = _

lib/core/profiling.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)

lib/core/readlineng.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
@@ -56,9 +56,7 @@ if PLATFORM == 'mac' and _readline:
 # http://mail.python.org/pipermail/python-dev/2003-August/037845.html
 # has the original discussion.
 if _readline:
-    try:
+    if not hasattr(_readline, "clear_history"):
-        _readline.clear_history()
-    except AttributeError:
         def clear_history():
             pass
 

lib/core/replication.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)

lib/core/revision.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
@@ -44,9 +44,12 @@ def getRevisionNumber():
             break
 
     if not retVal:
-        process = subprocess.Popen("git rev-parse --verify HEAD", shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+        try:
-        stdout, _ = process.communicate()
+            process = subprocess.Popen("git rev-parse --verify HEAD", shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
-        match = re.search(r"(?i)[0-9a-f]{32}", stdout or "")
+            stdout, _ = process.communicate()
-        retVal = match.group(0) if match else None
+            match = re.search(r"(?i)[0-9a-f]{32}", stdout or "")
+            retVal = match.group(0) if match else None
+        except:
+            pass
 
     return retVal[:7] if retVal else None

lib/core/session.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)

lib/core/settings.py

@@ -1,25 +1,23 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """
 
+import codecs
 import os
 import random
 import re
-import subprocess
 import string
 import sys
-import types
 
-from lib.core.datatype import AttribDict
 from lib.core.enums import DBMS
 from lib.core.enums import DBMS_DIRECTORY_NAME
 from lib.core.enums import OS
 
 # sqlmap version (<major>.<minor>.<month>.<monthly commit>)
-VERSION = "1.3.2.0"
+VERSION = "1.3.5.0"
 TYPE = "dev" if VERSION.count('.') > 2 and VERSION.split('.')[-1] != '0' else "stable"
 TYPE_COLORS = {"dev": 33, "stable": 90, "pip": 34}
 VERSION_STRING = "sqlmap/%s#%s" % ('.'.join(VERSION.split('.')[:-1]) if VERSION.count('.') > 2 and VERSION.split('.')[-1] == '0' else VERSION, TYPE)
@@ -67,6 +65,7 @@ ASTERISK_MARKER = "__ASTERISK_MARK__"
 REPLACEMENT_MARKER = "__REPLACEMENT_MARK__"
 BOUNDED_INJECTION_MARKER = "__BOUNDED_INJECTION_MARK__"
 SAFE_VARIABLE_MARKER = "__SAFE__"
+SAFE_HEX_MARKER = "__SAFE_HEX__"
 
 RANDOM_INTEGER_MARKER = "[RANDINT]"
 RANDOM_STRING_MARKER = "[RANDSTR]"
@@ -100,7 +99,7 @@ MAX_CONSECUTIVE_CONNECTION_ERRORS = 15
 PRECONNECT_CANDIDATE_TIMEOUT = 10
 
 # Servers known to cause issue with pre-connection mechanism (because of lack of multi-threaded support)
-PRECONNECT_INCOMPATIBLE_SERVERS = ("SimpleHTTP",)
+PRECONNECT_INCOMPATIBLE_SERVERS = ("SimpleHTTP", "BaseHTTP")
 
 # Maximum sleep time in "Murphy" (testing) mode
 MAX_MURPHY_SLEEP_TIME = 3
@@ -185,7 +184,7 @@ MAX_TIME_RESPONSES = 200
 MIN_UNION_RESPONSES = 5
 
 # After these number of blanks at the end inference should stop (just in case)
-INFERENCE_BLANK_BREAK = 10
+INFERENCE_BLANK_BREAK = 5
 
 # Use this replacement character for cases when inference is not able to retrieve the proper character value
 INFERENCE_UNKNOWN_CHAR = '?'
@@ -218,19 +217,20 @@ DUMMY_USER_PREFIX = "__dummy__"
 DEFAULT_PAGE_ENCODING = "iso-8859-1"
 
 try:
-    unicode(DEFAULT_PAGE_ENCODING, DEFAULT_PAGE_ENCODING)
+    codecs.lookup(DEFAULT_PAGE_ENCODING)
 except LookupError:
     DEFAULT_PAGE_ENCODING = "utf8"
 
+# Marker for program piped input
+STDIN_PIPE_DASH = '-'
+
 # URL used in dummy runs
 DUMMY_URL = "http://foo/bar?id=1"
 
-# System variables
-IS_WIN = subprocess.mswindows
-
 # The name of the operating system dependent module imported. The following names have currently been registered: 'posix', 'nt', 'mac', 'os2', 'ce', 'java', 'riscos'
 PLATFORM = os.name
 PYVERSION = sys.version.split()[0]
+IS_WIN = PLATFORM == "nt"
 
 # DBMS system databases
 MSSQL_SYSTEM_DBS = ("Northwind", "master", "model", "msdb", "pubs", "tempdb")
@@ -330,18 +330,20 @@ CURRENT_DB = "CD"
 SESSION_SQLITE_FILE = "session.sqlite"
 
 # Regular expressions used for finding file paths in error messages
-FILE_PATH_REGEXES = (r"<b>(?P<result>[^<>]+?)</b> on line \d+", r"in (?P<result>[^<>'\"]+?)['\"]? on line \d+", r"(?:[>(\[\s])(?P<result>[A-Za-z]:[\\/][\w. \\/-]*)", r"(?:[>(\[\s])(?P<result>/\w[/\w.~-]+)", r"href=['\"]file://(?P<result>/[^'\"]+)")
+FILE_PATH_REGEXES = (r"<b>(?P<result>[^<>]+?)</b> on line \d+", r"\bin (?P<result>[^<>'\"]+?)['\"]? on line \d+", r"(?:[>(\[\s])(?P<result>[A-Za-z]:[\\/][\w. \\/-]*)", r"(?:[>(\[\s])(?P<result>/\w[/\w.~-]+)", r"\bhref=['\"]file://(?P<result>/[^'\"]+)", r"\bin <b>(?P<result>[^<]+): line \d+")
 
 # Regular expressions used for parsing error messages (--parse-errors)
 ERROR_PARSING_REGEXES = (
     r"\[Microsoft\]\[ODBC SQL Server Driver\]\[SQL Server\](?P<result>[^<]+)",
     r"<b>[^<]*(fatal|error|warning|exception)[^<]*</b>:?\s*(?P<result>[^<]+)",
     r"(?m)^\s*(fatal|error|warning|exception):?\s*(?P<result>[^\n]+?)$",
+    r"(sql|dbc)[^>'\"]{0,32}(fatal|error|warning|exception)(</b>)?:\s*(?P<result>[^<>]+)",
     r"(?P<result>[^\n>]*SQL Syntax[^\n<]+)",
-    r"<li>Error Type:<br>(?P<result>.+?)</li>",
+    r"(?s)<li>Error Type:<br>(?P<result>.+?)</li>",
     r"CDbCommand (?P<result>[^<>\n]*SQL[^<>\n]+)",
     r"error '[0-9a-f]{8}'((<[^>]+>)|\s)+(?P<result>[^<>]+)",
-    r"\[[^\n\]]+(ODBC|JDBC)[^\n\]]+\](\[[^\]]+\])?(?P<result>[^\n]+(in query expression|\(SQL| at /[^ ]+pdo)[^\n<]+)"
+    r"\[[^\n\]]+(ODBC|JDBC)[^\n\]]+\](\[[^\]]+\])?(?P<result>[^\n]+(in query expression|\(SQL| at /[^ ]+pdo)[^\n<]+)",
+    r"(?P<result>query error: SELECT[^<>]+)"
 )
 
 # Regular expression used for parsing charset info from meta html headers
@@ -378,10 +380,10 @@ URI_HTTP_HEADER = "URI"
 URI_INJECTABLE_REGEX = r"//[^/]*/([^\.*?]+)\Z"
 
 # Regex used for masking sensitive data
-SENSITIVE_DATA_REGEX = r"(\s|=)(?P<result>[^\s=]*%s[^\s]*)\s"
+SENSITIVE_DATA_REGEX = r"(\s|=)(?P<result>[^\s=]*\b%s\b[^\s]*)\s"
 
 # Options to explicitly mask in anonymous (unhandled exception) reports (along with anything carrying the <hostname> inside)
-SENSITIVE_OPTIONS = ("hostname", "answers", "data", "dnsDomain", "googleDork", "authCred", "proxyCred", "tbl", "db", "col", "user", "cookie", "proxy", "fileRead", "fileWrite", "fileDest", "testParameter", "authCred")
+SENSITIVE_OPTIONS = ("hostname", "answers", "data", "dnsDomain", "googleDork", "authCred", "proxyCred", "tbl", "db", "col", "user", "cookie", "proxy", "fileRead", "fileWrite", "fileDest", "testParameter", "authCred", "sqlQuery")
 
 # Maximum number of threads (avoiding connection issues and/or DoS)
 MAX_NUMBER_OF_THREADS = 10
@@ -444,7 +446,7 @@ HASH_MOD_ITEM_DISPLAY = 11
 HASH_EMPTY_PASSWORD_MARKER = "<empty>"
 
 # Maximum integer value
-MAX_INT = sys.maxint
+MAX_INT = sys.maxsize
 
 # Replacement for unsafe characters in dump table filenames
 UNSAFE_DUMP_FILEPATH_REPLACEMENT = '_'
@@ -509,8 +511,6 @@ HTML_TITLE_REGEX = r"<title>(?P<result>[^<]+)</title>"
 # Table used for Base64 conversion in WordPress hash cracking routine
 ITOA64 = "./0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz"
 
-PICKLE_REDUCE_WHITELIST = (types.BooleanType, types.DictType, types.FloatType, types.IntType, types.ListType, types.LongType, types.NoneType, types.StringType, types.TupleType, types.UnicodeType, types.XRangeType, type(AttribDict()), type(set()))
-
 # Chars used to quickly distinguish if the user provided tainted parameter values
 DUMMY_SQL_INJECTION_CHARS = ";()'"
 
@@ -532,6 +532,9 @@ BRUTE_COLUMN_EXISTS_TEMPLATE = "EXISTS(SELECT %s FROM %s)"
 # Data inside shellcodeexec to be filled with random string
 SHELLCODEEXEC_RANDOM_STRING_MARKER = "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
 
+# Period after last-update to start nagging about the old revision
+LAST_UPDATE_NAGGING_DAYS = 60
+
 # Generic address for checking the Internet connection while using switch --check-internet
 CHECK_INTERNET_ADDRESS = "https://ipinfo.io/"
 
@@ -678,7 +681,10 @@ VALID_TIME_CHARS_RUN_THRESHOLD = 100
 CHECK_ZERO_COLUMNS_THRESHOLD = 10
 
 # Boldify all logger messages containing these "patterns"
-BOLD_PATTERNS = ("' injectable", "provided empty", "leftover chars", "might be injectable", "' is vulnerable", "is not injectable", "does not seem to be", "test failed", "test passed", "live test final result", "test shows that", "the back-end DBMS is", "created Github", "blocked by the target server", "protection is involved", "CAPTCHA", "specific response", "NULL connection is supported", "PASSED", "FAILED")
+BOLD_PATTERNS = ("' injectable", "provided empty", "leftover chars", "might be injectable", "' is vulnerable", "is not injectable", "does not seem to be", "test failed", "test passed", "live test final result", "test shows that", "the back-end DBMS is", "created Github", "blocked by the target server", "protection is involved", "CAPTCHA", "specific response", "NULL connection is supported", "PASSED", "FAILED", "for more than")
+
+# TLDs used in randomization of email-alike parameter values
+RANDOMIZATION_TLDS = ("com", "net", "ru", "org", "de", "jp", "cn", "fr", "it", "pl", "tv", "edu", "in", "ir", "es", "me", "info", "gr", "gov", "ca", "co", "se", "cz", "to", "vn", "nl", "cc", "az", "hu", "ua", "be", "no", "biz", "io", "ch", "ro", "sk", "eu", "us", "tw", "pt", "fi", "at", "lt", "kz", "cl", "hr", "pk", "lv", "la", "pe")
 
 # Generic www root directory names
 GENERIC_DOC_ROOT_DIRECTORY_NAMES = ("htdocs", "httpdocs", "public", "wwwroot", "www")
@@ -690,7 +696,7 @@ MAX_HELP_OPTION_LENGTH = 18
 MAX_CONNECT_RETRIES = 100
 
 # Strings for detecting formatting errors
-FORMAT_EXCEPTION_STRINGS = ("Type mismatch", "Error converting", "Please enter a", "Conversion failed", "String or binary data would be truncated", "Failed to convert", "unable to interpret text value", "Input string was not in a correct format", "System.FormatException", "java.lang.NumberFormatException", "ValueError: invalid literal", "TypeMismatchException", "CF_SQL_INTEGER", " for CFSQLTYPE ", "cfqueryparam cfsqltype", "InvalidParamTypeException", "Invalid parameter type", "is not of type numeric", "<cfif Not IsNumeric(", "invalid input syntax for integer", "invalid input syntax for type", "invalid number", "character to number conversion error", "unable to interpret text value", "String was not recognized as a valid", "Convert.ToInt", "cannot be converted to a ", "InvalidDataException")
+FORMAT_EXCEPTION_STRINGS = ("Type mismatch", "Error converting", "Please enter a", "Conversion failed", "String or binary data would be truncated", "Failed to convert", "unable to interpret text value", "Input string was not in a correct format", "System.FormatException", "java.lang.NumberFormatException", "ValueError: invalid literal", "TypeMismatchException", "CF_SQL_INTEGER", "CF_SQL_NUMERIC", " for CFSQLTYPE ", "cfqueryparam cfsqltype", "InvalidParamTypeException", "Invalid parameter type", "Attribute validation error for tag", "is not of type numeric", "<cfif Not IsNumeric(", "invalid input syntax for integer", "invalid input syntax for type", "invalid number", "character to number conversion error", "unable to interpret text value", "String was not recognized as a valid", "Convert.ToInt", "cannot be converted to a ", "InvalidDataException")
 
 # Regular expression used for extracting ASP.NET view state values
 VIEWSTATE_REGEX = r'(?i)(?P<name>__VIEWSTATE[^"]*)[^>]+value="(?P<result>[^"]+)'
@@ -710,6 +716,9 @@ RESTAPI_DEFAULT_ADDRESS = "127.0.0.1"
 # Default REST-JSON API server listen port
 RESTAPI_DEFAULT_PORT = 8775
 
+# Use "Supplementary Private Use Area-A"
+INVALID_UNICODE_PRIVATE_AREA = False
+
 # Format used for representing invalid unicode characters
 INVALID_UNICODE_CHAR_FORMAT = r"\x%02x"
 
@@ -758,8 +767,8 @@ METASPLOIT_SESSION_TIMEOUT = 120
 # Reference: http://www.postgresql.org/docs/9.0/static/catalog-pg-largeobject.html
 LOBLKSIZE = 2048
 
-# Suffix used to mark variables having keyword names
+# Prefix used to mark special variables (e.g. keywords, having special chars, etc.)
-EVALCODE_KEYWORD_SUFFIX = "_KEYWORD"
+EVALCODE_ENCODED_PREFIX = "EVAL_"
 
 # Reference: http://www.cookiecentral.com/faq/#3.5
 NETSCAPE_FORMAT_HEADER_COOKIES = "# Netscape HTTP Cookie File."
@@ -773,6 +782,9 @@ BRUTE_DOC_ROOT_PREFIXES = {
     OS.WINDOWS: ("/xampp", "/Program Files/xampp", "/wamp", "/Program Files/wampp", "/apache", "/Program Files/Apache Group/Apache", "/Program Files/Apache Group/Apache2", "/Program Files/Apache Group/Apache2.2", "/Program Files/Apache Group/Apache2.4", "/Inetpub/wwwroot", "/Inetpub/wwwroot/%TARGET%", "/Inetpub/vhosts/%TARGET%")
 }
 
+# Table prefix to use in "takeover" functionalities (i.e. auxiliary tables used by sqlmap at the vulnerable DBMS)
+TAKEOVER_TABLE_PREFIX = "sqlmap"
+
 # Suffixes used in brute force search for web server document root
 BRUTE_DOC_ROOT_SUFFIXES = ("", "html", "htdocs", "httpdocs", "php", "public", "src", "site", "build", "web", "www", "data", "sites/all", "www/build")
 
@@ -785,6 +797,9 @@ KB_CHARS_BOUNDARY_CHAR = 'q'
 # Letters of lower frequency used in kb.chars
 KB_CHARS_LOW_FREQUENCY_ALPHABET = "zqxjkvbp"
 
+# SQL keywords used for splitting in HTTP chunked transfer encoded requests (switch --chunk)
+HTTP_CHUNKED_SPLIT_KEYWORDS = ("SELECT", "UPDATE", "INSERT", "FROM", "LOAD_FILE", "UNION", "information_schema", "sysdatabases", "msysaccessobjects", "msysqueries", "sysmodules")
+
 # CSS style used in HTML dump format
 HTML_DUMP_CSS_STYLE = """<style>
 table{
@@ -809,3 +824,20 @@ th{
     font-size:12px;
 }
 </style>"""
+
+# Leaving (dirty) possibility to change values from here (e.g. `export SQLMAP__MAX_NUMBER_OF_THREADS=20`)
+for key, value in os.environ.items():
+    if key.upper().startswith("%s_" % SQLMAP_ENVIRONMENT_PREFIX):
+        _ = key[len(SQLMAP_ENVIRONMENT_PREFIX) + 1:].upper()
+        if _ in globals():
+            globals()[_] = value
+
+# Installing "reversible" unicode (decoding) error handler
+def _reversible(ex):
+    if isinstance(ex, UnicodeDecodeError):
+        if INVALID_UNICODE_PRIVATE_AREA:
+            return ("".join(unichr(int('000f00%2x' % ord(_), 16)) for _ in ex.object[ex.start:ex.end]), ex.end)
+        else:
+            return ("".join(INVALID_UNICODE_CHAR_FORMAT % ord(_) for _ in ex.object[ex.start:ex.end]).decode(UNICODE_ENCODING), ex.end)
+
+codecs.register_error("reversible", _reversible)

lib/core/shell.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)

lib/core/subprocessng.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
@@ -85,7 +85,7 @@ class Popen(subprocess.Popen):
         getattr(self, which).close()
         setattr(self, which, None)
 
-    if subprocess.mswindows:
+    if IS_WIN:
         def send(self, input):
             if not self.stdin:
                 return None

lib/core/target.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
@@ -12,7 +12,6 @@ import subprocess
 import sys
 import tempfile
 import time
-import urlparse
 
 from lib.core.common import Backend
 from lib.core.common import getSafeExString
@@ -27,6 +26,7 @@ from lib.core.common import randomStr
 from lib.core.common import readInput
 from lib.core.common import resetCookieJar
 from lib.core.common import urldecode
+from lib.core.compat import xrange
 from lib.core.data import conf
 from lib.core.data import kb
 from lib.core.data import logger
@@ -73,7 +73,9 @@ from lib.core.settings import URI_INJECTABLE_REGEX
 from lib.core.settings import USER_AGENT_ALIASES
 from lib.core.settings import XML_RECOGNITION_REGEX
 from lib.utils.hashdb import HashDB
-from thirdparty.odict.odict import OrderedDict
+from thirdparty import six
+from thirdparty.odict import OrderedDict
+from thirdparty.six.moves import urllib as _urllib
 
 def _setRequestParams():
     """
@@ -212,7 +214,7 @@ def _setRequestParams():
                 if not (kb.processUserMarks and kb.customInjectionMark in conf.data):
                     conf.data = getattr(conf.data, UNENCODED_ORIGINAL_VALUE, conf.data)
                     conf.data = conf.data.replace(kb.customInjectionMark, ASTERISK_MARKER)
-                    conf.data = re.sub(r"(?si)((Content-Disposition[^\n]+?name\s*=\s*[\"']?(?P<name>[^\"'\r\n]+)[\"']?).+?)(((\r)?\n)+--)", functools.partial(process, repl=r"\g<1>%s\g<4>" % kb.customInjectionMark), conf.data)
+                    conf.data = re.sub(r"(?si)((Content-Disposition[^\n]+?name\s*=\s*[\"']?(?P<name>[^\"'\r\n]+)[\"']?).+?)((%s)+--)" % ("\r\n" if "\r\n" in conf.data else '\n'), functools.partial(process, repl=r"\g<1>%s\g<4>" % kb.customInjectionMark), conf.data)
 
                 kb.postHint = POST_HINT.MULTIPART
 
@@ -276,7 +278,7 @@ def _setRequestParams():
 
             if not kb.processUserMarks:
                 if place == PLACE.URI:
-                    query = urlparse.urlsplit(value).query
+                    query = _urllib.parse.urlsplit(value).query
                     if query:
                         parameters = conf.parameters[PLACE.GET] = query
                         paramDict = paramToDict(PLACE.GET, parameters)
@@ -393,7 +395,7 @@ def _setRequestParams():
         raise SqlmapGenericException(errMsg)
 
     if conf.csrfToken:
-        if not any(re.search(conf.csrfToken, ' '.join(_), re.I) for _ in (conf.paramDict.get(PLACE.GET, {}), conf.paramDict.get(PLACE.POST, {}))) and not re.search(r"\b%s\b" % re.escape(conf.csrfToken), conf.data or "") and conf.csrfToken not in set(_[0].lower() for _ in conf.httpHeaders) and conf.csrfToken not in conf.paramDict.get(PLACE.COOKIE, {}):
+        if not any(re.search(conf.csrfToken, ' '.join(_), re.I) for _ in (conf.paramDict.get(PLACE.GET, {}), conf.paramDict.get(PLACE.POST, {}))) and not re.search(r"\b%s\b" % conf.csrfToken, conf.data or "") and conf.csrfToken not in set(_[0].lower() for _ in conf.httpHeaders) and conf.csrfToken not in conf.paramDict.get(PLACE.COOKIE, {}):
             errMsg = "anti-CSRF token parameter '%s' not " % conf.csrfToken._original
             errMsg += "found in provided GET, POST, Cookie or header values"
             raise SqlmapGenericException(errMsg)
@@ -408,7 +410,7 @@ def _setRequestParams():
                     message += "Do you want sqlmap to automatically update it in further requests? [y/N] "
 
                     if readInput(message, default='N', boolean=True):
-                        class _(unicode):
+                        class _(six.text_type):
                             pass
                         conf.csrfToken = _(re.escape(getUnicode(parameter)))
                         conf.csrfToken._original = getUnicode(parameter)
@@ -630,36 +632,6 @@ def _createTargetDirs():
     Create the output directory.
     """
 
-    for context in "output", "history":
-        directory = paths["SQLMAP_%s_PATH" % context.upper()]
-        try:
-            if not os.path.isdir(directory):
-                os.makedirs(directory)
-
-            _ = os.path.join(directory, randomStr())
-            open(_, "w+b").close()
-            os.remove(_)
-
-            if conf.outputDir and context == "output":
-                warnMsg = "using '%s' as the %s directory" % (directory, context)
-                logger.warn(warnMsg)
-        except (OSError, IOError) as ex:
-            try:
-                tempDir = tempfile.mkdtemp(prefix="sqlmap%s" % context)
-            except Exception as _:
-                errMsg = "unable to write to the temporary directory ('%s'). " % _
-                errMsg += "Please make sure that your disk is not full and "
-                errMsg += "that you have sufficient write permissions to "
-                errMsg += "create temporary files and/or directories"
-                raise SqlmapSystemException(errMsg)
-
-            warnMsg = "unable to %s %s directory " % ("create" if not os.path.isdir(directory) else "write to the", context)
-            warnMsg += "'%s' (%s). " % (directory, getUnicode(ex))
-            warnMsg += "Using temporary directory '%s' instead" % getUnicode(tempDir)
-            logger.warn(warnMsg)
-
-            paths["SQLMAP_%s_PATH" % context.upper()] = tempDir
-
     conf.outputPath = os.path.join(getUnicode(paths.SQLMAP_OUTPUT_PATH), normalizeUnicode(getUnicode(conf.hostname)))
 
     try:
@@ -741,7 +713,7 @@ def initTargetEnv():
         _setDBMS()
 
     if conf.data:
-        class _(unicode):
+        class _(six.text_type):
             pass
 
         kb.postUrlEncode = True
@@ -757,7 +729,7 @@ def initTargetEnv():
             setattr(conf.data, UNENCODED_ORIGINAL_VALUE, original)
             kb.postSpaceToPlus = '+' in original
 
-    match = re.search(INJECT_HERE_REGEX, conf.data or "") or re.search(INJECT_HERE_REGEX, conf.url or "")
+    match = re.search(INJECT_HERE_REGEX, "%s %s %s" % (conf.url, conf.data, conf.httpHeaders))
     kb.customInjectionMark = match.group(0) if match else CUSTOM_INJECTION_MARK_CHAR
 
 def setupTargetEnv():

lib/core/testing.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
@@ -8,21 +8,24 @@ See the file 'LICENSE' for copying permission
 import codecs
 import doctest
 import os
+import random
 import re
 import shutil
 import sys
 import tempfile
+import threading
 import time
 import traceback
 
 from extra.beep.beep import beep
+from extra.vulnserver import vulnserver
 from lib.controller.controller import start
-from lib.core.common import checkIntegrity
 from lib.core.common import clearConsoleLine
 from lib.core.common import dataToStdout
 from lib.core.common import getUnicode
 from lib.core.common import randomStr
 from lib.core.common import readXmlFile
+from lib.core.common import shellExec
 from lib.core.data import conf
 from lib.core.data import logger
 from lib.core.data import paths
@@ -44,6 +47,49 @@ class Failures(object):
 
 _failures = Failures()
 
+def vulnTest():
+    """
+    Runs the testing against 'vulnserver'
+    """
+
+    retVal = True
+    count, length = 0, 6
+    address, port = "127.0.0.10", random.randint(1025, 65535)
+
+    def _thread():
+        vulnserver.init(quiet=True)
+        vulnserver.run(address=address, port=port)
+
+    thread = threading.Thread(target=_thread)
+    thread.daemon = True
+    thread.start()
+
+    for options, checks in (
+        ("--flush-session --identify-waf", ("CloudFlare",)),
+        ("--flush-session --parse-errors", (": syntax error", "Type: boolean-based blind", "Type: time-based blind", "Type: UNION query", "back-end DBMS: SQLite", "3 columns")),
+        ("--banner --schema --dump -T users --binary-fields=surname --where 'id>3'", ("banner: '3", "INTEGER", "TEXT", "id", "name", "surname", "2 entries", "6E616D6569736E756C6C")),
+        ("--all --tamper=between,randomcase", ("5 entries", "luther", "blisset", "fluffy", "179ad45c6ce2cb97cf1029e212046e81", "NULL", "nameisnull", "testpass")),
+        ("--technique=B --hex --fresh-queries --threads=4 --sql-query='SELECT 987654321'", ("length of query output", ": '987654321'",)),
+        ("--technique=T --fresh-queries --sql-query='SELECT 1234'", (": '1234'",)),
+    ):
+        output = shellExec("%s %s -u http://%s:%d/?id=1 --batch %s" % (sys.executable, os.path.join(os.path.dirname(__file__), "..", "..", "sqlmap.py"), address, port, options))
+        output = getUnicode(output)
+
+        if not all(check in output for check in checks):
+            retVal = False
+
+        count += 1
+        status = '%d/%d (%d%%) ' % (count, length, round(100.0 * count / length))
+        dataToStdout("\r[%s] [INFO] complete: %s" % (time.strftime("%X"), status))
+
+    clearConsoleLine()
+    if retVal:
+        logger.info("vuln test final result: PASSED")
+    else:
+        logger.error("vuln test final result: FAILED")
+
+    return retVal
+
 def smokeTest():
     """
     Runs the basic smoke testing of a program
@@ -52,9 +98,6 @@ def smokeTest():
     retVal = True
     count, length = 0, 0
 
-    if not checkIntegrity():
-        retVal = False
-
     for root, _, files in os.walk(paths.SQLMAP_ROOT_PATH):
         if any(_ in root for _ in ("thirdparty", "extra")):
             continue

lib/core/threads.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
@@ -8,16 +8,18 @@ See the file 'LICENSE' for copying permission
 from __future__ import print_function
 
 import difflib
-import random
 import threading
 import time
 import traceback
 
+from lib.core.compat import WichmannHill
+from lib.core.compat import xrange
 from lib.core.data import conf
 from lib.core.data import kb
 from lib.core.data import logger
 from lib.core.datatype import AttribDict
 from lib.core.enums import PAYLOAD
+from lib.core.exception import SqlmapBaseException
 from lib.core.exception import SqlmapConnectionException
 from lib.core.exception import SqlmapThreadException
 from lib.core.exception import SqlmapUserQuitException
@@ -56,7 +58,7 @@ class _ThreadData(threading.local):
         self.lastRequestMsg = None
         self.lastRequestUID = 0
         self.lastRedirectURL = None
-        self.random = random.WichmannHill()
+        self.random = WichmannHill()
         self.resumed = False
         self.retriesCount = 0
         self.seqMatcher = difflib.SequenceMatcher(None)
@@ -95,9 +97,10 @@ def exceptionHandledFunction(threadFunction, silent=False):
         raise
     except Exception as ex:
         if not silent and kb.get("threadContinue"):
-            logger.error("thread %s: %s" % (threading.currentThread().getName(), ex.message))
+            errMsg = ex.message if isinstance(ex, SqlmapBaseException) else "%s: %s" % (type(ex).__name__, ex.message)
+            logger.error("thread %s: '%s'" % (threading.currentThread().getName(), errMsg))
 
-            if conf.get("verbose") > 1:
+            if conf.get("verbose") > 1 and not isinstance(ex, (SqlmapUserQuitException,)):
                 traceback.print_exc()
 
 def setDaemon(thread):

lib/core/unescaper.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)

lib/core/update.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)

lib/core/wordlist.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)

lib/parse/__init__.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)

lib/parse/banner.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)

lib/parse/cmdline.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
@@ -22,6 +22,7 @@ from lib.core.common import checkSystemEncoding
 from lib.core.common import dataToStdout
 from lib.core.common import expandMnemonics
 from lib.core.common import getUnicode
+from lib.core.compat import xrange
 from lib.core.data import cmdLineOptions
 from lib.core.data import conf
 from lib.core.data import logger
@@ -31,6 +32,7 @@ from lib.core.exception import SqlmapShellQuitException
 from lib.core.exception import SqlmapSyntaxException
 from lib.core.settings import BASIC_HELP_ITEMS
 from lib.core.settings import DUMMY_URL
+from lib.core.settings import INFERENCE_UNKNOWN_CHAR
 from lib.core.settings import IS_WIN
 from lib.core.settings import MAX_HELP_OPTION_LENGTH
 from lib.core.settings import VERSION_STRING
@@ -38,6 +40,7 @@ from lib.core.shell import autoCompletion
 from lib.core.shell import clearHistory
 from lib.core.shell import loadHistory
 from lib.core.shell import saveHistory
+from thirdparty.six.moves import input as _input
 
 def cmdLineParser(argv=None):
     """
@@ -52,7 +55,7 @@ def cmdLineParser(argv=None):
     # Reference: https://stackoverflow.com/a/4012683 (Note: previously used "...sys.getfilesystemencoding() or UNICODE_ENCODING")
     _ = getUnicode(os.path.basename(argv[0]), encoding=sys.stdin.encoding)
 
-    usage = "%s%s [options]" % ("python " if not IS_WIN else "", "\"%s\"" % _ if " " in _ else _)
+    usage = "%s%s [options]" % ("%s " % os.path.basename(sys.executable) if not IS_WIN else "", "\"%s\"" % _ if " " in _ else _)
     parser = OptionParser(usage=usage)
 
     try:
@@ -214,6 +217,9 @@ def cmdLineParser(argv=None):
         request.add_option("--force-ssl", dest="forceSSL", action="store_true",
                            help="Force usage of SSL/HTTPS")
 
+        request.add_option("--chunked", dest="chunked", action="store_true",
+                           help="Use HTTP chunked transfer encoded (POST) requests")
+
         request.add_option("--hpp", dest="hpp", action="store_true",
                            help="Use HTTP parameter pollution method")
 
@@ -443,7 +449,7 @@ def cmdLineParser(argv=None):
         enumeration.add_option("--last", dest="lastChar", type="int",
                                help="Last query output word character to retrieve")
 
-        enumeration.add_option("--sql-query", dest="query",
+        enumeration.add_option("--sql-query", dest="sqlQuery",
                                help="SQL statement to be executed")
 
         enumeration.add_option("--sql-shell", dest="sqlShell", action="store_true",
@@ -594,6 +600,12 @@ def cmdLineParser(argv=None):
         general.add_option("--parse-errors", dest="parseErrors", action="store_true",
                            help="Parse and display DBMS error messages from responses")
 
+        general.add_option("--preprocess", dest="preprocess",
+                           help="Use given script(s) for preprocessing of response data")
+
+        general.add_option("--repair", dest="repair", action="store_true",
+                           help="Redump entries having unknown character marker (%s)" % INFERENCE_UNKNOWN_CHAR)
+
         general.add_option("--save", dest="saveConfig",
                            help="Save options to a configuration INI file")
 
@@ -670,6 +682,10 @@ def cmdLineParser(argv=None):
                                  help="Simple wizard interface for beginner users")
 
         # Hidden and/or experimental options
+        parser.add_option("--base64", dest="base64Parameter",
+                          help=SUPPRESS_HELP)
+#                          help="Parameter(s) containing Base64 encoded values")
+
         parser.add_option("--crack", dest="hashFile",
                           help=SUPPRESS_HELP)
 #                          help="Load and crack hashes from a file (standalone)")
@@ -707,6 +723,9 @@ def cmdLineParser(argv=None):
         parser.add_option("--live-test", dest="liveTest", action="store_true",
                           help=SUPPRESS_HELP)
 
+        parser.add_option("--vuln-test", dest="vulnTest", action="store_true",
+                          help=SUPPRESS_HELP)
+
         parser.add_option("--stop-fail", dest="stopFail", action="store_true",
                           help=SUPPRESS_HELP)
 
@@ -744,7 +763,7 @@ def cmdLineParser(argv=None):
             return retVal
 
         parser.formatter._format_option_strings = parser.formatter.format_option_strings
-        parser.formatter.format_option_strings = type(parser.formatter.format_option_strings)(_, parser, type(parser))
+        parser.formatter.format_option_strings = type(parser.formatter.format_option_strings)(_, parser)
 
         # Dirty hack for making a short option '-hh'
         option = parser.get_option("--hh")
@@ -791,7 +810,7 @@ def cmdLineParser(argv=None):
                 command = None
 
                 try:
-                    command = raw_input("sqlmap-shell> ").strip()
+                    command = _input("sqlmap-shell> ").strip()
                     command = getUnicode(command, encoding=sys.stdin.encoding)
                 except (KeyboardInterrupt, EOFError):
                     print()
@@ -840,6 +859,14 @@ def cmdLineParser(argv=None):
             elif argv[i] == "-H":
                 if i + 1 < len(argv):
                     extraHeaders.append(argv[i + 1])
+            elif argv[i] == "-r":
+                for j in xrange(i + 2, len(argv)):
+                    value = argv[j]
+                    if os.path.isfile(value):
+                        argv[i + 1] += ",%s" % value
+                        argv[j] = ''
+                    else:
+                        break
             elif re.match(r"\A\d+!\Z", argv[i]) and argv[max(0, i - 1)] == "--threads" or re.match(r"\A--threads.+\d+!\Z", argv[i]):
                 argv[i] = argv[i][:-1]
                 conf.skipThreadCheck = True
@@ -890,7 +917,7 @@ def cmdLineParser(argv=None):
         if args.dummy:
             args.url = args.url or DUMMY_URL
 
-        if not any((args.direct, args.url, args.logFile, args.bulkFile, args.googleDork, args.configFile, args.requestFile, args.updateAll, args.smokeTest, args.liveTest, args.wizard, args.dependencies, args.purge, args.sitemapUrl, args.listTampers, args.hashFile)):
+        if not any((args.direct, args.url, args.logFile, args.bulkFile, args.googleDork, args.configFile, args.requestFile, args.updateAll, args.smokeTest, args.vulnTest, args.liveTest, args.wizard, args.dependencies, args.purge, args.sitemapUrl, args.listTampers, args.hashFile)):
             errMsg = "missing a mandatory option (-d, -u, -l, -m, -r, -g, -c, -x, --list-tampers, --wizard, --update, --purge or --dependencies). "
             errMsg += "Use -h for basic and -hh for advanced help\n"
             parser.error(errMsg)
@@ -904,7 +931,7 @@ def cmdLineParser(argv=None):
         # Protection against Windows dummy double clicking
         if IS_WIN:
             dataToStdout("\nPress Enter to continue...")
-            raw_input()
+            _input()
         raise
 
     debugMsg = "parsing command line"

lib/parse/configfile.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)

lib/parse/handler.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)

lib/parse/headers.py

@@ -1,17 +1,17 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """
 
-import itertools
 import os
 
 from lib.core.common import parseXmlFile
 from lib.core.data import kb
 from lib.core.data import paths
 from lib.parse.handler import FingerprintHandler
+from thirdparty.six.moves import filter as _filter
 
 def headersParser(headers):
     """
@@ -30,7 +30,7 @@ def headersParser(headers):
             "x-powered-by": os.path.join(paths.SQLMAP_XML_BANNER_PATH, "x-powered-by.xml"),
         }
 
-    for header in itertools.ifilter(lambda _: _ in kb.headerPaths, headers):
+    for header in _filter(lambda _: _ in kb.headerPaths, headers):
         value = headers[header]
         xmlfile = kb.headerPaths[header]
         handler = FingerprintHandler(value, kb.headersFp)

lib/parse/html.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)

lib/parse/payloads.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
@@ -11,6 +11,7 @@ import re
 from xml.etree import ElementTree as et
 
 from lib.core.common import getSafeExString
+from lib.core.compat import xrange
 from lib.core.data import conf
 from lib.core.data import paths
 from lib.core.datatype import AttribDict
@@ -24,8 +25,8 @@ def cleanupVals(text, tag):
     if tag in ("clause", "where"):
         text = text.split(',')
 
-    if isinstance(text, basestring):
+    if hasattr(text, "isdigit") and text.isdigit():
-        text = int(text) if text.isdigit() else text
+        text = int(text)
 
     elif isinstance(text, list):
         count = 0

lib/parse/sitemap.py

@@ -1,19 +1,19 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """
 
-import httplib
 import re
 
 from lib.core.common import readInput
 from lib.core.data import kb
 from lib.core.data import logger
+from lib.core.datatype import OrderedSet
 from lib.core.exception import SqlmapSyntaxException
 from lib.request.connect import Connect as Request
-from thirdparty.oset.pyoset import oset
+from thirdparty.six.moves import http_client as _http_client
 
 abortedFlag = None
 
@@ -26,11 +26,11 @@ def parseSitemap(url, retVal=None):
     try:
         if retVal is None:
             abortedFlag = False
-            retVal = oset()
+            retVal = OrderedSet()
 
         try:
             content = Request.getPage(url=url, raise404=True)[0] if not abortedFlag else ""
-        except httplib.InvalidURL:
+        except _http_client.InvalidURL:
             errMsg = "invalid URL given for sitemap ('%s')" % url
             raise SqlmapSyntaxException(errMsg)
 

lib/request/__init__.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)

lib/request/basic.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
@@ -7,15 +7,18 @@ See the file 'LICENSE' for copying permission
 
 import codecs
 import gzip
+import io
 import logging
 import re
-import StringIO
 import struct
 import zlib
 
 from lib.core.common import Backend
+from lib.core.common import decodeHex
 from lib.core.common import extractErrorMessage
 from lib.core.common import extractRegexResult
+from lib.core.common import filterNone
+from lib.core.common import getBytes
 from lib.core.common import getPublicTypeMembers
 from lib.core.common import getSafeExString
 from lib.core.common import getUnicode
@@ -41,13 +44,13 @@ from lib.core.settings import MAX_CONNECTION_TOTAL_SIZE
 from lib.core.settings import META_CHARSET_REGEX
 from lib.core.settings import PARSE_HEADERS_LIMIT
 from lib.core.settings import SELECT_FROM_TABLE_REGEX
-from lib.core.settings import UNICODE_ENCODING
 from lib.core.settings import VIEWSTATE_REGEX
 from lib.parse.headers import headersParser
 from lib.parse.html import htmlParser
 from lib.utils.htmlentities import htmlEntities
+from thirdparty import six
 from thirdparty.chardet import detect
-from thirdparty.odict.odict import OrderedDict
+from thirdparty.odict import OrderedDict
 
 def forgeHeaders(items=None, base=None):
     """
@@ -100,7 +103,7 @@ def forgeHeaders(items=None, base=None):
 
                 if ("%s=" % getUnicode(cookie.name)) in getUnicode(headers[HTTP_HEADER.COOKIE]):
                     if conf.loadCookies:
-                        conf.httpHeaders = filter(None, ((item if item[0] != HTTP_HEADER.COOKIE else None) for item in conf.httpHeaders))
+                        conf.httpHeaders = filterNone((item if item[0] != HTTP_HEADER.COOKIE else None) for item in conf.httpHeaders)
                     elif kb.mergeCookies is None:
                         message = "you provided a HTTP %s header value. " % HTTP_HEADER.COOKIE
                         message += "The target URL provided its own cookies within "
@@ -154,6 +157,9 @@ def checkCharEncoding(encoding, warn=True):
     'utf8'
     """
 
+    if isinstance(encoding, six.binary_type):
+        encoding = getUnicode(encoding)
+
     if isListLike(encoding):
         encoding = unArrayizeValue(encoding)
 
@@ -218,13 +224,13 @@ def checkCharEncoding(encoding, warn=True):
     # Reference: http://www.iana.org/assignments/character-sets
     # Reference: http://docs.python.org/library/codecs.html
     try:
-        codecs.lookup(encoding.encode(UNICODE_ENCODING) if isinstance(encoding, unicode) else encoding)
+        codecs.lookup(encoding)
-    except (LookupError, ValueError):
+    except:
         encoding = None
 
     if encoding:
         try:
-            unicode(randomStr(), encoding)
+            six.text_type(getBytes(randomStr()), encoding)
         except:
             if warn:
                 warnMsg = "invalid web page charset '%s'" % encoding
@@ -257,12 +263,12 @@ def decodePage(page, contentEncoding, contentType):
     if not page or (conf.nullConnection and len(page) < 2):
         return getUnicode(page)
 
-    if isinstance(contentEncoding, basestring) and contentEncoding:
+    if hasattr(contentEncoding, "lower"):
         contentEncoding = contentEncoding.lower()
     else:
         contentEncoding = ""
 
-    if isinstance(contentType, basestring) and contentType:
+    if hasattr(contentType, "lower"):
         contentType = contentType.lower()
     else:
         contentType = ""
@@ -273,9 +279,9 @@ def decodePage(page, contentEncoding, contentType):
 
         try:
             if contentEncoding == "deflate":
-                data = StringIO.StringIO(zlib.decompress(page, -15))  # Reference: http://stackoverflow.com/questions/1089662/python-inflate-and-deflate-implementations
+                data = io.BytesIO(zlib.decompress(page, -15))  # Reference: http://stackoverflow.com/questions/1089662/python-inflate-and-deflate-implementations
             else:
-                data = gzip.GzipFile("", "rb", 9, StringIO.StringIO(page))
+                data = gzip.GzipFile("", "rb", 9, io.BytesIO(page))
                 size = struct.unpack("<l", page[-4:])[0]  # Reference: http://pydoc.org/get.cgi/usr/local/lib/python2.5/gzip.py
                 if size > MAX_CONNECTION_TOTAL_SIZE:
                     raise Exception("size too large")
@@ -312,44 +318,41 @@ def decodePage(page, contentEncoding, contentType):
         kb.pageEncoding = conf.encoding
 
     # can't do for all responses because we need to support binary files too
-    if not isinstance(page, unicode) and "text/" in contentType:
+    if isinstance(page, six.binary_type) and "text/" in contentType:
-        if kb.heuristicMode:
+        # e.g. &#x9;&#195;&#235;&#224;&#226;&#224;
-            kb.pageEncoding = kb.pageEncoding or checkCharEncoding(getHeuristicCharEncoding(page))
+        if b"&#" in page:
-            page = getUnicode(page, kb.pageEncoding)
+            page = re.sub(b"&#x([0-9a-f]{1,2});", lambda _: decodeHex(_.group(1) if len(_.group(1)) == 2 else "0%s" % _.group(1)), page)
-        else:
+            page = re.sub(b"&#(\d{1,3});", lambda _: chr(int(_.group(1))) if int(_.group(1)) < 256 else _.group(0), page)
-            # e.g. &#195;&#235;&#224;&#226;&#224;
-            if "&#" in page:
-                page = re.sub(r"&#(\d{1,3});", lambda _: chr(int(_.group(1))) if int(_.group(1)) < 256 else _.group(0), page)
 
-            # e.g. %20%28%29
+        # e.g. %20%28%29
-            if "%" in page:
+        if b"%" in page:
-                page = re.sub(r"%([0-9a-fA-F]{2})", lambda _: _.group(1).decode("hex"), page)
+            page = re.sub(b"%([0-9a-fA-F]{2})", lambda _: decodeHex(_.group(1)), page)
 
-            # e.g. &amp;
+        # e.g. &amp;
-            page = re.sub(r"&([^;]+);", lambda _: chr(htmlEntities[_.group(1)]) if htmlEntities.get(_.group(1), 256) < 256 else _.group(0), page)
+        page = re.sub(b"&([^;]+);", lambda _: chr(htmlEntities[_.group(1)]) if htmlEntities.get(_.group(1), 256) < 256 else _.group(0), page)
 
-            kb.pageEncoding = kb.pageEncoding or checkCharEncoding(getHeuristicCharEncoding(page))
+        kb.pageEncoding = kb.pageEncoding or checkCharEncoding(getHeuristicCharEncoding(page))
 
-            if (kb.pageEncoding or "").lower() == "utf-8-sig":
+        if (kb.pageEncoding or "").lower() == "utf-8-sig":
-                kb.pageEncoding = "utf-8"
+            kb.pageEncoding = "utf-8"
-                if page and page.startswith("\xef\xbb\xbf"):  # Reference: https://docs.python.org/2/library/codecs.html (Note: noticed problems when "utf-8-sig" is left to Python for handling)
+            if page and page.startswith("\xef\xbb\xbf"):  # Reference: https://docs.python.org/2/library/codecs.html (Note: noticed problems when "utf-8-sig" is left to Python for handling)
-                    page = page[3:]
+                page = page[3:]
 
-            page = getUnicode(page, kb.pageEncoding)
+        page = getUnicode(page, kb.pageEncoding)
 
-            # e.g. &#8217;&#8230;&#8482;
+        # e.g. &#8217;&#8230;&#8482;
-            if "&#" in page:
+        if "&#" in page:
-                def _(match):
+            def _(match):
-                    retVal = match.group(0)
+                retVal = match.group(0)
-                    try:
+                try:
-                        retVal = unichr(int(match.group(1)))
+                    retVal = unichr(int(match.group(1)))
-                    except (ValueError, OverflowError):
+                except (ValueError, OverflowError):
-                        pass
+                    pass
-                    return retVal
+                return retVal
-                page = re.sub(r"&#(\d+);", _, page)
+            page = re.sub(r"&#(\d+);", _, page)
 
-            # e.g. &zeta;
+        # e.g. &zeta;
-            page = re.sub(r"&([^;]+);", lambda _: unichr(htmlEntities[_.group(1)]) if htmlEntities.get(_.group(1), 0) > 255 else _.group(0), page)
+        page = re.sub(r"&([^;]+);", lambda _: unichr(htmlEntities[_.group(1)]) if htmlEntities.get(_.group(1), 0) > 255 else _.group(0), page)
 
     return page
 

lib/request/basicauthhandler.py

@@ -1,19 +1,19 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """
 
-import urllib2
+from thirdparty.six.moves import urllib as _urllib
 
-class SmartHTTPBasicAuthHandler(urllib2.HTTPBasicAuthHandler):
+class SmartHTTPBasicAuthHandler(_urllib.request.HTTPBasicAuthHandler):
     """
     Reference: http://selenic.com/hg/rev/6c51a5056020
     Fix for a: http://bugs.python.org/issue8797
     """
     def __init__(self, *args, **kwargs):
-        urllib2.HTTPBasicAuthHandler.__init__(self, *args, **kwargs)
+        _urllib.request.HTTPBasicAuthHandler.__init__(self, *args, **kwargs)
         self.retried_req = set()
         self.retried_count = 0
 
@@ -30,8 +30,8 @@ class SmartHTTPBasicAuthHandler(urllib2.HTTPBasicAuthHandler):
             self.retried_count = 0
         else:
             if self.retried_count > 5:
-                raise urllib2.HTTPError(req.get_full_url(), 401, "basic auth failed", headers, None)
+                raise _urllib.error.HTTPError(req.get_full_url(), 401, "basic auth failed", headers, None)
             else:
                 self.retried_count += 1
 
-        return urllib2.HTTPBasicAuthHandler.http_error_auth_reqed(self, auth_header, host, req, headers)
+        return _urllib.request.HTTPBasicAuthHandler.http_error_auth_reqed(self, auth_header, host, req, headers)

lib/request/chunkedhandler.py

@@ -0,0 +1,43 @@
+#!/usr/bin/env python2
+
+"""
+Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
+See the file 'LICENSE' for copying permission
+"""
+
+from lib.core.data import conf
+from thirdparty.six.moves import urllib as _urllib
+
+class ChunkedHandler(_urllib.request.HTTPHandler):
+    """
+    Ensures that HTTPHandler is working properly in case of Chunked Transfer-Encoding
+    """
+
+    def _http_request(self, request):
+        host = request.get_host()
+        if not host:
+            raise _urllib.error.URLError("no host given")
+
+        if request.has_data():  # POST
+            data = request.get_data()
+            if not request.has_header("Content-type"):
+                request.add_unredirected_header(
+                    "Content-type",
+                    "application/x-www-form-urlencoded")
+            if not request.has_header("Content-length") and not conf.chunked:
+                request.add_unredirected_header(
+                    "Content-length", "%d" % len(data))
+
+        sel_host = host
+        if request.has_proxy():
+            sel_host = _urllib.parse.urlsplit(request.get_selector()).netloc
+
+        if not request.has_header("Host"):
+            request.add_unredirected_header("Host", sel_host)
+        for name, value in self.parent.addheaders:
+            name = name.capitalize()
+            if not request.has_header(name):
+                request.add_unredirected_header(name, value)
+        return request
+
+    http_request = _http_request

lib/request/comparison.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
@@ -8,6 +8,7 @@ See the file 'LICENSE' for copying permission
 import re
 
 from lib.core.common import extractRegexResult
+from lib.core.common import getBytes
 from lib.core.common import getFilteredPageContent
 from lib.core.common import listToStrValue
 from lib.core.common import removeDynamicContent
@@ -28,6 +29,7 @@ from lib.core.settings import LOWER_RATIO_BOUND
 from lib.core.settings import UPPER_RATIO_BOUND
 from lib.core.settings import URI_HTTP_HEADER
 from lib.core.threads import getCurrentThreadData
+from thirdparty import six
 
 def comparison(page, headers, code=None, getRatioValue=False, pageLength=None):
     _ = _adjust(_comparison(page, headers, code, getRatioValue, pageLength), getRatioValue)
@@ -105,10 +107,10 @@ def _comparison(page, headers, code, getRatioValue, pageLength):
     else:
         # Preventing "Unicode equal comparison failed to convert both arguments to Unicode"
         # (e.g. if one page is PDF and the other is HTML)
-        if isinstance(seqMatcher.a, str) and isinstance(page, unicode):
+        if isinstance(seqMatcher.a, six.binary_type) and isinstance(page, six.text_type):
-            page = page.encode(kb.pageEncoding or DEFAULT_PAGE_ENCODING, "ignore")
+            page = getBytes(page, kb.pageEncoding or DEFAULT_PAGE_ENCODING, "ignore")
-        elif isinstance(seqMatcher.a, unicode) and isinstance(page, str):
+        elif isinstance(seqMatcher.a, six.text_type) and isinstance(page, six.binary_type):
-            seqMatcher.a = seqMatcher.a.encode(kb.pageEncoding or DEFAULT_PAGE_ENCODING, "ignore")
+            seqMatcher.a = getBytes(seqMatcher.a, kb.pageEncoding or DEFAULT_PAGE_ENCODING, "ignore")
 
         if any(_ is None for _ in (page, seqMatcher.a)):
             return None

lib/request/connect.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
@@ -6,9 +6,6 @@ See the file 'LICENSE' for copying permission
 """
 
 import binascii
-import compiler
-import httplib
-import keyword
 import logging
 import re
 import socket
@@ -16,9 +13,6 @@ import string
 import struct
 import time
 import traceback
-import urllib
-import urllib2
-import urlparse
 
 try:
     import websocket
@@ -32,12 +26,15 @@ from lib.core.agent import agent
 from lib.core.common import asciifyUrl
 from lib.core.common import calculateDeltaSeconds
 from lib.core.common import checkSameHost
+from lib.core.common import chunkSplitPostData
 from lib.core.common import clearConsoleLine
 from lib.core.common import dataToStdout
 from lib.core.common import escapeJsonValue
 from lib.core.common import evaluateCode
 from lib.core.common import extractRegexResult
+from lib.core.common import filterNone
 from lib.core.common import findMultipartPostBoundary
+from lib.core.common import getBytes
 from lib.core.common import getCurrentThreadData
 from lib.core.common import getHeader
 from lib.core.common import getHostHeader
@@ -58,10 +55,10 @@ from lib.core.common import singleTimeLogMessage
 from lib.core.common import singleTimeWarnMessage
 from lib.core.common import stdev
 from lib.core.common import wasLastResponseDelayed
-from lib.core.common import unicodeencode
 from lib.core.common import unsafeVariableNaming
 from lib.core.common import urldecode
 from lib.core.common import urlencode
+from lib.core.compat import xrange
 from lib.core.data import conf
 from lib.core.data import kb
 from lib.core.data import logger
@@ -85,6 +82,7 @@ from lib.core.exception import SqlmapConnectionException
 from lib.core.exception import SqlmapGenericException
 from lib.core.exception import SqlmapSyntaxException
 from lib.core.exception import SqlmapTokenException
+from lib.core.exception import SqlmapUserQuitException
 from lib.core.exception import SqlmapValueException
 from lib.core.settings import ASTERISK_MARKER
 from lib.core.settings import BOUNDARY_BACKSLASH_MARKER
@@ -92,7 +90,7 @@ from lib.core.settings import DEFAULT_CONTENT_TYPE
 from lib.core.settings import DEFAULT_COOKIE_DELIMITER
 from lib.core.settings import DEFAULT_GET_POST_DELIMITER
 from lib.core.settings import DEFAULT_USER_AGENT
-from lib.core.settings import EVALCODE_KEYWORD_SUFFIX
+from lib.core.settings import EVALCODE_ENCODED_PREFIX
 from lib.core.settings import HTTP_ACCEPT_HEADER_VALUE
 from lib.core.settings import HTTP_ACCEPT_ENCODING_HEADER_VALUE
 from lib.core.settings import MAX_CONNECTION_CHUNK_SIZE
@@ -123,7 +121,10 @@ from lib.request.basic import processResponse
 from lib.request.direct import direct
 from lib.request.comparison import comparison
 from lib.request.methodrequest import MethodRequest
-from thirdparty.odict.odict import OrderedDict
+from thirdparty import six
+from thirdparty.odict import OrderedDict
+from thirdparty.six.moves import http_client as _http_client
+from thirdparty.six.moves import urllib as _urllib
 from thirdparty.socks.socks import ProxyError
 
 class Connect(object):
@@ -192,7 +193,7 @@ class Connect(object):
 
     @staticmethod
     def _connReadProxy(conn):
-        retVal = ""
+        retVal = b""
 
         if not kb.dnsMode and conn:
             headers = conn.info()
@@ -272,14 +273,22 @@ class Connect(object):
         checking = kwargs.get("checking", False)
         skipRead = kwargs.get("skipRead", False)
         finalCode = kwargs.get("finalCode", False)
+        chunked = kwargs.get("chunked", False) or conf.chunked
 
         if multipart:
             post = multipart
 
+        if not post:
+            chunked = False
+
+        elif chunked:
+            post = _urllib.parse.unquote(post)
+            post = chunkSplitPostData(post)
+
         websocket_ = url.lower().startswith("ws")
 
-        if not urlparse.urlsplit(url).netloc:
+        if not _urllib.parse.urlsplit(url).netloc:
-            url = urlparse.urljoin(conf.url, url)
+            url = _urllib.parse.urljoin(conf.url, url)
 
         # flag to know if we are dealing with the same target host
         target = checkSameHost(url, conf.url)
@@ -300,7 +309,7 @@ class Connect(object):
         code = None
         status = None
 
-        _ = urlparse.urlsplit(url)
+        _ = _urllib.parse.urlsplit(url)
         requestMsg = u"HTTP request [#%d]:\r\n%s " % (threadData.lastRequestUID, method or (HTTPMETHOD.POST if post is not None else HTTPMETHOD.GET))
         requestMsg += getUnicode(("%s%s" % (_.path or "/", ("?%s" % _.query) if _.query else "")) if not any((refreshing, crawling, checking)) else url)
         responseMsg = u"HTTP response "
@@ -328,7 +337,7 @@ class Connect(object):
                 pass
 
             elif target:
-                if conf.forceSSL and urlparse.urlparse(url).scheme != "https":
+                if conf.forceSSL and _urllib.parse.urlparse(url).scheme != "https":
                     url = re.sub(r"(?i)\Ahttp:", "https:", url)
                     url = re.sub(r"(?i):80/", ":443/", url)
 
@@ -353,7 +362,7 @@ class Connect(object):
                 url = "%s?%s" % (url, get)
                 requestMsg += "?%s" % get
 
-            requestMsg += " %s" % httplib.HTTPConnection._http_vsn_str
+            requestMsg += " %s" % _http_client.HTTPConnection._http_vsn_str
 
             # Prepare HTTP headers
             headers = forgeHeaders({HTTP_HEADER.COOKIE: cookie, HTTP_HEADER.USER_AGENT: ua, HTTP_HEADER.REFERER: referer, HTTP_HEADER.HOST: host}, base=None if target else {})
@@ -398,18 +407,19 @@ class Connect(object):
             if conf.keepAlive:
                 headers[HTTP_HEADER.CONNECTION] = "keep-alive"
 
+            if chunked:
+                headers[HTTP_HEADER.TRANSFER_ENCODING] = "chunked"
+
             if auxHeaders:
                 headers = forgeHeaders(auxHeaders, headers)
 
-            for key, value in headers.items():
+            for key, value in list(headers.items()):
                 del headers[key]
-                value = unicodeencode(value, kb.pageEncoding)
                 for char in (r"\r", r"\n"):
                     value = re.sub(r"(%s)([^ \t])" % char, r"\g<1>\t\g<2>", value)
-                headers[unicodeencode(key, kb.pageEncoding)] = value.strip("\r\n")
+                headers[getBytes(key)] = getBytes(value.strip("\r\n"))
 
-            url = unicodeencode(url)
+            post = getBytes(post)
-            post = unicodeencode(post)
 
             if websocket_:
                 ws = websocket.WebSocket()
@@ -419,7 +429,7 @@ class Connect(object):
                 page = ws.recv()
                 ws.close()
                 code = ws.status
-                status = httplib.responses[code]
+                status = _http_client.responses[code]
 
                 class _(dict):
                     pass
@@ -427,7 +437,7 @@ class Connect(object):
                 responseHeaders = _(ws.getheaders())
                 responseHeaders.headers = ["%s: %s\r\n" % (_[0].capitalize(), _[1]) for _ in responseHeaders.items()]
 
-                requestHeaders += "\r\n".join(["%s: %s" % (getUnicode(key.capitalize() if isinstance(key, basestring) else key), getUnicode(value)) for (key, value) in responseHeaders.items()])
+                requestHeaders += "\r\n".join(["%s: %s" % (getUnicode(key.capitalize() if hasattr(key, "capitalize") else key), getUnicode(value)) for (key, value) in responseHeaders.items()])
                 requestMsg += "\r\n%s" % requestHeaders
 
                 if post is not None:
@@ -440,15 +450,15 @@ class Connect(object):
                 logger.log(CUSTOM_LOGGING.TRAFFIC_OUT, requestMsg)
             else:
                 if method and method not in (HTTPMETHOD.GET, HTTPMETHOD.POST):
-                    method = unicodeencode(method)
+                    method = getBytes(method)
                     req = MethodRequest(url, post, headers)
                     req.set_method(method)
                 elif url is not None:
-                    req = urllib2.Request(url, post, headers)
+                    req = _urllib.request.Request(url, post, headers)
                 else:
                     return None, None, None
 
-                requestHeaders += "\r\n".join(["%s: %s" % (getUnicode(key.capitalize() if isinstance(key, basestring) else key), getUnicode(value)) for (key, value) in req.header_items()])
+                requestHeaders += "\r\n".join(["%s: %s" % (getUnicode(key.capitalize() if hasattr(key, "capitalize") else key), getUnicode(value)) for (key, value) in req.header_items()])
 
                 if not getRequestHeader(req, HTTP_HEADER.COOKIE) and conf.cj:
                     conf.cj._policy._now = conf.cj._now = int(time.time())
@@ -456,7 +466,7 @@ class Connect(object):
                     requestHeaders += "\r\n%s" % ("Cookie: %s" % ";".join("%s=%s" % (getUnicode(cookie.name), getUnicode(cookie.value)) for cookie in cookies))
 
                 if post is not None:
-                    if not getRequestHeader(req, HTTP_HEADER.CONTENT_LENGTH):
+                    if not getRequestHeader(req, HTTP_HEADER.CONTENT_LENGTH) and not chunked:
                         requestHeaders += "\r\n%s: %d" % (string.capwords(HTTP_HEADER.CONTENT_LENGTH), len(post))
 
                 if not getRequestHeader(req, HTTP_HEADER.CONNECTION):
@@ -467,7 +477,8 @@ class Connect(object):
                 if post is not None:
                     requestMsg += "\r\n\r\n%s" % getUnicode(post)
 
-                requestMsg += "\r\n"
+                if not chunked:
+                    requestMsg += "\r\n"
 
                 if not multipart:
                     threadData.lastRequestMsg = requestMsg
@@ -482,7 +493,7 @@ class Connect(object):
                             for char in (r"\r", r"\n"):
                                 cookie.value = re.sub(r"(%s)([^ \t])" % char, r"\g<1>\t\g<2>", cookie.value)
 
-                conn = urllib2.urlopen(req)
+                conn = _urllib.request.urlopen(req)
 
                 if not kb.authHeader and getRequestHeader(req, HTTP_HEADER.AUTHORIZATION) and (conf.authType or "").lower() == AUTH_TYPE.BASIC.lower():
                     kb.authHeader = getRequestHeader(req, HTTP_HEADER.AUTHORIZATION)
@@ -538,7 +549,7 @@ class Connect(object):
                         if re.search(r"\Ahttps?://", refresh, re.I):
                             url = refresh
                         else:
-                            url = urlparse.urljoin(url, refresh)
+                            url = _urllib.parse.urljoin(url, refresh)
 
                         threadData.lastRedirectMsg = (threadData.lastRequestUID, page)
                         kwargs["refreshing"] = True
@@ -570,7 +581,7 @@ class Connect(object):
             else:
                 raise
 
-        except urllib2.HTTPError as ex:
+        except _urllib.error.HTTPError as ex:
             page = None
             responseHeaders = None
 
@@ -592,10 +603,10 @@ class Connect(object):
             except:
                 pass
             finally:
-                page = page if isinstance(page, unicode) else getUnicode(page)
+                page = getUnicode(page)
 
             code = ex.code
-            status = getSafeExString(ex)
+            status = getattr(ex, "reason", None) or getSafeExString(ex).split(": ", 1)[-1]
 
             kb.originalCode = kb.originalCode or code
             threadData.lastHTTPError = (threadData.lastRequestUID, code, status)
@@ -604,7 +615,7 @@ class Connect(object):
             responseMsg += "[#%d] (%s %s):\r\n" % (threadData.lastRequestUID, code, status)
 
             if responseHeaders:
-                logHeaders = "\r\n".join(["%s: %s" % (getUnicode(key.capitalize() if isinstance(key, basestring) else key), getUnicode(value)) for (key, value) in responseHeaders.items()])
+                logHeaders = getUnicode("".join(responseHeaders.headers).strip())
 
             logHTTPTraffic(requestMsg, "%s%s\r\n\r\n%s" % (responseMsg, logHeaders, (page or "")[:MAX_CONNECTION_CHUNK_SIZE]), start, time.time())
 
@@ -619,22 +630,28 @@ class Connect(object):
                 logger.log(CUSTOM_LOGGING.TRAFFIC_IN, responseMsg)
 
             if ex.code != conf.ignoreCode:
-                if ex.code == httplib.UNAUTHORIZED:
+                if ex.code == _http_client.UNAUTHORIZED:
                     errMsg = "not authorized, try to provide right HTTP "
                     errMsg += "authentication type and valid credentials (%d)" % code
                     raise SqlmapConnectionException(errMsg)
-                elif ex.code == httplib.NOT_FOUND:
+                elif chunked and ex.code in (_http_client.METHOD_NOT_ALLOWED, _http_client.LENGTH_REQUIRED):
+                    warnMsg = "turning off HTTP chunked transfer encoding "
+                    warnMsg += "as it seems that the target site doesn't support it (%d)" % code
+                    singleTimeWarnMessage(warnMsg)
+                    conf.chunked = kwargs["chunked"] = False
+                    return Connect.getPage(**kwargs)
+                elif ex.code == _http_client.NOT_FOUND:
                     if raise404:
                         errMsg = "page not found (%d)" % code
                         raise SqlmapConnectionException(errMsg)
                     else:
                         debugMsg = "page not found (%d)" % code
                         singleTimeLogMessage(debugMsg, logging.DEBUG)
-                elif ex.code == httplib.GATEWAY_TIMEOUT:
+                elif ex.code == _http_client.GATEWAY_TIMEOUT:
                     if ignoreTimeout:
                         return None if not conf.ignoreTimeouts else "", None, None
                     else:
-                        warnMsg = "unable to connect to the target URL (%d - %s)" % (ex.code, httplib.responses[ex.code])
+                        warnMsg = "unable to connect to the target URL (%d - %s)" % (ex.code, _http_client.responses[ex.code])
                         if threadData.retriesCount < conf.retries and not kb.threadException:
                             warnMsg += ". sqlmap is going to retry the request"
                             logger.critical(warnMsg)
@@ -648,7 +665,7 @@ class Connect(object):
                     debugMsg = "got HTTP error code: %d (%s)" % (code, status)
                     logger.debug(debugMsg)
 
-        except (urllib2.URLError, socket.error, socket.timeout, httplib.HTTPException, struct.error, binascii.Error, ProxyError, SqlmapCompressionException, WebSocketException, TypeError, ValueError):
+        except (_urllib.error.URLError, socket.error, socket.timeout, _http_client.HTTPException, struct.error, binascii.Error, ProxyError, SqlmapCompressionException, WebSocketException, TypeError, ValueError, OverflowError):
             tbMsg = traceback.format_exc()
 
             if checking:
@@ -716,8 +733,11 @@ class Connect(object):
 
                     kb.connErrorChoice = readInput(message, default='N', boolean=True)
 
-                if kb.connErrorChoice is False:
+                if kb.connErrorChoice is not None:
-                    raise SqlmapConnectionException(warnMsg)
+                    if kb.connErrorChoice:
+                        raise SqlmapConnectionException(warnMsg)
+                    else:
+                        raise SqlmapUserQuitException
 
             if "forcibly closed" in tbMsg:
                 logger.critical(warnMsg)
@@ -739,42 +759,54 @@ class Connect(object):
                 raise SqlmapConnectionException(warnMsg)
 
         finally:
-            if isinstance(page, basestring) and not isinstance(page, unicode):
+            if isinstance(page, six.binary_type):
                 if HTTP_HEADER.CONTENT_TYPE in (responseHeaders or {}) and not re.search(TEXT_CONTENT_TYPE_REGEX, responseHeaders[HTTP_HEADER.CONTENT_TYPE]):
-                    page = unicode(page, errors="ignore")
+                    page = six.text_type(page, errors="ignore")
                 else:
                     page = getUnicode(page)
+
+            for function in kb.preprocessFunctions:
+                try:
+                    page, responseHeaders, code = function(page, responseHeaders, code)
+                except Exception as ex:
+                    errMsg = "error occurred while running preprocess "
+                    errMsg += "function '%s' ('%s')" % (function.__name__, getSafeExString(ex))
+                    raise SqlmapGenericException(errMsg)
+
+            threadData.lastPage = page
+            threadData.lastCode = code
+
             socket.setdefaulttimeout(conf.timeout)
 
         processResponse(page, responseHeaders, status)
 
-        if conn and getattr(conn, "redurl", None):
-            _ = urlparse.urlsplit(conn.redurl)
-            _ = ("%s%s" % (_.path or "/", ("?%s" % _.query) if _.query else ""))
-            requestMsg = re.sub(r"(\n[A-Z]+ ).+?( HTTP/\d)", r"\g<1>%s\g<2>" % getUnicode(_).replace("\\", "\\\\"), requestMsg, 1)
-
-            if kb.resendPostOnRedirect is False:
-                requestMsg = re.sub(r"(\[#\d+\]:\n)POST ", r"\g<1>GET ", requestMsg)
-                requestMsg = re.sub(r"(?i)Content-length: \d+\n", "", requestMsg)
-                requestMsg = re.sub(r"(?s)\n\n.+", "\n", requestMsg)
-
-            responseMsg += "[#%d] (%d %s):\r\n" % (threadData.lastRequestUID, conn.code, status)
-        else:
-            responseMsg += "[#%d] (%s %s):\r\n" % (threadData.lastRequestUID, code, status)
-
-        if responseHeaders:
-            logHeaders = "\r\n".join(["%s: %s" % (getUnicode(key.capitalize() if isinstance(key, basestring) else key), getUnicode(value)) for (key, value) in responseHeaders.items()])
-
         if not skipLogTraffic:
+            if conn and getattr(conn, "redurl", None):
+                _ = _urllib.parse.urlsplit(conn.redurl)
+                _ = ("%s%s" % (_.path or "/", ("?%s" % _.query) if _.query else ""))
+                requestMsg = re.sub(r"(\n[A-Z]+ ).+?( HTTP/\d)", r"\g<1>%s\g<2>" % getUnicode(_).replace("\\", "\\\\"), requestMsg, 1)
+
+                if kb.resendPostOnRedirect is False:
+                    requestMsg = re.sub(r"(\[#\d+\]:\n)POST ", r"\g<1>GET ", requestMsg)
+                    requestMsg = re.sub(r"(?i)Content-length: \d+\n", "", requestMsg)
+                    requestMsg = re.sub(r"(?s)\n\n.+", "\n", requestMsg)
+
+                responseMsg += "[#%d] (%s %s):\r\n" % (threadData.lastRequestUID, conn.code, status)
+            elif "\n" not in responseMsg:
+                responseMsg += "[#%d] (%s %s):\r\n" % (threadData.lastRequestUID, code, status)
+
+            if responseHeaders:
+                logHeaders = getUnicode("".join(responseHeaders.headers).strip() if six.PY2 else responseHeaders.__bytes__())
+
             logHTTPTraffic(requestMsg, "%s%s\r\n\r\n%s" % (responseMsg, logHeaders, (page or "")[:MAX_CONNECTION_CHUNK_SIZE]), start, time.time())
 
-        if conf.verbose <= 5:
+            if conf.verbose <= 5:
-            responseMsg += getUnicode(logHeaders)
+                responseMsg += getUnicode(logHeaders)
-        elif conf.verbose > 5:
+            elif conf.verbose > 5:
-            responseMsg += "%s\r\n\r\n%s" % (logHeaders, (page or "")[:MAX_CONNECTION_CHUNK_SIZE])
+                responseMsg += "%s\r\n\r\n%s" % (logHeaders, (page or "")[:MAX_CONNECTION_CHUNK_SIZE])
 
-        if not multipart:
+            if not multipart:
-            logger.log(CUSTOM_LOGGING.TRAFFIC_IN, responseMsg)
+                logger.log(CUSTOM_LOGGING.TRAFFIC_IN, responseMsg)
 
         return page, responseHeaders, code
 
@@ -818,7 +850,7 @@ class Connect(object):
 
         if conf.httpHeaders:
             headers = OrderedDict(conf.httpHeaders)
-            contentType = max(headers[_] if _.upper() == HTTP_HEADER.CONTENT_TYPE.upper() else None for _ in headers)
+            contentType = max(headers[_] if _.upper() == HTTP_HEADER.CONTENT_TYPE.upper() else "" for _ in headers) or None
 
             if (kb.postHint or conf.skipUrlEncode) and postUrlEncode:
                 postUrlEncode = False
@@ -837,11 +869,11 @@ class Connect(object):
                         payload = function(payload=payload, headers=auxHeaders, delimiter=delimiter, hints=hints)
                     except Exception as ex:
                         errMsg = "error occurred while running tamper "
-                        errMsg += "function '%s' ('%s')" % (function.func_name, getSafeExString(ex))
+                        errMsg += "function '%s' ('%s')" % (function.__name__, getSafeExString(ex))
                         raise SqlmapGenericException(errMsg)
 
-                    if not isinstance(payload, basestring):
+                    if not isinstance(payload, six.string_types):
-                        errMsg = "tamper function '%s' returns " % function.func_name
+                        errMsg = "tamper function '%s' returns " % function.__name__
                         errMsg += "invalid payload type ('%s')" % type(payload)
                         raise SqlmapValueException(errMsg)
 
@@ -965,6 +997,10 @@ class Connect(object):
         if conf.csrfToken:
             def _adjustParameter(paramString, parameter, newValue):
                 retVal = paramString
+
+                if urlencode(parameter) in paramString:
+                    parameter = urlencode(parameter)
+
                 match = re.search(r"%s=[^&]*" % re.escape(parameter), paramString, re.I)
                 if match:
                     retVal = re.sub("(?i)%s" % re.escape(match.group(0)), ("%s=%s" % (parameter, newValue)).replace('\\', r'\\'), paramString)
@@ -972,10 +1008,13 @@ class Connect(object):
                     match = re.search(r"(%s[\"']:[\"'])([^\"']+)" % re.escape(parameter), paramString, re.I)
                     if match:
                         retVal = re.sub("(?i)%s" % re.escape(match.group(0)), "%s%s" % (match.group(1), newValue), paramString)
+
                 return retVal
 
             token = AttribDict()
             page, headers, code = Connect.getPage(url=conf.csrfUrl or conf.url, data=conf.data if conf.csrfUrl == conf.url else None, method=conf.method if conf.csrfUrl == conf.url else None, cookie=conf.parameters.get(PLACE.COOKIE), direct=True, silent=True, ua=conf.parameters.get(PLACE.USER_AGENT), referer=conf.parameters.get(PLACE.REFERER), host=conf.parameters.get(PLACE.HOST))
+            page = urldecode(page)  # for anti-CSRF tokens with special characters in their name (e.g. 'foo:bar=...')
+
             match = re.search(r"(?i)<input[^>]+\bname=[\"']?(?P<name>%s)\b[^>]*\bvalue=[\"']?(?P<value>[^>'\"]*)" % conf.csrfToken, page or "", re.I)
 
             if not match:
@@ -998,7 +1037,7 @@ class Connect(object):
                     token.value = "".join(chr(int(_)) for _ in match.group(1).replace(' ', "").split(','))
 
             if not token:
-                if conf.csrfUrl and conf.csrfToken and conf.csrfUrl != conf.url and code == httplib.OK:
+                if conf.csrfUrl and conf.csrfToken and conf.csrfUrl != conf.url and code == _http_client.OK:
                     if headers and "text/plain" in headers.get(HTTP_HEADER.CONTENT_TYPE, ""):
                         token.name = conf.csrfToken
                         token.value = page
@@ -1040,10 +1079,10 @@ class Connect(object):
         if conf.rParam:
             def _randomizeParameter(paramString, randomParameter):
                 retVal = paramString
-                match = re.search(r"(\A|\b)%s=(?P<value>[^&;]+)" % re.escape(randomParameter), paramString)
+                match = re.search(r"(\A|\b)%s=(?P<value>[^&;]*)" % re.escape(randomParameter), paramString)
                 if match:
                     origValue = match.group("value")
-                    retVal = re.sub(r"(\A|\b)%s=[^&;]+" % re.escape(randomParameter), "%s=%s" % (randomParameter, randomizeParameterValue(origValue)), paramString)
+                    retVal = re.sub(r"(\A|\b)%s=[^&;]*" % re.escape(randomParameter), "%s=%s" % (randomParameter, randomizeParameterValue(origValue)), paramString)
                 return retVal
 
             for randomParameter in conf.rParam:
@@ -1062,14 +1101,13 @@ class Connect(object):
             delimiter = conf.paramDel or DEFAULT_GET_POST_DELIMITER
             variables = {"uri": uri, "lastPage": threadData.lastPage, "_locals": locals()}
             originals = {}
-            keywords = keyword.kwlist
 
             if not get and PLACE.URI in conf.parameters:
-                query = urlparse.urlsplit(uri).query or ""
+                query = _urllib.parse.urlsplit(uri).query or ""
             else:
                 query = None
 
-            for item in filter(None, (get, post if not kb.postHint else None, query)):
+            for item in filterNone((get, post if not kb.postHint else None, query)):
                 for part in item.split(delimiter):
                     if '=' in part:
                         name, value = part.split('=', 1)
@@ -1077,8 +1115,6 @@ class Connect(object):
                         if safeVariableNaming(name) != name:
                             conf.evalCode = re.sub(r"\b%s\b" % re.escape(name), safeVariableNaming(name), conf.evalCode)
                             name = safeVariableNaming(name)
-                        elif name in keywords:
-                            name = "%s%s" % (name, EVALCODE_KEYWORD_SUFFIX)
                         value = urldecode(value, convall=True, spaceplus=(item == post and kb.postSpaceToPlus))
                         variables[name] = value
 
@@ -1090,31 +1126,29 @@ class Connect(object):
                         if safeVariableNaming(name) != name:
                             conf.evalCode = re.sub(r"\b%s\b" % re.escape(name), safeVariableNaming(name), conf.evalCode)
                             name = safeVariableNaming(name)
-                        elif name in keywords:
-                            name = "%s%s" % (name, EVALCODE_KEYWORD_SUFFIX)
                         value = urldecode(value, convall=True)
                         variables[name] = value
 
             while True:
                 try:
-                    compiler.parse(unicodeencode(conf.evalCode.replace(';', '\n')))
+                    compile(getBytes(conf.evalCode.replace(';', '\n')), "", "exec")
                 except SyntaxError as ex:
                     if ex.text:
                         original = replacement = ex.text.strip()
+
                         if '=' in original:
                             name, value = original.split('=', 1)
                             name = name.strip()
                             if safeVariableNaming(name) != name:
                                 replacement = re.sub(r"\b%s\b" % re.escape(name), safeVariableNaming(name), replacement)
-                            elif name in keywords:
-                                replacement = re.sub(r"\b%s\b" % re.escape(name), "%s%s" % (name, EVALCODE_KEYWORD_SUFFIX), replacement)
                         else:
                             for _ in re.findall(r"[A-Za-z_]+", original)[::-1]:
-                                if _ in keywords:
+                                if safeVariableNaming(_) != _:
-                                    replacement = replacement.replace(_, "%s%s" % (_, EVALCODE_KEYWORD_SUFFIX))
+                                    replacement = replacement.replace(_, safeVariableNaming(_))
                                     break
+
                         if original == replacement:
-                            conf.evalCode = conf.evalCode.replace(EVALCODE_KEYWORD_SUFFIX, "")
+                            conf.evalCode = conf.evalCode.replace(EVALCODE_ENCODED_PREFIX, "")
                             break
                         else:
                             conf.evalCode = conf.evalCode.replace(getUnicode(ex.text.strip(), UNICODE_ENCODING), replacement)
@@ -1127,11 +1161,6 @@ class Connect(object):
             evaluateCode(conf.evalCode, variables)
 
             for variable in list(variables.keys()):
-                if variable.endswith(EVALCODE_KEYWORD_SUFFIX):
-                    value = variables[variable]
-                    del variables[variable]
-                    variables[variable.replace(EVALCODE_KEYWORD_SUFFIX, "")] = value
-
                 if unsafeVariableNaming(variable) != variable:
                     value = variables[variable]
                     del variables[variable]
@@ -1141,7 +1170,7 @@ class Connect(object):
 
             for name, value in variables.items():
                 if name != "__builtins__" and originals.get(name, "") != value:
-                    if isinstance(value, (basestring, int)):
+                    if isinstance(value, (int, six.string_types)):
                         found = False
                         value = getUnicode(value, UNICODE_ENCODING)
 
@@ -1236,7 +1265,7 @@ class Connect(object):
                     warnMsg += "10 or more)"
                     logger.critical(warnMsg)
 
-        if conf.safeFreq > 0:
+        if (conf.safeFreq or 0) > 0:
             kb.queryCounter += 1
             if kb.queryCounter % conf.safeFreq == 0:
                 if conf.safeUrl:
@@ -1284,7 +1313,7 @@ class Connect(object):
 
         if conf.secondUrl:
             page, headers, code = Connect.getPage(url=conf.secondUrl, cookie=cookie, ua=ua, silent=silent, auxHeaders=auxHeaders, response=response, raise404=False, ignoreTimeout=timeBasedCompare, refreshing=True)
-        elif kb.secondReq and IPS_WAF_CHECK_PAYLOAD not in urllib.unquote(value or ""):
+        elif kb.secondReq and IPS_WAF_CHECK_PAYLOAD not in _urllib.parse.unquote(value or ""):
             def _(value):
                 if kb.customInjectionMark in (value or ""):
                     if payload is None:
@@ -1295,10 +1324,9 @@ class Connect(object):
             page, headers, code = Connect.getPage(url=_(kb.secondReq[0]), post=_(kb.secondReq[2]), method=kb.secondReq[1], cookie=kb.secondReq[3], silent=silent, auxHeaders=dict(auxHeaders, **dict(kb.secondReq[4])), response=response, raise404=False, ignoreTimeout=timeBasedCompare, refreshing=True)
 
         threadData.lastQueryDuration = calculateDeltaSeconds(start)
-        threadData.lastPage = page
-        threadData.lastCode = code
 
-        kb.originalCode = kb.originalCode or code
+        kb.originalCode = code if kb.originalCode is None else kb.originalCode
+        kb.originalPage = page if kb.originalPage is None else kb.originalPage
 
         if kb.testMode:
             kb.testQueryCount += 1

lib/request/direct.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
@@ -25,6 +25,7 @@ from lib.core.enums import CUSTOM_LOGGING
 from lib.core.enums import DBMS
 from lib.core.enums import EXPECTED
 from lib.core.enums import TIMEOUT_STATE
+from lib.core.settings import TAKEOVER_TABLE_PREFIX
 from lib.core.settings import UNICODE_ENCODING
 from lib.utils.timeout import timeout
 
@@ -53,7 +54,7 @@ def direct(query, content=True):
 
     if not select and "EXEC " not in query.upper():
         timeout(func=conf.dbmsConnector.execute, args=(query,), duration=conf.timeout, default=None)
-    elif not (output and "sqlmapoutput" not in query and "sqlmapfile" not in query):
+    elif not (output and ("%soutput" % TAKEOVER_TABLE_PREFIX) not in query and ("%sfile" % TAKEOVER_TABLE_PREFIX) not in query):
         output, state = timeout(func=conf.dbmsConnector.select, args=(query,), duration=conf.timeout, default=None)
         if state == TIMEOUT_STATE.NORMAL:
             hashDBWrite(query, output, True)

lib/request/dns.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)

lib/request/httpshandler.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
@@ -6,17 +6,18 @@ See the file 'LICENSE' for copying permission
 """
 
 import distutils.version
-import httplib
 import re
 import socket
-import urllib2
 
+from lib.core.common import filterNone
 from lib.core.common import getSafeExString
 from lib.core.data import conf
 from lib.core.data import kb
 from lib.core.data import logger
 from lib.core.exception import SqlmapConnectionException
 from lib.core.settings import PYVERSION
+from thirdparty.six.moves import http_client as _http_client
+from thirdparty.six.moves import urllib as _urllib
 
 ssl = None
 try:
@@ -25,9 +26,9 @@ try:
 except ImportError:
     pass
 
-_protocols = filter(None, (getattr(ssl, _, None) for _ in ("PROTOCOL_TLSv1_2", "PROTOCOL_TLSv1_1", "PROTOCOL_TLSv1", "PROTOCOL_SSLv3", "PROTOCOL_SSLv23", "PROTOCOL_SSLv2")))
+_protocols = filterNone(getattr(ssl, _, None) for _ in ("PROTOCOL_TLSv1_2", "PROTOCOL_TLSv1_1", "PROTOCOL_TLSv1", "PROTOCOL_SSLv3", "PROTOCOL_SSLv23", "PROTOCOL_SSLv2"))
 
-class HTTPSConnection(httplib.HTTPSConnection):
+class HTTPSConnection(_http_client.HTTPSConnection):
     """
     Connection class that enables usage of newer SSL protocols.
 
@@ -35,7 +36,7 @@ class HTTPSConnection(httplib.HTTPSConnection):
     """
 
     def __init__(self, *args, **kwargs):
-        httplib.HTTPSConnection.__init__(self, *args, **kwargs)
+        _http_client.HTTPSConnection.__init__(self, *args, **kwargs)
 
     def connect(self):
         def create_sock():
@@ -63,7 +64,7 @@ class HTTPSConnection(httplib.HTTPSConnection):
                         break
                     else:
                         sock.close()
-                except (ssl.SSLError, socket.error, httplib.BadStatusLine) as ex:
+                except (ssl.SSLError, socket.error, _http_client.BadStatusLine) as ex:
                     self._tunnel_host = None
                     logger.debug("SSL connection error occurred ('%s')" % getSafeExString(ex))
 
@@ -83,7 +84,7 @@ class HTTPSConnection(httplib.HTTPSConnection):
                         break
                     else:
                         sock.close()
-                except (ssl.SSLError, socket.error, httplib.BadStatusLine) as ex:
+                except (ssl.SSLError, socket.error, _http_client.BadStatusLine) as ex:
                     self._tunnel_host = None
                     logger.debug("SSL connection error occurred ('%s')" % getSafeExString(ex))
 
@@ -94,14 +95,6 @@ class HTTPSConnection(httplib.HTTPSConnection):
                 errMsg += " (please retry with Python >= 2.7.9)"
             raise SqlmapConnectionException(errMsg)
 
-class HTTPSHandler(urllib2.HTTPSHandler):
+class HTTPSHandler(_urllib.request.HTTPSHandler):
     def https_open(self, req):
-        return self.do_open(HTTPSConnection if ssl else httplib.HTTPSConnection, req)
+        return self.do_open(HTTPSConnection if ssl else _http_client.HTTPSConnection, req)
-
-# Bug fix (http://bugs.python.org/issue17849)
-
-def _(self, *args):
-    return self._readline()
-
-httplib.LineAndFileWrapper._readline = httplib.LineAndFileWrapper.readline
-httplib.LineAndFileWrapper.readline = _

lib/request/inject.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
@@ -17,6 +17,7 @@ from lib.core.common import calculateDeltaSeconds
 from lib.core.common import cleanQuery
 from lib.core.common import expandAsteriskForColumns
 from lib.core.common import extractExpectedValue
+from lib.core.common import filterNone
 from lib.core.common import getPublicTypeMembers
 from lib.core.common import getTechniqueData
 from lib.core.common import hashDBRetrieve
@@ -31,6 +32,7 @@ from lib.core.common import pushValue
 from lib.core.common import randomStr
 from lib.core.common import readInput
 from lib.core.common import singleTimeWarnMessage
+from lib.core.compat import xrange
 from lib.core.data import conf
 from lib.core.data import kb
 from lib.core.data import logger
@@ -59,6 +61,7 @@ from lib.techniques.dns.test import dnsTest
 from lib.techniques.dns.use import dnsUse
 from lib.techniques.error.use import errorUse
 from lib.techniques.union.use import unionUse
+from thirdparty import six
 
 def _goDns(payload, expression):
     value = None
@@ -334,7 +337,7 @@ def _goUnion(expression, unpack=True, dump=False):
 
     output = unionUse(expression, unpack=unpack, dump=dump)
 
-    if isinstance(output, basestring):
+    if isinstance(output, six.string_types):
         output = parseUnionPage(output)
 
     return output
@@ -346,8 +349,13 @@ def getValue(expression, blind=True, union=True, error=True, time=True, fromUser
     affected parameter.
     """
 
-    if conf.hexConvert:
+    if conf.hexConvert and expected != EXPECTED.BOOL and Backend.getIdentifiedDbms():
-        charsetType = CHARSET_TYPE.HEXADECIMAL
+        if not hasattr(queries[Backend.getIdentifiedDbms()], "hex"):
+            warnMsg = "switch '--hex' is currently not supported on DBMS %s" % Backend.getIdentifiedDbms()
+            singleTimeWarnMessage(warnMsg)
+            conf.hexConvert = False
+        else:
+            charsetType = CHARSET_TYPE.HEXADECIMAL
 
     kb.safeCharEncode = safeCharEncode
     kb.resumeValues = resumeValue
@@ -424,7 +432,7 @@ def getValue(expression, blind=True, union=True, error=True, time=True, fromUser
                     found = (value is not None) or (value is None and expectingNone) or count >= MAX_TECHNIQUES_PER_VALUE
 
                 if found and conf.dnsDomain:
-                    _ = "".join(filter(None, (key if isTechniqueAvailable(value) else None for key, value in {'E': PAYLOAD.TECHNIQUE.ERROR, 'Q': PAYLOAD.TECHNIQUE.QUERY, 'U': PAYLOAD.TECHNIQUE.UNION}.items())))
+                    _ = "".join(filterNone(key if isTechniqueAvailable(value) else None for key, value in {'E': PAYLOAD.TECHNIQUE.ERROR, 'Q': PAYLOAD.TECHNIQUE.QUERY, 'U': PAYLOAD.TECHNIQUE.UNION}.items()))
                     warnMsg = "option '--dns-domain' will be ignored "
                     warnMsg += "as faster techniques are usable "
                     warnMsg += "(%s) " % _
@@ -478,7 +486,7 @@ def getValue(expression, blind=True, union=True, error=True, time=True, fromUser
         singleTimeWarnMessage(warnMsg)
 
     # Dirty patch (safe-encoded unicode characters)
-    if isinstance(value, unicode) and "\\x" in value:
+    if isinstance(value, six.text_type) and "\\x" in value:
         try:
             candidate = eval(repr(value).replace("\\\\x", "\\x").replace("u'", "'", 1)).decode(conf.encoding or UNICODE_ENCODING)
             if "\\x" not in candidate:

lib/request/methodrequest.py

@@ -1,19 +1,19 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """
 
-import urllib2
+from thirdparty.six.moves import urllib as _urllib
 
-class MethodRequest(urllib2.Request):
+class MethodRequest(_urllib.request.Request):
     """
-    Used to create HEAD/PUT/DELETE/... requests with urllib2
+    Used to create HEAD/PUT/DELETE/... requests with urllib
     """
 
     def set_method(self, method):
         self.method = method.upper()
 
     def get_method(self):
-        return getattr(self, 'method', urllib2.Request.get_method(self))
+        return getattr(self, 'method', _urllib.request.Request.get_method(self))