Fixes #3560

*  Add the `--chunk` option to send requests in chunks

* solve the httplib&urllib2 content-legnth

* remove info

* Solve the error caused by the mix of get mode and chunk

* add CHUNKED_KEYWORDS `union`

.gitattributes

@@ -3,6 +3,8 @@
 *.md5 text eol=lf
 *.py text eol=lf
 *.xml text eol=lf
+LICENSE text eol=lf
+COMMITMENT text eol=lf
 
 *_ binary
 *.dll binary

README.md

@@ -1,6 +1,6 @@
 # sqlmap
 
-[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://api.travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![PyPI version](https://badge.fury.io/py/sqlmap.svg)](https://badge.fury.io/py/sqlmap) [![GitHub closed issues](https://img.shields.io/github/issues-closed-raw/sqlmapproject/sqlmap.svg?colorB=ff69b4)](https://github.com/sqlmapproject/sqlmap/issues?q=is%3Aissue+is%3Aclosed) [![Twitter](https://img.shields.io/badge/[email protected])](https://twitter.com/sqlmap)
+[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![PyPI version](https://badge.fury.io/py/sqlmap.svg)](https://badge.fury.io/py/sqlmap) [![GitHub closed issues](https://img.shields.io/github/issues-closed-raw/sqlmapproject/sqlmap.svg?colorB=ff69b4)](https://github.com/sqlmapproject/sqlmap/issues?q=is%3Aissue+is%3Aclosed) [![Twitter](https://img.shields.io/badge/[email protected])](https://twitter.com/sqlmap)
 
 sqlmap is an open source penetration testing tool that automates the process of detecting and exploiting SQL injection flaws and taking over of database servers. It comes with a powerful detection engine, many niche features for the ultimate penetration tester and a broad range of switches lasting from database fingerprinting, over data fetching from the database, to accessing the underlying file system and executing commands on the operating system via out-of-band connections.
 

doc/THANKS.md

@@ -764,6 +764,9 @@ ultramegaman, <seclists(at)ultramegaman.com>
 Vinicius, <viniciusmaxdaloop(at)gmail.com>
 * for reporting a minor bug
 
+w8ay
+* for contributing an implementation for chunked transfer-encoding (switch --chunked)
+
 wanglei, <wanglei(at)17uxi.cn>
 * for reporting a minor bug
 

doc/THIRD-PARTY.md

@@ -15,11 +15,6 @@ This file lists bundled packages and their associated licensing terms.
   Copyright (C) 2013, Jonathan Hartley.
 * The Fcrypt library located under thirdparty/fcrypt/.
   Copyright (C) 2000, 2001, 2004 Carey Evans.
-* The Odict library located under thirdparty/odict/.
-  Copyright (C) 2005, Nicola Larosa, Michael Foord.
-* The Oset library located under thirdparty/oset/.
-  Copyright (C) 2010, BlueDynamics Alliance, Austria.
-  Copyright (C) 2009, Raymond Hettinger, and others.
 * The PrettyPrint library located under thirdparty/prettyprint/.
   Copyright (C) 2010, Chris Hall.
 * The SocksiPy library located under thirdparty/socks/.
@@ -281,6 +276,10 @@ be bound by the terms and conditions of this License Agreement.
 
 * The bottle web framework library located under thirdparty/bottle/.
   Copyright (C) 2012, Marcel Hellkamp.
+* The ordereddict library located under thirdparty/odict/.
+  Copyright (C) 2009, Raymond Hettinger.
+* The six Python 2 and 3 compatibility library located under thirdparty/six/.
+  Copyright (C) 2010-2018, Benjamin Peterson.
 * The Termcolor library located under thirdparty/termcolor/.
   Copyright (C) 2008-2011, Volvox Development Team.
 

doc/translations/README-bg-BG.md

@@ -1,6 +1,6 @@
 # sqlmap
 
-[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://api.travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![Лиценз](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![Twitter](https://img.shields.io/badge/[email protected])](https://twitter.com/sqlmap)
+[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![Лиценз](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![Twitter](https://img.shields.io/badge/[email protected])](https://twitter.com/sqlmap)
 
 sqlmap e инструмент за тестване и проникване, с отворен код, който автоматизира процеса на откриване и използване на недостатъците на SQL база данните чрез SQL инжекция, която ги взима от сървъра. Снабден е с мощен детектор, множество специални функции за най-добрия тестер и широк спектър от функции, които могат да се използват за множество цели - извличане на данни от базата данни, достъп до основната файлова система и изпълняване на команди на операционната система.
 

doc/translations/README-es-MX.md

@@ -1,6 +1,6 @@
 # sqlmap
 
-[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://api.travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![Twitter](https://img.shields.io/badge/[email protected])](https://twitter.com/sqlmap)
+[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![Twitter](https://img.shields.io/badge/[email protected])](https://twitter.com/sqlmap)
 
 sqlmap es una herramienta para pruebas de penetración "penetration testing" de software libre que automatiza el proceso de detección y explotación de fallos mediante inyección de SQL además de tomar el control de servidores de bases de datos. Contiene un poderoso motor de detección, así como muchas de las funcionalidades escenciales para el "pentester" y una amplia gama de opciones desde la recopilación de información para identificar el objetivo conocido como "fingerprinting" mediante la extracción de información de la base de datos, hasta el acceso al sistema de archivos subyacente para ejecutar comandos en el sistema operativo a través de conexiones alternativas conocidas como "Out-of-band".
 

doc/translations/README-fr-FR.md

@@ -1,6 +1,6 @@
 # sqlmap
 
-[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://api.travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![Twitter](https://img.shields.io/badge/[email protected])](https://twitter.com/sqlmap)
+[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![Twitter](https://img.shields.io/badge/[email protected])](https://twitter.com/sqlmap)
 
 **sqlmap** est un outil Open Source de test d'intrusion. Cet outil permet d'automatiser le processus de détection et d'exploitation des failles d'injection SQL afin de prendre le contrôle des serveurs de base de données. __sqlmap__ dispose d'un puissant moteur de détection utilisant les techniques les plus récentes et les plus dévastatrices de tests d'intrusion comme L'Injection SQL, qui permet d'accéder à la base de données, au système de fichiers sous-jacent et permet aussi l'exécution des commandes sur le système d'exploitation.
 

doc/translations/README-gr-GR.md

@@ -1,6 +1,6 @@
 # sqlmap
 
-[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://api.travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![Twitter](https://img.shields.io/badge/[email protected])](https://twitter.com/sqlmap)
+[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![Twitter](https://img.shields.io/badge/[email protected])](https://twitter.com/sqlmap)
 
 Το sqlmap είναι πρόγραμμα ανοιχτού κώδικα, που αυτοματοποιεί την εύρεση και εκμετάλλευση ευπαθειών τύπου SQL Injection σε βάσεις δεδομένων. Έρχεται με μια δυνατή μηχανή αναγνώρισης ευπαθειών, πολλά εξειδικευμένα χαρακτηριστικά για τον απόλυτο penetration tester όπως και με ένα μεγάλο εύρος επιλογών αρχίζοντας από την αναγνώριση της βάσης δεδομένων, κατέβασμα δεδομένων της βάσης, μέχρι και πρόσβαση στο βαθύτερο σύστημα αρχείων και εκτέλεση εντολών στο απευθείας στο λειτουργικό μέσω εκτός ζώνης συνδέσεων.
 

doc/translations/README-hr-HR.md

@@ -1,6 +1,6 @@
 # sqlmap
 
-[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://api.travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![Twitter](https://img.shields.io/badge/[email protected])](https://twitter.com/sqlmap)
+[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![Twitter](https://img.shields.io/badge/[email protected])](https://twitter.com/sqlmap)
 
 sqlmap je alat namijenjen za penetracijsko testiranje koji automatizira proces detekcije i eksploatacije sigurnosnih propusta SQL injekcije te preuzimanje poslužitelja baze podataka. Dolazi s moćnim mehanizmom za detekciju, mnoštvom korisnih opcija za napredno penetracijsko testiranje te široki spektar opcija od onih za prepoznavanja baze podataka, preko dohvaćanja podataka iz baze, do pristupa zahvaćenom datotečnom sustavu i izvršavanja komandi na operacijskom sustavu korištenjem tzv. "out-of-band" veza.
 

doc/translations/README-id-ID.md

@@ -1,6 +1,6 @@
 # sqlmap
 
-[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://api.travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![Twitter](https://img.shields.io/badge/[email protected])](https://twitter.com/sqlmap)
+[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![Twitter](https://img.shields.io/badge/[email protected])](https://twitter.com/sqlmap)
 
 sqlmap merupakan alat _(tool)_ bantu _open source_ dalam melakukan tes penetrasi yang mengotomasi proses deteksi dan eksploitasi kelemahan _SQL injection_ dan pengambil-alihan server basisdata. sqlmap dilengkapi dengan pendeteksi canggih, fitur-fitur hanal bagi _penetration tester_, beragam cara untuk mendeteksi basisdata, hingga mengakses _file system_ dan mengeksekusi perintah dalam sistem operasi melalui koneksi _out-of-band_. 
 

doc/translations/README-it-IT.md

@@ -1,6 +1,6 @@
 # sqlmap
 
-[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://api.travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![Twitter](https://img.shields.io/badge/[email protected])](https://twitter.com/sqlmap)
+[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![Twitter](https://img.shields.io/badge/[email protected])](https://twitter.com/sqlmap)
 
 sqlmap è uno strumento open source per il penetration testing. Il suo scopo è quello di rendere automatico il processo di scoperta ed exploit di vulnerabilità di tipo SQL injection al fine di compromettere database online. Dispone di un potente motore per la ricerca di vulnerabilità, molti strumenti di nicchia anche per il più esperto penetration tester ed un'ampia gamma di controlli che vanno dal fingerprinting di database allo scaricamento di dati, fino all'accesso al file system sottostante e l'esecuzione di comandi nel sistema operativo attraverso connessioni out-of-band.
 

doc/translations/README-ja-JP.md

@@ -1,6 +1,6 @@
 # sqlmap
 
-[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://api.travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![Twitter](https://img.shields.io/badge/[email protected])](https://twitter.com/sqlmap)
+[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![Twitter](https://img.shields.io/badge/[email protected])](https://twitter.com/sqlmap)
 
 sqlmapはオープンソースのペネトレーションテスティングツールです。SQLインジェクションの脆弱性の検出、活用、そしてデータベースサーバ奪取のプロセスを自動化します。
 強力な検出エンジン、ペネトレーションテスターのための多くのニッチ機能、持続的なデータベースのフィンガープリンティングから、データベースのデータ取得やアウトオブバンド接続を介したオペレーティング・システム上でのコマンド実行、ファイルシステムへのアクセスなどの広範囲に及ぶスイッチを提供します。

doc/translations/README-pl-PL.md

@@ -1,6 +1,6 @@
 # sqlmap
 
-[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://api.travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![Twitter](https://img.shields.io/badge/[email protected])](https://twitter.com/sqlmap)
+[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![Twitter](https://img.shields.io/badge/[email protected])](https://twitter.com/sqlmap)
 
 sqlmap to open sourceowe narzędzie do testów penetracyjnych, które automatyzuje procesy detekcji, przejmowania i testowania odporności serwerów SQL na podatność na iniekcję niechcianego kodu. Zawiera potężny mechanizm detekcji, wiele niszowych funkcji dla zaawansowanych testów penetracyjnych oraz szeroki wachlarz opcji począwszy od identyfikacji bazy danych, poprzez wydobywanie z nich danych, a nawet pozwalającuch na dostęp do systemu plików o uruchamianie poleceń w systemie operacyjnym serwera poprzez niestandardowe połączenia.
 

doc/translations/README-pt-BR.md

@@ -1,6 +1,6 @@
 # sqlmap
 
-[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://api.travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![Twitter](https://img.shields.io/badge/[email protected])](https://twitter.com/sqlmap)
+[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![Twitter](https://img.shields.io/badge/[email protected])](https://twitter.com/sqlmap)
 
 sqlmap é uma ferramenta de teste de penetração de código aberto que automatiza o processo de detecção e exploração de falhas de injeção SQL. Com essa ferramenta é possível assumir total controle de servidores de banco de dados em páginas web vulneráveis, inclusive de base de dados fora do sistema invadido. Ele possui um motor de detecção poderoso, empregando as últimas e mais devastadoras técnicas de teste de penetração por SQL Injection, que permite acessar a base de dados, o sistema de arquivos subjacente e executar comandos no sistema operacional.
 

doc/translations/README-ru-RUS.md

@@ -1,6 +1,6 @@
 # sqlmap
 
-[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://api.travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![Twitter](https://img.shields.io/badge/[email protected])](https://twitter.com/sqlmap)
+[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![Twitter](https://img.shields.io/badge/[email protected])](https://twitter.com/sqlmap)
 
 sqlmap - это инструмент для тестирования уязвимостей с открытым исходным кодом, который автоматизирует процесс обнаружения и использования ошибок SQL-инъекций и захвата серверов баз данных. Он оснащен мощным механизмом обнаружения, множеством приятных функций для профессионального тестера уязвимостей и широким спектром скриптов, которые упрощают работу с базами данных, от сбора данных из базы данных, до доступа к базовой файловой системе и выполнения команд в операционной системе через out-of-band соединение.
 

doc/translations/README-tr-TR.md

@@ -1,6 +1,6 @@
 # sqlmap
 
-[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://api.travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![Twitter](https://img.shields.io/badge/[email protected])](https://twitter.com/sqlmap)
+[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![Twitter](https://img.shields.io/badge/[email protected])](https://twitter.com/sqlmap)
 
 sqlmap sql injection açıklarını otomatik olarak tespit ve istismar etmeye yarayan açık kaynak bir penetrasyon aracıdır. sqlmap gelişmiş tespit özelliğinin yanı sıra penetrasyon testleri sırasında gerekli olabilecek bir çok aracı, -uzak veritabınınından, veri indirmek, dosya sistemine erişmek, dosya çalıştırmak gibi - işlevleri de barındırmaktadır.
 

doc/translations/README-uk-UA.md

@@ -1,6 +1,6 @@
 # sqlmap
 
-[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://api.travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![PyPI version](https://badge.fury.io/py/sqlmap.svg)](https://badge.fury.io/py/sqlmap) [![GitHub closed issues](https://img.shields.io/github/issues-closed-raw/sqlmapproject/sqlmap.svg?colorB=ff69b4)](https://github.com/sqlmapproject/sqlmap/issues?q=is%3Aissue+is%3Aclosed) [![Twitter](https://img.shields.io/badge/[email protected])](https://twitter.com/sqlmap)
+[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![PyPI version](https://badge.fury.io/py/sqlmap.svg)](https://badge.fury.io/py/sqlmap) [![GitHub closed issues](https://img.shields.io/github/issues-closed-raw/sqlmapproject/sqlmap.svg?colorB=ff69b4)](https://github.com/sqlmapproject/sqlmap/issues?q=is%3Aissue+is%3Aclosed) [![Twitter](https://img.shields.io/badge/[email protected])](https://twitter.com/sqlmap)
 
 sqlmap - це інструмент для тестування вразливостей з відкритим сирцевим кодом, який автоматизує процес виявлення і використання дефектів SQL-ін'єкцій, а також захоплення серверів баз даних. Він оснащений потужним механізмом виявлення, безліччю приємних функцій для професійного тестувальника вразливостей і широким спектром скриптів, які спрощують роботу з базами даних - від відбитка бази даних до доступу до базової файлової системи та виконання команд в операційній системі через out-of-band з'єднання.
 

doc/translations/README-zh-CN.md

@@ -1,6 +1,6 @@
 # sqlmap
 
-[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://api.travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![Twitter](https://img.shields.io/badge/[email protected])](https://twitter.com/sqlmap)
+[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![Twitter](https://img.shields.io/badge/[email protected])](https://twitter.com/sqlmap)
 
 sqlmap 是一个开源的渗透测试工具，可以用来自动化的检测，利用SQL注入漏洞，获取数据库服务器的权限。它具有功能强大的检测引擎,针对各种不同类型数据库的渗透测试的功能选项，包括获取数据库中存储的数据，访问操作系统文件甚至可以通过外带数据连接的方式执行操作系统命令。
 

extra/__init__.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)

extra/beep/beep.py

@@ -8,7 +8,6 @@ See the file 'LICENSE' for copying permission
 """
 
 import os
-import subprocess
 import sys
 import wave
 
@@ -16,11 +15,11 @@ BEEP_WAV_FILENAME = os.path.join(os.path.dirname(__file__), "beep.wav")
 
 def beep():
     try:
-        if subprocess.mswindows:
+        if sys.platform == "nt":
             _win_wav_play(BEEP_WAV_FILENAME)
         elif sys.platform == "darwin":
             _mac_beep()
-        elif sys.platform == "linux2":
+        elif sys.platform.startswith("linux"):
             _linux_wav_play(BEEP_WAV_FILENAME)
         else:
             _speaker_beep()

extra/cloak/cloak.py

@@ -10,19 +10,21 @@ See the file 'LICENSE' for copying permission
 from __future__ import print_function
 
 import os
+import struct
 import sys
 import zlib
 
 from optparse import OptionError
 from optparse import OptionParser
 
+if sys.version_info >= (3, 0):
+    xrange = range
+
 def hideAscii(data):
-    retVal = ""
+    retVal = b""
     for i in xrange(len(data)):
-        if ord(data[i]) < 128:
+        value = data[i] if isinstance(data[i], int) else ord(data[i])
-            retVal += chr(ord(data[i]) ^ 127)
+        retVal += struct.pack('B', value ^ (127 if value < 128 else 0))
-        else:
-            retVal += data[i]
 
     return retVal
 
@@ -39,7 +41,8 @@ def decloak(inputFile=None, data=None):
             data = f.read()
     try:
         data = zlib.decompress(hideAscii(data))
-    except:
+    except Exception as ex:
+        print(ex)
         print('ERROR: the provided input file \'%s\' does not contain valid cloaked content' % inputFile)
         sys.exit(1)
     finally:

extra/dbgtool/dbgtool.py

@@ -11,7 +11,6 @@ from __future__ import print_function
 
 import os
 import sys
-import struct
 
 from optparse import OptionError
 from optparse import OptionParser
@@ -34,7 +33,7 @@ def convert(inputFile):
     fileContent = fp.read()
 
     for fileChar in fileContent:
-        unsignedFileChar = struct.unpack("B", fileChar)[0]
+        unsignedFileChar = fileChar if sys.version_info >= (3, 0) else ord(fileChar)
 
         if unsignedFileChar != 0:
             counter2 += 1

extra/icmpsh/icmpsh_m.py

@@ -22,7 +22,6 @@
 import os
 import select
 import socket
-import subprocess
 import sys
 
 def setNonBlocking(fd):
@@ -37,7 +36,7 @@ def setNonBlocking(fd):
     fcntl.fcntl(fd, fcntl.F_SETFL, flags)
 
 def main(src, dst):
-    if subprocess.mswindows:
+    if sys.platform == "nt":
         sys.stderr.write('icmpsh master can only run on Posix systems\n')
         sys.exit(255)
 

extra/safe2bin/__init__.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)

extra/safe2bin/safe2bin.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 safe2bin.py - Simple safe(hex) to binary format converter
@@ -18,6 +18,9 @@ import sys
 from optparse import OptionError
 from optparse import OptionParser
 
+if sys.version_info >= (3, 0):
+    xrange = range
+
 # Regex used for recognition of hex encoded characters
 HEX_ENCODED_CHAR_REGEX = r"(?P<result>\\x[0-9A-Fa-f]{2})"
 

extra/shutils/drei.sh

@@ -0,0 +1,7 @@
+#!/bin/bash
+
+# Copyright (c) 2006-2013 sqlmap developers (http://sqlmap.org/)
+# See the file 'LICENSE' for copying permission
+
+# Stress test against Python3
+for i in $(find . -iname "*.py" | grep -v __init__); do python3 -c 'import '`echo $i | cut -d '.' -f 2 | cut -d '/' -f 2- | sed 's/\//./g'`''; done

extra/shutils/duplicates.py

@@ -9,21 +9,22 @@ from __future__ import print_function
 
 import sys
 
-if len(sys.argv) > 0:
+if __name__ == "__main__":
-    items = list()
+    if len(sys.argv) > 1:
+        items = list()
 
-    with open(sys.argv[1], 'r') as f:
+        with open(sys.argv[1], 'r') as f:
-        for item in f.readlines():
+            for item in f.readlines():
-            item = item.strip()
+                item = item.strip()
-            try:
+                try:
-                str.encode(item)
+                    str.encode(item)
-                if item in items:
+                    if item in items:
-                    if item:
+                        if item:
-                        print(item)
+                            print(item)
-                else:
+                    else:
-                    items.append(item)
+                        items.append(item)
-            except:
+                except:
-                pass
+                    pass
 
-    with open(sys.argv[1], 'w+') as f:
+        with open(sys.argv[1], 'w+') as f:
-        f.writelines("\n".join(items))
+            f.writelines("\n".join(items))

extra/shutils/newlines.py

@@ -1,8 +1,5 @@
 #! /usr/bin/env python
 
-# Runs pylint on all python scripts found in a directory tree
-# Reference: http://rowinggolfer.blogspot.com/2009/08/pylint-recursively.html
-
 from __future__ import print_function
 
 import os
@@ -11,9 +8,10 @@ import sys
 def check(filepath):
     if filepath.endswith(".py"):
         content = open(filepath, "rb").read()
+        pattern = "\n\n\n".encode("ascii")
 
-        if "\n\n\n" in content:
+        if pattern in content:
-            index = content.find("\n\n\n")
+            index = content.find(pattern)
             print(filepath, repr(content[index - 30:index + 30]))
 
 if __name__ == "__main__":

extra/shutils/precommit-hook.sh

@@ -12,13 +12,11 @@ chmod +x .git/hooks/pre-commit
 
 PROJECT="../../"
 SETTINGS="../../lib/core/settings.py"
-CHECKSUM="../../txt/checksum.md5"
 
 declare -x SCRIPTPATH="${0}"
 
 PROJECT_FULLPATH=${SCRIPTPATH%/*}/$PROJECT
 SETTINGS_FULLPATH=${SCRIPTPATH%/*}/$SETTINGS
-CHECKSUM_FULLPATH=${SCRIPTPATH%/*}/$CHECKSUM
 
 git diff $SETTINGS_FULLPATH | grep "VERSION =" > /dev/null && exit 0
 
@@ -37,6 +35,3 @@ then
     fi
     git add "$SETTINGS_FULLPATH"
 fi
-
-truncate -s 0 "$CHECKSUM_FULLPATH"
-cd $PROJECT_FULLPATH && for i in $(find . -name "*.py" -o -name "*.xml" -o -name "*_" -o -type f -regex "./[^./]*" | sort); do git ls-files $i --error-unmatch &>/dev/null && md5sum $i | stdbuf -i0 -o0 -e0 sed 's/\.\///' >> "$CHECKSUM_FULLPATH"; git add "$CHECKSUM_FULLPATH"; done

extra/shutils/pypi.sh

@@ -26,7 +26,8 @@ setup(
     name='sqlmap',
     version='$VERSION',
     description='Automatic SQL injection and database takeover tool',
-    long_description='sqlmap is an open source penetration testing tool that automates the process of detecting and exploiting SQL injection flaws and taking over of database servers. It comes with a powerful detection engine, many niche features for the ultimate penetration tester and a broad range of switches lasting from database fingerprinting, over data fetching from the database, to accessing the underlying file system and executing commands on the operating system via out-of-band connections.',
+    long_description=open('README.rst').read(),
+    long_description_content_type='text/x-rst',
     author='Bernardo Damele Assumpcao Guimaraes, Miroslav Stampar',
     author_email='[email protected], [email protected]',
     url='http://sqlmap.org',
@@ -158,7 +159,7 @@ Links
 -  User's manual: https://github.com/sqlmapproject/sqlmap/wiki
 -  Frequently Asked Questions (FAQ):
    https://github.com/sqlmapproject/sqlmap/wiki/FAQ
--  Twitter: [@sqlmap](https://twitter.com/sqlmap)
+-  Twitter: https://twitter.com/sqlmap
 -  Demos: http://www.youtube.com/user/inquisb/videos
 -  Screenshots: https://github.com/sqlmapproject/sqlmap/wiki/Screenshots
 
@@ -176,7 +177,6 @@ Links
 EOF
 sed -i "s/^VERSION =.*/VERSION = \"$VERSION\"/g" sqlmap/lib/core/settings.py
 sed -i "s/^TYPE =.*/TYPE = \"$TYPE\"/g" sqlmap/lib/core/settings.py
-sed -i "s/.*lib\/core\/settings\.py/`md5sum sqlmap/lib/core/settings.py | cut -d ' ' -f 1`  lib\/core\/settings\.py/g" sqlmap/txt/checksum.md5
 for file in $(find sqlmap -type f | grep -v -E "\.(git|yml)"); do echo include $file >> MANIFEST.in; done
 python setup.py sdist upload
 rm -rf $TMP_DIR

extra/shutils/regressiontest.py

@@ -1,166 +0,0 @@
-#!/usr/bin/env python
-
-# Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
-# See the file 'LICENSE' for copying permission
-
-from __future__ import print_function
-
-import codecs
-import inspect
-import os
-import re
-import smtplib
-import subprocess
-import sys
-import time
-import traceback
-
-from email.mime.multipart import MIMEMultipart
-from email.mime.text import MIMEText
-
-sys.path.append(os.path.normpath("%s/../../" % os.path.dirname(inspect.getfile(inspect.currentframe()))))
-
-from lib.core.revision import getRevisionNumber
-
-START_TIME = time.strftime("%H:%M:%S %d-%m-%Y", time.gmtime())
-SQLMAP_HOME = "/opt/sqlmap"
-
-SMTP_SERVER = "127.0.0.1"
-SMTP_PORT = 25
-SMTP_TIMEOUT = 30
-FROM = "[email protected]"
-# TO = "[email protected]"
-TO = ["[email protected]", "[email protected]"]
-SUBJECT = "regression test started on %s using revision %s" % (START_TIME, getRevisionNumber())
-TARGET = "debian"
-
-def prepare_email(content):
-    global FROM
-    global TO
-    global SUBJECT
-
-    msg = MIMEMultipart()
-    msg["Subject"] = SUBJECT
-    msg["From"] = FROM
-    msg["To"] = TO if isinstance(TO, basestring) else ','.join(TO)
-
-    msg.attach(MIMEText(content))
-
-    return msg
-
-def send_email(msg):
-    global SMTP_SERVER
-    global SMTP_PORT
-    global SMTP_TIMEOUT
-
-    try:
-        s = smtplib.SMTP(host=SMTP_SERVER, port=SMTP_PORT, timeout=SMTP_TIMEOUT)
-        s.sendmail(FROM, TO, msg.as_string())
-        s.quit()
-    # Catch all for SMTP exceptions
-    except smtplib.SMTPException as ex:
-        print("Failure to send email: '%s" % ex)
-
-def failure_email(msg):
-    msg = prepare_email(msg)
-    send_email(msg)
-    sys.exit(1)
-
-def main():
-    global SUBJECT
-
-    content = ""
-    test_counts = []
-    attachments = {}
-
-    updateproc = subprocess.Popen("cd /opt/sqlmap/ ; python /opt/sqlmap/sqlmap.py --update", shell=True, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
-    stdout, stderr = updateproc.communicate()
-
-    if stderr:
-        failure_email("Update of sqlmap failed with error:\n\n%s" % stderr)
-
-    regressionproc = subprocess.Popen("python /opt/sqlmap/sqlmap.py --live-test", shell=True, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, close_fds=False)
-    stdout, stderr = regressionproc.communicate()
-
-    if stderr:
-        failure_email("Execution of regression test failed with error:\n\n%s" % stderr)
-
-    failed_tests = re.findall(r"running live test case: (.+?) \((\d+)\/\d+\)[\r]*\n.+test failed (at parsing items: (.+))?\s*\- scan folder: (\/.+) \- traceback: (.*?)( - SQL injection not detected)?[\r]*\n", stdout)
-
-    for failed_test in failed_tests:
-        title = failed_test[0]
-        test_count = int(failed_test[1])
-        parse = failed_test[3] if failed_test[3] else None
-        output_folder = failed_test[4]
-        traceback = False if failed_test[5] == "False" else bool(failed_test[5])
-        detected = False if failed_test[6] else True
-
-        test_counts.append(test_count)
-
-        console_output_file = os.path.join(output_folder, "console_output")
-        log_file = os.path.join(output_folder, TARGET, "log")
-        traceback_file = os.path.join(output_folder, "traceback")
-
-        if os.path.exists(console_output_file):
-            console_output_fd = codecs.open(console_output_file, "rb", "utf8")
-            console_output = console_output_fd.read()
-            console_output_fd.close()
-            attachments[test_count] = str(console_output)
-
-        if os.path.exists(log_file):
-            log_fd = codecs.open(log_file, "rb", "utf8")
-            log = log_fd.read()
-            log_fd.close()
-
-        if os.path.exists(traceback_file):
-            traceback_fd = codecs.open(traceback_file, "rb", "utf8")
-            traceback = traceback_fd.read()
-            traceback_fd.close()
-
-        content += "Failed test case '%s' (#%d)" % (title, test_count)
-
-        if parse:
-            content += " at parsing: %s:\n\n" % parse
-            content += "### Log file:\n\n"
-            content += "%s\n\n" % log
-        elif not detected:
-            content += " - SQL injection not detected\n\n"
-        else:
-            content += "\n\n"
-
-        if traceback:
-            content += "### Traceback:\n\n"
-            content += "%s\n\n" % str(traceback)
-
-        content += "#######################################################################\n\n"
-
-    end_string = "Regression test finished at %s" % time.strftime("%H:%M:%S %d-%m-%Y", time.gmtime())
-
-    if content:
-        content += end_string
-        SUBJECT = "Failed %s (%s)" % (SUBJECT, ", ".join("#%d" % count for count in test_counts))
-
-        msg = prepare_email(content)
-
-        for test_count, attachment in attachments.items():
-            attachment = MIMEText(attachment)
-            attachment.add_header("Content-Disposition", "attachment", filename="test_case_%d_console_output.txt" % test_count)
-            msg.attach(attachment)
-
-        send_email(msg)
-    else:
-        SUBJECT = "Successful %s" % SUBJECT
-        msg = prepare_email("All test cases were successful\n\n%s" % end_string)
-        send_email(msg)
-
-if __name__ == "__main__":
-    log_fd = open("/tmp/sqlmapregressiontest.log", "wb")
-    log_fd.write("Regression test started at %s\n" % START_TIME)
-
-    try:
-        main()
-    except Exception:
-        log_fd.write("An exception has occurred:\n%s" % str(traceback.format_exc()))
-
-    log_fd.write("Regression test finished at %s\n\n" % time.strftime("%H:%M:%S %d-%m-%Y", time.gmtime()))
-    log_fd.close()

extra/sqlharvest/sqlharvest.py

@@ -1,143 +0,0 @@
-#!/usr/bin/env python
-
-"""
-Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
-See the file 'LICENSE' for copying permission
-"""
-
-from __future__ import print_function
-
-import cookielib
-import re
-import socket
-import sys
-import urllib
-import urllib2
-import ConfigParser
-
-from operator import itemgetter
-
-TIMEOUT = 10
-CONFIG_FILE = 'sqlharvest.cfg'
-TABLES_FILE = 'tables.txt'
-USER_AGENT = 'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; AskTB5.3)'
-SEARCH_URL = 'http://www.google.com/m?source=mobileproducts&dc=gorganic'
-MAX_FILE_SIZE = 2 * 1024 * 1024  # if a result (.sql) file for downloading is more than 2MB in size just skip it
-QUERY = 'CREATE TABLE ext:sql'
-REGEX_URLS = r';u=([^"]+?)&q='
-REGEX_RESULT = r'(?i)CREATE TABLE\s*(/\*.*\*/)?\s*(IF NOT EXISTS)?\s*(?P<result>[^\(;]+)'
-
-def main():
-    tables = dict()
-    cookies = cookielib.CookieJar()
-    cookie_processor = urllib2.HTTPCookieProcessor(cookies)
-    opener = urllib2.build_opener(cookie_processor)
-    opener.addheaders = [("User-Agent", USER_AGENT)]
-
-    conn = opener.open(SEARCH_URL)
-    page = conn.read()  # set initial cookie values
-
-    config = ConfigParser.ConfigParser()
-    config.read(CONFIG_FILE)
-
-    if not config.has_section("options"):
-        config.add_section("options")
-    if not config.has_option("options", "index"):
-        config.set("options", "index", "0")
-
-    i = int(config.get("options", "index"))
-
-    try:
-        with open(TABLES_FILE, 'r') as f:
-            for line in f.xreadlines():
-                if len(line) > 0 and ',' in line:
-                    temp = line.split(',')
-                    tables[temp[0]] = int(temp[1])
-    except:
-        pass
-
-    socket.setdefaulttimeout(TIMEOUT)
-
-    files, old_files = None, None
-    try:
-        while True:
-            abort = False
-            old_files = files
-            files = []
-
-            try:
-                conn = opener.open("%s&q=%s&start=%d&sa=N" % (SEARCH_URL, QUERY.replace(' ', '+'), i * 10))
-                page = conn.read()
-                for match in re.finditer(REGEX_URLS, page):
-                    files.append(urllib.unquote(match.group(1)))
-                    if len(files) >= 10:
-                        break
-                abort = (files == old_files)
-
-            except KeyboardInterrupt:
-                raise
-
-            except Exception as ex:
-                print(ex)
-
-            if abort:
-                break
-
-            sys.stdout.write("\n---------------\n")
-            sys.stdout.write("Result page #%d\n" % (i + 1))
-            sys.stdout.write("---------------\n")
-
-            for sqlfile in files:
-                print(sqlfile)
-
-                try:
-                    req = urllib2.Request(sqlfile)
-                    response = urllib2.urlopen(req)
-
-                    if "Content-Length" in response.headers:
-                        if int(response.headers.get("Content-Length")) > MAX_FILE_SIZE:
-                            continue
-
-                    page = response.read()
-                    found = False
-                    counter = 0
-
-                    for match in re.finditer(REGEX_RESULT, page):
-                        counter += 1
-                        table = match.group("result").strip().strip("`\"'").replace('"."', ".").replace("].[", ".").strip('[]')
-
-                        if table and not any(_ in table for _ in ('>', '<', '--', ' ')):
-                            found = True
-                            sys.stdout.write('*')
-
-                            if table in tables:
-                                tables[table] += 1
-                            else:
-                                tables[table] = 1
-                    if found:
-                        sys.stdout.write("\n")
-
-                except KeyboardInterrupt:
-                    raise
-
-                except Exception as ex:
-                    print(ex)
-
-            else:
-                i += 1
-
-    except KeyboardInterrupt:
-        pass
-
-    finally:
-        with open(TABLES_FILE, 'w+') as f:
-            tables = sorted(tables.items(), key=itemgetter(1), reverse=True)
-            for table, count in tables:
-                f.write("%s,%d\n" % (table, count))
-
-        config.set("options", "index", str(i + 1))
-        with open(CONFIG_FILE, 'w+') as f:
-            config.write(f)
-
-if __name__ == "__main__":
-    main()

extra/vulnserver/vulnserver.py

@@ -0,0 +1,148 @@
+#!/usr/bin/env python
+
+"""
+vulnserver.py - Trivial SQLi vulnerable HTTP server (Note: for testing purposes)
+
+Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
+See the file 'LICENSE' for copying permission
+"""
+
+from __future__ import print_function
+
+import re
+import sqlite3
+import sys
+import traceback
+
+if sys.version_info >= (3, 0):
+    from http.client import FOUND
+    from http.client import NOT_FOUND
+    from http.client import OK
+    from http.server import BaseHTTPRequestHandler
+    from http.server import HTTPServer
+    from socketserver import ThreadingMixIn
+    from urllib.parse import parse_qs
+    from urllib.parse import unquote_plus
+else:
+    from BaseHTTPServer import BaseHTTPRequestHandler
+    from BaseHTTPServer import HTTPServer
+    from httplib import FOUND
+    from httplib import NOT_FOUND
+    from httplib import OK
+    from SocketServer import ThreadingMixIn
+    from urlparse import parse_qs
+    from urllib import unquote_plus
+
+SCHEMA = """
+    CREATE TABLE users (
+        id INTEGER,
+        name TEXT,
+        surname TEXT
+    );
+    INSERT INTO users (id, name, surname) VALUES (1, 'luther', 'blisset');
+    INSERT INTO users (id, name, surname) VALUES (2, 'fluffy', 'bunny');
+    INSERT INTO users (id, name, surname) VALUES (3, 'wu', 'ming');
+    INSERT INTO users (id, name, surname) VALUES (4, 'sqlmap/1.0-dev (http://sqlmap.org)', 'user agent header');
+    INSERT INTO users (id, name, surname) VALUES (5, NULL, 'nameisnull');
+"""
+
+LISTEN_ADDRESS = "localhost"
+LISTEN_PORT = 8440
+
+_conn = None
+_cursor = None
+_server = None
+
+def init():
+    global _conn
+    global _cursor
+
+    _conn = sqlite3.connect(":memory:", isolation_level=None, check_same_thread=False)
+    _cursor = _conn.cursor()
+
+    _cursor.executescript(SCHEMA)
+
+class ThreadingServer(ThreadingMixIn, HTTPServer):
+    def finish_request(self, *args, **kwargs):
+        try:
+            HTTPServer.finish_request(self, *args, **kwargs)
+        except Exception:
+            traceback.print_exc()
+
+class ReqHandler(BaseHTTPRequestHandler):
+    def do_REQUEST(self):
+        path, query = self.path.split('?', 1) if '?' in self.path else (self.path, "")
+        params = {}
+
+        if query:
+            params.update(parse_qs(query))
+
+        if hasattr(self, "data"):
+            params.update(parse_qs(self.data))
+
+        for key in params:
+            if params[key]:
+                params[key] = params[key][-1]
+
+        self.url, self.params = path, params
+
+        if self.url == '/':
+            if "id" not in params:
+                self.send_response(FOUND)
+                self.send_header("Connection", "close")
+                self.send_header("Location", "/?id=1")
+                self.end_headers()
+            else:
+                self.send_response(OK)
+                self.send_header("Content-type", "text/html")
+                self.send_header("Connection", "close")
+                self.end_headers()
+
+                try:
+                    _cursor.execute("SELECT * FROM users WHERE id=%s LIMIT 0, 1" % self.params.get("id", ""))
+
+                    output = "<b>SQL results:</b>\n"
+                    output += "<table border=\"1\">\n"
+                    for row in _cursor.fetchall():
+                        output += "<tr>"
+                        for value in row:
+                            output += "<td>%s</td>" % value
+                        output += "</tr>\n"
+                    output += "</table>\n"
+                    output += "</body></html>";
+                except Exception as ex:
+                    output = "%s: %s" % (re.search(r"'([^']+)'", str(type(ex))).group(1), ex)
+
+                self.wfile.write(output.encode("utf8"))
+        else:
+            self.send_response(NOT_FOUND)
+            self.send_header("Connection", "close")
+            self.end_headers()
+
+    def do_GET(self):
+        self.do_REQUEST()
+
+    def do_POST(self):
+        length = int(self.headers.get("Content-length", 0))
+        if length:
+            data = self.rfile.read(length)
+            data = unquote_plus(data.decode("utf8"))
+            self.data = data
+        self.do_REQUEST()
+
+def run(address=LISTEN_ADDRESS, port=LISTEN_PORT):
+    global _server
+    try:
+        _server = ThreadingServer((address, port), ReqHandler)
+        print("[i] running HTTP server at '%s:%d'" % (address, port))
+        _server.serve_forever()
+    except KeyboardInterrupt:
+        _server.socket.close()
+        raise
+
+if __name__ == "__main__":
+    try:
+        init()
+        run(sys.argv[1] if len(sys.argv) > 1 else LISTEN_ADDRESS, int(sys.argv[2] if len(sys.argv) > 2 else LISTEN_PORT))
+    except KeyboardInterrupt:
+        print("\r[x] Ctrl-C received")

extra/wafdetectify/__init__.py

@@ -1,8 +0,0 @@
-#!/usr/bin/env python
-
-"""
-Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
-See the file 'LICENSE' for copying permission
-"""
-
-pass

extra/wafdetectify/wafdetectify.py

@@ -1,134 +0,0 @@
-#!/usr/bin/env python
-
-"""
-Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
-See the file 'LICENSE' for copying permission
-"""
-
-from __future__ import print_function
-
-import cookielib
-import glob
-import httplib
-import inspect
-import os
-import re
-import socket
-import ssl
-import subprocess
-import sys
-import urllib2
-
-sys.dont_write_bytecode = True
-
-if hasattr(ssl, "_create_unverified_context"):
-    ssl._create_default_https_context = ssl._create_unverified_context
-
-NAME, VERSION, AUTHOR = "WAF Detectify", "0.1", "sqlmap developers (@sqlmap)"
-TIMEOUT = 10
-HEADERS = {"User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:61.0) Gecko/20100101 Firefox/61.0", "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8", "Accept-Language": "en-US,en;q=0.5", "Accept-Encoding": "identity", "Cache-Control": "max-age=0"}
-SQLMAP_DIR = os.path.abspath(os.path.join(os.path.dirname(__file__), "..", ".."))
-SCRIPTS_DIR = os.path.join(SQLMAP_DIR, "waf")
-LEVEL_COLORS = {"o": "\033[00;94m", "x": "\033[00;91m", "!": "\033[00;93m", "i": "\033[00;92m"}
-CACHE = {}
-WAF_FUNCTIONS = []
-
-def get_page(get=None, url=None, host=None, data=None):
-    key = (get, url, host, data)
-
-    if key in CACHE:
-        return CACHE[key]
-
-    page, headers, code = None, {}, httplib.OK
-
-    url = url or ("%s%s%s" % (sys.argv[1], '?' if '?' not in sys.argv[1] else '&', get) if get else sys.argv[1])
-    if not url.startswith("http"):
-        url = "http://%s" % url
-
-    try:
-        req = urllib2.Request("".join(url[_].replace(' ', "%20") if _ > url.find('?') else url[_] for _ in xrange(len(url))), data, HEADERS)
-        conn = urllib2.urlopen(req, timeout=TIMEOUT)
-        page = conn.read()
-        headers = conn.info()
-    except Exception as ex:
-        code = getattr(ex, "code", None)
-        page = ex.read() if hasattr(ex, "read") else getattr(ex, "msg", "")
-        headers = ex.info() if hasattr(ex, "info") else {}
-
-    result = CACHE[key] = page, headers, code
-
-    return result
-
-def colorize(message):
-    if not subprocess.mswindows and sys.stdout.isatty():
-        message = re.sub(r"\[(.)\]", lambda match: "[%s%s\033[00;49m]" % (LEVEL_COLORS[match.group(1)], match.group(1)), message)
-        message = message.replace("@sqlmap", "\033[00;96m@sqlmap\033[00;49m")
-        message = message.replace(NAME, "\033[00;93m%s\033[00;49m" % NAME)
-
-    return message
-
-def main():
-    global WAF_FUNCTIONS
-
-    print(colorize("%s #v%s\n by: %s\n" % (NAME, VERSION, AUTHOR)))
-
-    if len(sys.argv) < 2:
-        exit(colorize("[x] usage: python %s <hostname>" % os.path.split(__file__)[-1]))
-
-    cookie_jar = cookielib.CookieJar()
-    opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cookie_jar))
-    urllib2.install_opener(opener)
-
-    sys.path.insert(0, SQLMAP_DIR)
-
-    for found in glob.glob(os.path.join(SCRIPTS_DIR, "*.py")):
-        dirname, filename = os.path.split(found)
-        dirname = os.path.abspath(dirname)
-
-        if filename == "__init__.py":
-            continue
-
-        if dirname not in sys.path:
-            sys.path.insert(0, dirname)
-
-        try:
-            if filename[:-3] in sys.modules:
-                del sys.modules[filename[:-3]]
-            module = __import__(filename[:-3].encode(sys.getfilesystemencoding() or "utf8"))
-        except ImportError as ex:
-            exit(colorize("[x] cannot import WAF script '%s' (%s)" % (filename[:-3], ex)))
-
-        _ = dict(inspect.getmembers(module))
-        if "detect" not in _:
-            exit(colorize("[x] missing function 'detect(get_page)' in WAF script '%s'" % found))
-        else:
-            WAF_FUNCTIONS.append((_["detect"], _.get("__product__", filename[:-3])))
-
-    WAF_FUNCTIONS = sorted(WAF_FUNCTIONS, key=lambda _: "generic" in _[1].lower())
-
-    print(colorize("[i] checking '%s'..." % sys.argv[1]))
-
-    hostname = sys.argv[1].split("//")[-1].split('/')[0]
-    try:
-        socket.getaddrinfo(hostname, None)
-    except socket.gaierror:
-        print(colorize("[x] host '%s' does not exist" % hostname))
-        exit(1)
-
-    found = False
-    for function, product in WAF_FUNCTIONS:
-        if found and "unknown" in product.lower():
-            continue
-
-        if function(get_page):
-            exit(colorize("[!] WAF/IPS identified as '%s'" % product))
-
-    if not found:
-        print(colorize("[o] nothing found"))
-
-    print()
-
-    exit(int(not found))
-
-if __name__ == "__main__":
-    main()

lib/__init__.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)

lib/controller/__init__.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)

lib/controller/action.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)

lib/controller/checks.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
@@ -6,7 +6,6 @@ See the file 'LICENSE' for copying permission
 """
 
 import copy
-import httplib
 import logging
 import os
 import random
@@ -22,6 +21,7 @@ from lib.core.agent import agent
 from lib.core.common import Backend
 from lib.core.common import extractRegexResult
 from lib.core.common import extractTextTagContent
+from lib.core.common import filterNone
 from lib.core.common import findDynamicContent
 from lib.core.common import Format
 from lib.core.common import getFilteredPageContent
@@ -48,6 +48,7 @@ from lib.core.common import unArrayizeValue
 from lib.core.common import urlencode
 from lib.core.common import wasLastResponseDBMSError
 from lib.core.common import wasLastResponseHTTPError
+from lib.core.compat import xrange
 from lib.core.convert import unicodeencode
 from lib.core.defaults import defaults
 from lib.core.data import conf
@@ -106,6 +107,8 @@ from lib.request.inject import checkBooleanExpression
 from lib.request.templates import getPageTemplate
 from lib.techniques.union.test import unionTest
 from lib.techniques.union.use import configUnion
+from thirdparty import six
+from thirdparty.six.moves import http_client as _http_client
 
 def checkSqlInjection(place, parameter, value):
     # Store here the details about boundaries and payload used to
@@ -579,7 +582,7 @@ def checkSqlInjection(place, parameter, value):
                                         else:
                                             errorSet = set()
 
-                                        candidates = filter(None, (_.strip() if _.strip() in trueRawResponse and _.strip() not in falseRawResponse else None for _ in (trueSet - falseSet - errorSet)))
+                                        candidates = filterNone(_.strip() if _.strip() in trueRawResponse and _.strip() not in falseRawResponse else None for _ in (trueSet - falseSet - errorSet))
 
                                         if candidates:
                                             candidates = sorted(candidates, key=lambda _: len(_))
@@ -593,7 +596,7 @@ def checkSqlInjection(place, parameter, value):
                                             logger.info(infoMsg)
 
                                         if not any((conf.string, conf.notString)):
-                                            candidates = filter(None, (_.strip() if _.strip() in falseRawResponse and _.strip() not in trueRawResponse else None for _ in (falseSet - trueSet)))
+                                            candidates = filterNone(_.strip() if _.strip() in falseRawResponse and _.strip() not in trueRawResponse else None for _ in (falseSet - trueSet))
 
                                             if candidates:
                                                 candidates = sorted(candidates, key=lambda _: len(_))
@@ -692,7 +695,7 @@ def checkSqlInjection(place, parameter, value):
                             # Test for UNION query SQL injection
                             reqPayload, vector = unionTest(comment, place, parameter, value, prefix, suffix)
 
-                            if isinstance(reqPayload, basestring):
+                            if isinstance(reqPayload, six.string_types):
                                 infoMsg = "%s parameter '%s' is '%s' injectable" % (paramType, parameter, title)
                                 logger.info(infoMsg)
 
@@ -1337,6 +1340,9 @@ def checkWaf():
     if any((conf.string, conf.notString, conf.regexp, conf.dummy, conf.offline, conf.skipWaf)):
         return None
 
+    if kb.originalCode == _http_client.NOT_FOUND:
+        return None
+
     _ = hashDBRetrieve(HASHDB_KEYS.CHECK_WAF_RESULT, True)
     if _ is not None:
         if _:
@@ -1421,17 +1427,24 @@ def identifyWaf():
         page, headers, code = None, None, None
         try:
             pushValue(kb.redirectChoice)
+            pushValue(kb.resendPostOnRedirect)
+
             kb.redirectChoice = REDIRECTION.YES
+            kb.resendPostOnRedirect = True
+
             if kwargs.get("get"):
                 kwargs["get"] = urlencode(kwargs["get"])
             kwargs["raise404"] = False
             kwargs["silent"] = True
             kwargs["finalCode"] = True
+
             page, headers, code = Request.getPage(*args, **kwargs)
         except Exception:
             pass
         finally:
+            kb.resendPostOnRedirect = popValue()
             kb.redirectChoice = popValue()
+
         return page or "", headers or {}, code
 
     retVal = []
@@ -1497,46 +1510,59 @@ def checkNullConnection():
     if conf.data:
         return False
 
-    infoMsg = "testing NULL connection to the target URL"
+    _ = hashDBRetrieve(HASHDB_KEYS.CHECK_NULL_CONNECTION_RESULT, True)
-    logger.info(infoMsg)
+    if _ is not None:
+        kb.nullConnection = _
 
-    pushValue(kb.pageCompress)
+        if _:
-    kb.pageCompress = False
+            dbgMsg = "resuming NULL connection method '%s'" % _
+            logger.debug(dbgMsg)
 
-    try:
+    else:
-        page, headers, _ = Request.getPage(method=HTTPMETHOD.HEAD, raise404=False)
+        infoMsg = "testing NULL connection to the target URL"
+        logger.info(infoMsg)
 
-        if not page and HTTP_HEADER.CONTENT_LENGTH in (headers or {}):
+        pushValue(kb.pageCompress)
-            kb.nullConnection = NULLCONNECTION.HEAD
+        kb.pageCompress = False
 
-            infoMsg = "NULL connection is supported with HEAD method ('Content-Length')"
+        try:
-            logger.info(infoMsg)
+            page, headers, _ = Request.getPage(method=HTTPMETHOD.HEAD, raise404=False)
-        else:
-            page, headers, _ = Request.getPage(auxHeaders={HTTP_HEADER.RANGE: "bytes=-1"})
 
-            if page and len(page) == 1 and HTTP_HEADER.CONTENT_RANGE in (headers or {}):
+            if not page and HTTP_HEADER.CONTENT_LENGTH in (headers or {}):
-                kb.nullConnection = NULLCONNECTION.RANGE
+                kb.nullConnection = NULLCONNECTION.HEAD
 
-                infoMsg = "NULL connection is supported with GET method ('Range')"
+                infoMsg = "NULL connection is supported with HEAD method ('Content-Length')"
                 logger.info(infoMsg)
             else:
-                _, headers, _ = Request.getPage(skipRead=True)
+                page, headers, _ = Request.getPage(auxHeaders={HTTP_HEADER.RANGE: "bytes=-1"})
 
-                if HTTP_HEADER.CONTENT_LENGTH in (headers or {}):
+                if page and len(page) == 1 and HTTP_HEADER.CONTENT_RANGE in (headers or {}):
-                    kb.nullConnection = NULLCONNECTION.SKIP_READ
+                    kb.nullConnection = NULLCONNECTION.RANGE
 
-                    infoMsg = "NULL connection is supported with 'skip-read' method"
+                    infoMsg = "NULL connection is supported with GET method ('Range')"
                     logger.info(infoMsg)
+                else:
+                    _, headers, _ = Request.getPage(skipRead=True)
 
-    except SqlmapConnectionException:
+                    if HTTP_HEADER.CONTENT_LENGTH in (headers or {}):
-        pass
+                        kb.nullConnection = NULLCONNECTION.SKIP_READ
 
-    finally:
+                        infoMsg = "NULL connection is supported with 'skip-read' method"
-        kb.pageCompress = popValue()
+                        logger.info(infoMsg)
 
-    return kb.nullConnection is not None
+        except SqlmapConnectionException:
+            pass
+
+        finally:
+            kb.pageCompress = popValue()
+            kb.nullConnection = False if kb.nullConnection is None else kb.nullConnection
+            hashDBWrite(HASHDB_KEYS.CHECK_NULL_CONNECTION_RESULT, kb.nullConnection, True)
+
+    return kb.nullConnection in getPublicTypeMembers(NULLCONNECTION, True)
 
 def checkConnection(suppressOutput=False):
+    threadData = getCurrentThreadData()
+
     if not re.search(r"\A\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\Z", conf.hostname):
         if not any((conf.proxy, conf.tor, conf.dummy, conf.offline)):
             try:
@@ -1561,8 +1587,7 @@ def checkConnection(suppressOutput=False):
 
     try:
         kb.originalPageTime = time.time()
-        page, headers, _ = Request.queryPage(content=True, noteResponseTime=False)
+        Request.queryPage(content=True, noteResponseTime=False)
-        kb.originalPage = kb.pageTemplate = page
 
         kb.errorIsNone = False
 
@@ -1585,8 +1610,6 @@ def checkConnection(suppressOutput=False):
         else:
             kb.errorIsNone = True
 
-        threadData = getCurrentThreadData()
-
         if kb.redirectChoice == REDIRECTION.YES and threadData.lastRedirectURL and threadData.lastRedirectURL[0] == threadData.lastRequestUID:
             if (threadData.lastRedirectURL[1] or "").startswith("https://") and unicodeencode(conf.hostname) in threadData.lastRedirectURL[1]:
                 conf.url = re.sub(r"https?://", "https://", conf.url)
@@ -1603,7 +1626,7 @@ def checkConnection(suppressOutput=False):
             warnMsg += "any addressing issues"
             singleTimeWarnMessage(warnMsg)
 
-        if any(code in kb.httpErrorCodes for code in (httplib.NOT_FOUND, )):
+        if any(code in kb.httpErrorCodes for code in (_http_client.NOT_FOUND, )):
             errMsg = getSafeExString(ex)
             logger.critical(errMsg)
 
@@ -1617,6 +1640,9 @@ def checkConnection(suppressOutput=False):
                 kb.ignoreNotFound = True
         else:
             raise
+    finally:
+        kb.originalPage = kb.pageTemplate = threadData.lastPage
+        kb.originalCode = threadData.lastCode
 
     return True
 

lib/controller/controller.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
@@ -40,6 +40,7 @@ from lib.core.common import safeCSValue
 from lib.core.common import showHttpErrorCodes
 from lib.core.common import urlencode
 from lib.core.common import urldecode
+from lib.core.compat import xrange
 from lib.core.data import conf
 from lib.core.data import kb
 from lib.core.data import logger

lib/controller/handler.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)

lib/core/__init__.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)

lib/core/agent.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
@@ -9,6 +9,7 @@ import re
 
 from lib.core.common import Backend
 from lib.core.common import extractRegexResult
+from lib.core.common import filterNone
 from lib.core.common import getSQLSnippet
 from lib.core.common import getUnicode
 from lib.core.common import isDBMSVersionAtLeast
@@ -23,6 +24,7 @@ from lib.core.common import splitFields
 from lib.core.common import unArrayizeValue
 from lib.core.common import urlencode
 from lib.core.common import zeroDepthSearch
+from lib.core.compat import xrange
 from lib.core.data import conf
 from lib.core.data import kb
 from lib.core.data import queries
@@ -105,7 +107,7 @@ class Agent(object):
             if place == PLACE.URI:
                 origValue = origValue.split(kb.customInjectionMark)[0]
             else:
-                origValue = filter(None, (re.search(_, origValue.split(BOUNDED_INJECTION_MARKER)[0]) for _ in (r"\w+\Z", r"[^\"'><]+\Z", r"[^ ]+\Z")))[0].group(0)
+                origValue = filterNone(re.search(_, origValue.split(BOUNDED_INJECTION_MARKER)[0]) for _ in (r"\w+\Z", r"[^\"'><]+\Z", r"[^ ]+\Z"))[0].group(0)
             origValue = origValue[origValue.rfind('/') + 1:]
             for char in ('?', '=', ':', ',', '&'):
                 if char in origValue:

lib/core/bigarray.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
@@ -16,6 +16,7 @@ import os
 import sys
 import tempfile
 
+from lib.core.compat import xrange
 from lib.core.enums import MKSTEMP_PREFIX
 from lib.core.exception import SqlmapSystemException
 from lib.core.settings import BIGARRAY_CHUNK_SIZE

lib/core/common.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
@@ -7,15 +7,16 @@ See the file 'LICENSE' for copying permission
 
 import binascii
 import codecs
+import collections
 import contextlib
-import cookielib
 import copy
 import distutils
 import getpass
 import hashlib
-import httplib
 import inspect
+import io
 import json
+import keyword
 import locale
 import logging
 import ntpath
@@ -32,14 +33,8 @@ import tempfile
 import threading
 import time
 import types
-import urllib
-import urllib2
-import urlparse
 import unicodedata
 
-from ConfigParser import DEFAULTSECT
-from ConfigParser import RawConfigParser
-from StringIO import StringIO
 from difflib import SequenceMatcher
 from math import sqrt
 from optparse import OptionValueError
@@ -51,10 +46,7 @@ from extra.beep.beep import beep
 from extra.cloak.cloak import decloak
 from extra.safe2bin.safe2bin import safecharencode
 from lib.core.bigarray import BigArray
-from lib.core.data import conf
+from lib.core.compat import xrange
-from lib.core.data import kb
-from lib.core.data import logger
-from lib.core.data import paths
 from lib.core.convert import base64pickle
 from lib.core.convert import base64unpickle
 from lib.core.convert import hexdecode
@@ -62,6 +54,10 @@ from lib.core.convert import htmlunescape
 from lib.core.convert import stdoutencode
 from lib.core.convert import unicodeencode
 from lib.core.convert import utf8encode
+from lib.core.data import conf
+from lib.core.data import kb
+from lib.core.data import logger
+from lib.core.data import paths
 from lib.core.decorators import cachedmethod
 from lib.core.defaults import defaults
 from lib.core.dicts import DBMS_DICT
@@ -115,6 +111,7 @@ from lib.core.settings import DEV_EMAIL_ADDRESS
 from lib.core.settings import DUMMY_USER_INJECTION
 from lib.core.settings import DYNAMICITY_BOUNDARY_LENGTH
 from lib.core.settings import ERROR_PARSING_REGEXES
+from lib.core.settings import EVALCODE_ENCODED_PREFIX
 from lib.core.settings import FILE_PATH_REGEXES
 from lib.core.settings import FORCE_COOKIE_EXPIRATION_TIME
 from lib.core.settings import FORM_SEARCH_REGEX
@@ -124,6 +121,7 @@ from lib.core.settings import GITHUB_REPORT_OAUTH_TOKEN
 from lib.core.settings import GOOGLE_ANALYTICS_COOKIE_PREFIX
 from lib.core.settings import HASHDB_MILESTONE_VALUE
 from lib.core.settings import HOST_ALIASES
+from lib.core.settings import HTTP_CHUNKED_SPLIT_KEYWORDS
 from lib.core.settings import IGNORE_SAVE_OPTIONS
 from lib.core.settings import INFERENCE_UNKNOWN_CHAR
 from lib.core.settings import INVALID_UNICODE_CHAR_FORMAT
@@ -155,9 +153,9 @@ from lib.core.settings import REFLECTED_REPLACEMENT_REGEX
 from lib.core.settings import REFLECTED_REPLACEMENT_TIMEOUT
 from lib.core.settings import REFLECTED_VALUE_MARKER
 from lib.core.settings import REFLECTIVE_MISS_THRESHOLD
-from lib.core.settings import SAFE_VARIABLE_MARKER
 from lib.core.settings import SENSITIVE_DATA_REGEX
 from lib.core.settings import SENSITIVE_OPTIONS
+from lib.core.settings import STDIN_PIPE_DASH
 from lib.core.settings import SUPPORTED_DBMS
 from lib.core.settings import TEXT_TAG_REGEX
 from lib.core.settings import TIME_STDEV_COEFF
@@ -171,14 +169,18 @@ from lib.core.settings import VERSION_STRING
 from lib.core.settings import WEBSCARAB_SPLITTER
 from lib.core.threads import getCurrentThreadData
 from lib.utils.sqlalchemy import _sqlalchemy
+from thirdparty import six
 from thirdparty.clientform.clientform import ParseResponse
 from thirdparty.clientform.clientform import ParseError
 from thirdparty.colorama.initialise import init as coloramainit
 from thirdparty.magic import magic
-from thirdparty.odict.odict import OrderedDict
+from thirdparty.odict import OrderedDict
+from thirdparty.six.moves import configparser as _configparser
+from thirdparty.six.moves import http_client as _http_client
+from thirdparty.six.moves import urllib as _urllib
 from thirdparty.termcolor.termcolor import colored
 
-class UnicodeRawConfigParser(RawConfigParser):
+class UnicodeRawConfigParser(_configparser.RawConfigParser):
     """
     RawConfigParser with unicode writing support
     """
@@ -189,7 +191,7 @@ class UnicodeRawConfigParser(RawConfigParser):
         """
 
         if self._defaults:
-            fp.write("[%s]\n" % DEFAULTSECT)
+            fp.write("[%s]\n" % _configparser.DEFAULTSECT)
 
             for (key, value) in self._defaults.items():
                 fp.write("%s = %s\n" % (key, getUnicode(value, UNICODE_ENCODING).replace('\n', '\n\t')))
@@ -227,7 +229,7 @@ class Format(object):
         if versions is None and Backend.getVersionList():
             versions = Backend.getVersionList()
 
-        return Backend.getDbms() if versions is None else "%s %s" % (Backend.getDbms(), " and ".join(filter(None, versions)))
+        return Backend.getDbms() if versions is None else "%s %s" % (Backend.getDbms(), " and ".join(filterNone(versions)))
 
     @staticmethod
     def getErrorParsedDBMSes():
@@ -356,7 +358,7 @@ class Backend:
 
     @staticmethod
     def setVersion(version):
-        if isinstance(version, basestring):
+        if isinstance(version, six.string_types):
             kb.dbmsVersion = [version]
 
         return kb.dbmsVersion
@@ -365,7 +367,7 @@ class Backend:
     def setVersionList(versionsList):
         if isinstance(versionsList, list):
             kb.dbmsVersion = versionsList
-        elif isinstance(versionsList, basestring):
+        elif isinstance(versionsList, six.string_types):
             Backend.setVersion(versionsList)
         else:
             logger.error("invalid format of versionsList")
@@ -388,7 +390,7 @@ class Backend:
             return None
 
         # Little precaution, in theory this condition should always be false
-        elif kb.os is not None and isinstance(os, basestring) and kb.os.lower() != os.lower():
+        elif kb.os is not None and isinstance(os, six.string_types) and kb.os.lower() != os.lower():
             msg = "sqlmap previously fingerprinted back-end DBMS "
             msg += "operating system %s. However now it has " % kb.os
             msg += "been fingerprinted to be %s. " % os
@@ -407,7 +409,7 @@ class Backend:
                     warnMsg = "invalid value"
                     logger.warn(warnMsg)
 
-        elif kb.os is None and isinstance(os, basestring):
+        elif kb.os is None and isinstance(os, six.string_types):
             kb.os = os.capitalize()
 
         return kb.os
@@ -417,7 +419,7 @@ class Backend:
         if version is None:
             return None
 
-        elif kb.osVersion is None and isinstance(version, basestring):
+        elif kb.osVersion is None and isinstance(version, six.string_types):
             kb.osVersion = version
 
     @staticmethod
@@ -437,7 +439,7 @@ class Backend:
         while True:
             choice = readInput(msg, default='1')
 
-            if isinstance(choice, basestring) and choice.isdigit() and int(choice) in (1, 2):
+            if hasattr(choice, "isdigit") and choice.isdigit() and int(choice) in (1, 2):
                 kb.arch = 32 if int(choice) == 1 else 64
                 break
             else:
@@ -500,7 +502,7 @@ class Backend:
 
     @staticmethod
     def getVersion():
-        versions = filter(None, flattenValue(kb.dbmsVersion))
+        versions = filterNone(flattenValue(kb.dbmsVersion))
         if not isNoneValue(versions):
             return versions[0]
         else:
@@ -508,7 +510,7 @@ class Backend:
 
     @staticmethod
     def getVersionList():
-        versions = filter(None, flattenValue(kb.dbmsVersion))
+        versions = filterNone(flattenValue(kb.dbmsVersion))
         if not isNoneValue(versions):
             return versions
         else:
@@ -641,7 +643,7 @@ def paramToDict(place, parameters=None):
                                             if isinstance(value, (list, tuple, set, dict)):
                                                 if value:
                                                     walk(head, value)
-                                            elif isinstance(value, (bool, int, float, basestring)):
+                                            elif isinstance(value, (bool, int, float, six.string_types)):
                                                 original = current[key]
                                                 if isinstance(value, bool):
                                                     current[key] = "%s%s" % (getUnicode(value).lower(), BOUNDED_INJECTION_MARKER)
@@ -786,7 +788,7 @@ def getManualDirectories():
             else:
                 targets.add('.'.join(_[:-1]))
 
-            targets = filter(None, targets)
+            targets = filterNone(targets)
 
             for prefix in BRUTE_DOC_ROOT_PREFIXES.get(Backend.getOs(), DEFAULT_DOC_ROOTS[OS.LINUX]):
                 if BRUTE_DOC_ROOT_TARGET_MARK in prefix and re.match(IP_ADDRESS_REGEX, conf.hostname):
@@ -885,7 +887,7 @@ def setColor(message, color=None, bold=False, level=None):
         if bold or color:
             retVal = colored(message, color=color, on_color=None, attrs=("bold",) if bold else None)
         elif level:
-            level = getattr(logging, level, None) if isinstance(level, basestring) else level
+            level = getattr(logging, level, None) if isinstance(level, six.string_types) else level
             retVal = LOGGER_HANDLER.colorize(message, level)
 
     return retVal
@@ -936,7 +938,7 @@ def dataToStdout(data, forceOutput=False, bold=False, content_type=None, status=
             if multiThreadMode:
                 logging._releaseLock()
 
-            kb.prependFlag = isinstance(data, basestring) and (len(data) == 1 and data not in ('\n', '\r') or len(data) > 2 and data[0] == '\r' and data[-1] != '\n')
+            kb.prependFlag = isinstance(data, six.string_types) and (len(data) == 1 and data not in ('\n', '\r') or len(data) > 2 and data[0] == '\r' and data[-1] != '\n')
 
 def dataToTrafficFile(data):
     if not conf.trafficFile:
@@ -1069,7 +1071,7 @@ def readInput(message, default=None, checkBatch=True, boolean=False):
             finally:
                 logging._releaseLock()
 
-    if retVal and default and isinstance(default, basestring) and len(default) == 1:
+    if retVal and default and isinstance(default, six.string_types) and len(default) == 1:
         retVal = retVal.strip()
 
     if boolean:
@@ -1163,6 +1165,14 @@ def getHeader(headers, key):
             break
     return retVal
 
+def checkPipedInput():
+    """
+    Checks whether input to program has been provided via standard input (e.g. cat /tmp/req.txt | python sqlmap.py -r -)
+    # Reference: https://stackoverflow.com/a/33873570
+    """
+
+    return not os.isatty(sys.stdin.fileno())
+
 def checkFile(filename, raiseOnError=True):
     """
     Checks for file existence and readability
@@ -1176,19 +1186,22 @@ def checkFile(filename, raiseOnError=True):
     if filename:
         filename = filename.strip('"\'')
 
-    try:
+    if filename == STDIN_PIPE_DASH:
-        if filename is None or not os.path.isfile(filename):
+        return checkPipedInput()
-            valid = False
+    else:
-    except:
-        valid = False
-
-    if valid:
         try:
-            with open(filename, "rb"):
+            if filename is None or not os.path.isfile(filename):
-                pass
+                valid = False
         except:
             valid = False
 
+        if valid:
+            try:
+                with open(filename, "rb"):
+                    pass
+            except:
+                valid = False
+
     if not valid and raiseOnError:
         raise SqlmapSystemException("unable to read file '%s'" % filename)
 
@@ -1262,6 +1275,7 @@ def setPaths(rootPath):
     paths.SQLMAP_EXTRAS_PATH = os.path.join(paths.SQLMAP_ROOT_PATH, "extra")
     paths.SQLMAP_PROCS_PATH = os.path.join(paths.SQLMAP_ROOT_PATH, "procs")
     paths.SQLMAP_SHELL_PATH = os.path.join(paths.SQLMAP_ROOT_PATH, "shell")
+    paths.SQLMAP_SETTINGS_PATH = os.path.join(paths.SQLMAP_ROOT_PATH, "lib", "core", "settings.py")
     paths.SQLMAP_TAMPER_PATH = os.path.join(paths.SQLMAP_ROOT_PATH, "tamper")
     paths.SQLMAP_WAF_PATH = os.path.join(paths.SQLMAP_ROOT_PATH, "waf")
     paths.SQLMAP_TXT_PATH = os.path.join(paths.SQLMAP_ROOT_PATH, "txt")
@@ -1293,7 +1307,6 @@ def setPaths(rootPath):
     paths.GITHUB_HISTORY = os.path.join(paths.SQLMAP_HISTORY_PATH, "github.hst")
 
     # sqlmap files
-    paths.CHECKSUM_MD5 = os.path.join(paths.SQLMAP_TXT_PATH, "checksum.md5")
     paths.COMMON_COLUMNS = os.path.join(paths.SQLMAP_TXT_PATH, "common-columns.txt")
     paths.COMMON_TABLES = os.path.join(paths.SQLMAP_TXT_PATH, "common-tables.txt")
     paths.COMMON_OUTPUTS = os.path.join(paths.SQLMAP_TXT_PATH, 'common-outputs.txt')
@@ -1312,7 +1325,7 @@ def setPaths(rootPath):
     paths.PGSQL_XML = os.path.join(paths.SQLMAP_XML_BANNER_PATH, "postgresql.xml")
 
     for path in paths.values():
-        if any(path.endswith(_) for _ in (".md5", ".txt", ".xml", ".zip")):
+        if any(path.endswith(_) for _ in (".txt", ".xml", ".zip")):
             checkFile(path)
 
 def weAreFrozen():
@@ -1454,14 +1467,14 @@ def parseTargetUrl():
         conf.url = conf.url.replace('?', URI_QUESTION_MARKER)
 
     try:
-        urlSplit = urlparse.urlsplit(conf.url)
+        urlSplit = _urllib.parse.urlsplit(conf.url)
     except ValueError as ex:
         errMsg = "invalid URL '%s' has been given ('%s'). " % (conf.url, getSafeExString(ex))
         errMsg += "Please be sure that you don't have any leftover characters (e.g. '[' or ']') "
         errMsg += "in the hostname part"
         raise SqlmapGenericException(errMsg)
 
-    hostnamePort = urlSplit.netloc.split(":") if not re.search(r"\[.+\]", urlSplit.netloc) else filter(None, (re.search(r"\[.+\]", urlSplit.netloc).group(0), re.search(r"\](:(?P<port>\d+))?", urlSplit.netloc).group("port")))
+    hostnamePort = urlSplit.netloc.split(":") if not re.search(r"\[.+\]", urlSplit.netloc) else filterNone((re.search(r"\[.+\]", urlSplit.netloc).group(0), re.search(r"\](:(?P<port>\d+))?", urlSplit.netloc).group("port")))
 
     conf.scheme = (urlSplit.scheme.strip().lower() or "http") if not conf.forceSSL else "https"
     conf.path = urlSplit.path.strip()
@@ -1653,7 +1666,7 @@ def parseUnionPage(page):
     else:
         data = page
 
-    if len(data) == 1 and isinstance(data[0], basestring):
+    if len(data) == 1 and isinstance(data[0], six.string_types):
         data = data[0]
 
     return data
@@ -1807,6 +1820,19 @@ def normalizePath(filepath):
 
     return retVal
 
+def safeFilepathEncode(filepath):
+    """
+    Returns filepath in (ASCII) format acceptable for OS handling (e.g. reading)
+    """
+
+    retVal = filepath
+
+    if filepath and isinstance(filepath, unicode):
+        retVal = filepath.encode(sys.getfilesystemencoding() or UNICODE_ENCODING)
+
+    return retVal
+
+
 def safeExpandUser(filepath):
     """
     Patch for a Python Issue18171 (http://bugs.python.org/issue18171)
@@ -1838,7 +1864,7 @@ def safeStringFormat(format_, params):
     else:
         retVal = re.sub(r"(\A|[^A-Za-z0-9])(%d)([^A-Za-z0-9]|\Z)", r"\g<1>%s\g<3>", format_)
 
-    if isinstance(params, basestring):
+    if isinstance(params, six.string_types):
         retVal = retVal.replace("%s", params, 1)
     elif not isListLike(params):
         retVal = retVal.replace("%s", getUnicode(params), 1)
@@ -2051,7 +2077,7 @@ def parseXmlFile(xmlFile, handler):
     """
 
     try:
-        with contextlib.closing(StringIO(readCachedFileContent(xmlFile))) as stream:
+        with contextlib.closing(io.StringIO(readCachedFileContent(xmlFile))) as stream:
             parse(stream, handler)
     except (SAXParseException, UnicodeError) as ex:
         errMsg = "something appears to be wrong with "
@@ -2360,15 +2386,30 @@ def getUnicode(value, encoding=None, noneToNull=False):
     if noneToNull and value is None:
         return NULL
 
-    if isinstance(value, unicode):
+    if isinstance(value, six.text_type):
         return value
-    elif isinstance(value, basestring):
+    elif isinstance(value, six.binary_type):
+        # Heuristics (if encoding not explicitly specified)
+        candidates = filterNone((encoding, kb.get("pageEncoding") if kb.get("originalPage") else None, conf.get("encoding"), UNICODE_ENCODING, sys.getfilesystemencoding()))
+        if all(_ in value for _ in ('<', '>')):
+            pass
+        elif any(_ in value for _ in (":\\", '/', '.')) and '\n' not in value:
+            candidates = filterNone((encoding, sys.getfilesystemencoding(), kb.get("pageEncoding") if kb.get("originalPage") else None, UNICODE_ENCODING, conf.get("encoding")))
+        elif conf.get("encoding") and '\n' not in value:
+            candidates = filterNone((encoding, conf.get("encoding"), kb.get("pageEncoding") if kb.get("originalPage") else None, sys.getfilesystemencoding(), UNICODE_ENCODING))
+
+        for candidate in candidates:
+            try:
+                return six.text_type(value, candidate)
+            except UnicodeDecodeError:
+                pass
+
         while True:
             try:
-                return unicode(value, encoding or (kb.get("pageEncoding") if kb.get("originalPage") else None) or UNICODE_ENCODING)
+                return six.text_type(value, encoding or (kb.get("pageEncoding") if kb.get("originalPage") else None) or UNICODE_ENCODING)
             except UnicodeDecodeError as ex:
                 try:
-                    return unicode(value, UNICODE_ENCODING)
+                    return six.text_type(value, UNICODE_ENCODING)
                 except:
                     value = value[:ex.start] + "".join(INVALID_UNICODE_CHAR_FORMAT % ord(_) for _ in value[ex.start:ex.end]) + value[ex.end:]
     elif isListLike(value):
@@ -2376,9 +2417,9 @@ def getUnicode(value, encoding=None, noneToNull=False):
         return value
     else:
         try:
-            return unicode(value)
+            return six.text_type(value)
         except UnicodeDecodeError:
-            return unicode(str(value), errors="ignore")  # encoding ignored for non-basestring instances
+            return six.text_type(str(value), errors="ignore")  # encoding ignored for non-basestring instances
 
 def longestCommonPrefix(*sequences):
     """
@@ -2530,9 +2571,9 @@ def extractErrorMessage(page):
 
     retVal = None
 
-    if isinstance(page, basestring):
+    if isinstance(page, six.string_types):
         for regex in ERROR_PARSING_REGEXES:
-            match = re.search(regex, page, re.DOTALL | re.IGNORECASE)
+            match = re.search(regex, page, re.IGNORECASE)
 
             if match:
                 retVal = htmlunescape(match.group("result")).replace("<br>", "\n").strip()
@@ -2612,7 +2653,7 @@ def urldecode(value, encoding=None, unsafe="%%&=;+%s" % CUSTOM_INJECTION_MARK_CH
             pass
         finally:
             if convall:
-                result = urllib.unquote_plus(value) if spaceplus else urllib.unquote(value)
+                result = _urllib.parse.unquote_plus(value) if spaceplus else _urllib.parse.unquote(value)
             else:
                 def _(match):
                     charset = reduce(lambda x, y: x.replace(y, ""), unsafe, string.printable)
@@ -2620,7 +2661,7 @@ def urldecode(value, encoding=None, unsafe="%%&=;+%s" % CUSTOM_INJECTION_MARK_CH
                     return char if char in charset else match.group(0)
                 result = value
                 if spaceplus:
-                    result = result.replace('+', ' ')  # plus sign has a special meaning in URL encoded data (hence the usage of urllib.unquote_plus in convall case)
+                    result = result.replace('+', ' ')  # plus sign has a special meaning in URL encoded data (hence the usage of _urllib.parse.unquote_plus in convall case)
                 result = re.sub(r"%([0-9a-fA-F]{2})", _, result)
 
     if isinstance(result, str):
@@ -2659,7 +2700,7 @@ def urlencode(value, safe="%&=-_", convall=False, limit=False, spaceplus=False):
             value = re.sub(r"%(?![0-9a-fA-F]{2})", "%25", value)
 
         while True:
-            result = urllib.quote(utf8encode(value), safe)
+            result = _urllib.parse.quote(utf8encode(value), safe)
 
             if limit and len(result) > URLENCODE_CHAR_LIMIT:
                 if count >= len(URLENCODE_FAILSAFE_CHARS):
@@ -2674,7 +2715,7 @@ def urlencode(value, safe="%&=-_", convall=False, limit=False, spaceplus=False):
                 break
 
         if spaceplus:
-            result = result.replace(urllib.quote(' '), '+')
+            result = result.replace(_urllib.parse.quote(' '), '+')
 
     return result
 
@@ -2797,7 +2838,7 @@ def extractTextTagContent(page):
         except MemoryError:
             page = page.replace(REFLECTED_VALUE_MARKER, "")
 
-    return filter(None, (_.group("result").strip() for _ in re.finditer(TEXT_TAG_REGEX, page)))
+    return filterNone(_.group("result").strip() for _ in re.finditer(TEXT_TAG_REGEX, page))
 
 def trimAlphaNum(value):
     """
@@ -2829,7 +2870,7 @@ def isNumPosStrValue(value):
     False
     """
 
-    return (value and isinstance(value, basestring) and value.isdigit() and int(value) > 0) or (isinstance(value, int) and value > 0)
+    return (hasattr(value, "isdigit") and value.isdigit() and int(value) > 0) or (isinstance(value, int) and value > 0)
 
 @cachedmethod
 def aliasToDbmsEnum(dbms):
@@ -2956,6 +2997,21 @@ def filterControlChars(value, replacement=' '):
 
     return filterStringValue(value, PRINTABLE_CHAR_REGEX, replacement)
 
+def filterNone(values):
+    """
+    Emulates filterNone([...]) functionality
+
+    >>> filterNone([1, 2, "", None, 3])
+    [1, 2, 3]
+    """
+
+    retVal = values
+
+    if isinstance(values, collections.Iterable):
+        retVal = [_ for _ in values if _]
+
+    return retVal
+
 def isDBMSVersionAtLeast(version):
     """
     Checks if the recognized DBMS version is at least the version
@@ -2977,7 +3033,7 @@ def isDBMSVersionAtLeast(version):
 
         value = filterStringValue(value, '[0-9.><=]')
 
-        if value and isinstance(value, basestring):
+        if value and isinstance(value, six.string_types):
             if value.startswith(">="):
                 value = float(value.replace(">=", ""))
             elif value.startswith(">"):
@@ -3002,7 +3058,7 @@ def parseSqliteTableSchema(value):
         table = {}
         columns = {}
 
-        for match in re.finditer(r"(\w+)[\"'`]?\s+(INT|INTEGER|TINYINT|SMALLINT|MEDIUMINT|BIGINT|UNSIGNED BIG INT|INT2|INT8|INTEGER|CHARACTER|VARCHAR|VARYING CHARACTER|NCHAR|NATIVE CHARACTER|NVARCHAR|TEXT|CLOB|LONGTEXT|BLOB|NONE|REAL|DOUBLE|DOUBLE PRECISION|FLOAT|REAL|NUMERIC|DECIMAL|BOOLEAN|DATE|DATETIME|NUMERIC)\b", value, re.I):
+        for match in re.finditer(r"(\w+)[\"'`]?\s+(INT|INTEGER|TINYINT|SMALLINT|MEDIUMINT|BIGINT|UNSIGNED BIG INT|INT2|INT8|INTEGER|CHARACTER|VARCHAR|VARYING CHARACTER|NCHAR|NATIVE CHARACTER|NVARCHAR|TEXT|CLOB|LONGTEXT|BLOB|NONE|REAL|DOUBLE|DOUBLE PRECISION|FLOAT|REAL|NUMERIC|DECIMAL|BOOLEAN|DATE|DATETIME|NUMERIC)\b", decodeStringEscape(value), re.I):
             retVal = True
             columns[match.group(1)] = match.group(2)
 
@@ -3107,7 +3163,7 @@ def saveConfig(conf, filename):
                 elif datatype == OPTION_TYPE.STRING:
                     value = ""
 
-            if isinstance(value, basestring):
+            if isinstance(value, six.string_types):
                 value = value.replace("\n", "\n ")
 
             config.set(family, option, value)
@@ -3138,7 +3194,7 @@ def initTechnique(technique=None):
             for key, value in kb.injection.conf.items():
                 if value and (not hasattr(conf, key) or (hasattr(conf, key) and not getattr(conf, key))):
                     setattr(conf, key, value)
-                    debugMsg = "resuming configuration option '%s' (%s)" % (key, value)
+                    debugMsg = "resuming configuration option '%s' (%s)" % (key, ("'%s'" % value) if isinstance(value, six.string_types) else value)
                     logger.debug(debugMsg)
 
                     if value and key == "optimize":
@@ -3263,9 +3319,9 @@ def showHttpErrorCodes():
 
     if kb.httpErrorCodes:
         warnMsg = "HTTP error codes detected during run:\n"
-        warnMsg += ", ".join("%d (%s) - %d times" % (code, httplib.responses[code] if code in httplib.responses else '?', count) for code, count in kb.httpErrorCodes.items())
+        warnMsg += ", ".join("%d (%s) - %d times" % (code, _http_client.responses[code] if code in _http_client.responses else '?', count) for code, count in kb.httpErrorCodes.items())
         logger.warn(warnMsg)
-        if any((str(_).startswith('4') or str(_).startswith('5')) and _ != httplib.INTERNAL_SERVER_ERROR and _ != kb.originalCode for _ in kb.httpErrorCodes.keys()):
+        if any((str(_).startswith('4') or str(_).startswith('5')) and _ != _http_client.INTERNAL_SERVER_ERROR and _ != kb.originalCode for _ in kb.httpErrorCodes.keys()):
             msg = "too many 4xx and/or 5xx HTTP error codes "
             msg += "could mean that some kind of protection is involved (e.g. WAF)"
             logger.debug(msg)
@@ -3275,13 +3331,19 @@ def openFile(filename, mode='r', encoding=UNICODE_ENCODING, errors="replace", bu
     Returns file handle of a given filename
     """
 
-    try:
+    if filename == STDIN_PIPE_DASH:
-        return codecs.open(filename, mode, encoding, errors, buffering)
+        if filename not in kb.cache.content:
-    except IOError:
+            kb.cache.content[filename] = sys.stdin.read()
-        errMsg = "there has been a file opening error for filename '%s'. " % filename
+
-        errMsg += "Please check %s permissions on a file " % ("write" if mode and ('w' in mode or 'a' in mode or '+' in mode) else "read")
+        return contextlib.closing(io.StringIO(readCachedFileContent(filename)))
-        errMsg += "and that it's not locked by another process."
+    else:
-        raise SqlmapSystemException(errMsg)
+        try:
+            return codecs.open(filename, mode, encoding, errors, buffering)
+        except IOError:
+            errMsg = "there has been a file opening error for filename '%s'. " % filename
+            errMsg += "Please check %s permissions on a file " % ("write" if mode and ('w' in mode or 'a' in mode or '+' in mode) else "read")
+            errMsg += "and that it's not locked by another process."
+            raise SqlmapSystemException(errMsg)
 
 def decodeIntToUnicode(value):
     """
@@ -3345,15 +3407,14 @@ def checkIntegrity():
 
     retVal = True
 
-    if os.path.isfile(paths.CHECKSUM_MD5):
+    baseTime = os.path.getmtime(paths.SQLMAP_SETTINGS_PATH)
-        for checksum, _ in (re.split(r'\s+', _) for _ in getFileItems(paths.CHECKSUM_MD5)):
+    for root, dirnames, filenames in os.walk(paths.SQLMAP_ROOT_PATH):
-            path = os.path.normpath(os.path.join(paths.SQLMAP_ROOT_PATH, _))
+        for filename in filenames:
-            if not os.path.isfile(path):
+            if re.search(r"(\.py|\.xml|_)\Z", filename):
-                logger.error("missing file detected '%s'" % path)
+                filepath = os.path.join(root, filename)
-                retVal = False
+                if os.path.getmtime(filepath) > baseTime:
-            elif md5File(path) != checksum:
+                    logger.error("wrong modification time of '%s'" % filepath)
-                logger.error("wrong checksum of file '%s' detected" % path)
+                    retVal = False
-                retVal = False
 
     return retVal
 
@@ -3396,10 +3457,10 @@ def getLatestRevision():
     """
 
     retVal = None
-    req = urllib2.Request(url="https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/lib/core/settings.py")
+    req = _urllib.request.Request(url="https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/lib/core/settings.py")
 
     try:
-        content = urllib2.urlopen(req).read()
+        content = _urllib.request.urlopen(req).read()
         retVal = extractRegexResult(r"VERSION\s*=\s*[\"'](?P<result>[\d.]+)", content)
     except:
         pass
@@ -3439,10 +3500,10 @@ def createGithubIssue(errMsg, excMsg):
         ex = None
         errMsg = errMsg[errMsg.find("\n"):]
 
-        req = urllib2.Request(url="https://api.github.com/search/issues?q=%s" % urllib.quote("repo:sqlmapproject/sqlmap Unhandled exception (#%s)" % key))
+        req = _urllib.request.Request(url="https://api.github.com/search/issues?q=%s" % _urllib.parse.quote("repo:sqlmapproject/sqlmap Unhandled exception (#%s)" % key))
 
         try:
-            content = urllib2.urlopen(req).read()
+            content = _urllib.request.urlopen(req).read()
             _ = json.loads(content)
             duplicate = _["total_count"] > 0
             closed = duplicate and _["items"][0]["state"] == "closed"
@@ -3457,10 +3518,10 @@ def createGithubIssue(errMsg, excMsg):
             pass
 
         data = {"title": "Unhandled exception (#%s)" % key, "body": "```%s\n```\n```\n%s```" % (errMsg, excMsg)}
-        req = urllib2.Request(url="https://api.github.com/repos/sqlmapproject/sqlmap/issues", data=json.dumps(data), headers={"Authorization": "token %s" % GITHUB_REPORT_OAUTH_TOKEN.decode("base64")})
+        req = _urllib.request.Request(url="https://api.github.com/repos/sqlmapproject/sqlmap/issues", data=json.dumps(data), headers={"Authorization": "token %s" % GITHUB_REPORT_OAUTH_TOKEN.decode("base64")})
 
         try:
-            content = urllib2.urlopen(req).read()
+            content = _urllib.request.urlopen(req).read()
         except Exception as ex:
             content = None
 
@@ -3492,14 +3553,14 @@ def maskSensitiveData(msg):
 
     retVal = getUnicode(msg)
 
-    for item in filter(None, (conf.get(_) for _ in SENSITIVE_OPTIONS)):
+    for item in filterNone(conf.get(_) for _ in SENSITIVE_OPTIONS):
         regex = SENSITIVE_DATA_REGEX % re.sub(r"(\W)", r"\\\1", getUnicode(item))
         while extractRegexResult(regex, retVal):
             value = extractRegexResult(regex, retVal)
             retVal = retVal.replace(value, '*' * len(value))
 
     # Just in case (for problematic parameters regarding user encoding)
-    for match in re.finditer(r"(?i)[ -]-(u|url|data|cookie)( |=)(.*?)(?= -?-[a-z]|\Z)", retVal):
+    for match in re.finditer(r"(?i)[ -]-(u|url|data|cookie|auth-\w+|proxy)( |=)(.*?)(?= -?-[a-z]|\Z)", retVal):
         retVal = retVal.replace(match.group(3), '*' * len(match.group(3)))
 
     # Fail-safe substitution
@@ -3543,8 +3604,8 @@ def intersect(containerA, containerB, lowerCase=False):
         containerB = arrayizeValue(containerB)
 
         if lowerCase:
-            containerA = [val.lower() if isinstance(val, basestring) else val for val in containerA]
+            containerA = [val.lower() if hasattr(val, "lower") else val for val in containerA]
-            containerB = [val.lower() if isinstance(val, basestring) else val for val in containerB]
+            containerB = [val.lower() if hasattr(val, "lower") else val for val in containerB]
 
         retVal = [val for val in containerA if val in containerB]
 
@@ -3595,14 +3656,14 @@ def removeReflectiveValues(content, payload, suppressWarning=False):
             regex = _(filterStringValue(payload, r"[A-Za-z0-9]", REFLECTED_REPLACEMENT_REGEX.encode("string_escape")))
 
             if regex != payload:
-                if all(part.lower() in content.lower() for part in filter(None, regex.split(REFLECTED_REPLACEMENT_REGEX))[1:]):  # fast optimization check
+                if all(part.lower() in content.lower() for part in filterNone(regex.split(REFLECTED_REPLACEMENT_REGEX))[1:]):  # fast optimization check
                     parts = regex.split(REFLECTED_REPLACEMENT_REGEX)
                     retVal = content.replace(payload, REFLECTED_VALUE_MARKER)  # dummy approach
 
                     if len(parts) > REFLECTED_MAX_REGEX_PARTS:  # preventing CPU hogs
                         regex = _("%s%s%s" % (REFLECTED_REPLACEMENT_REGEX.join(parts[:REFLECTED_MAX_REGEX_PARTS // 2]), REFLECTED_REPLACEMENT_REGEX, REFLECTED_REPLACEMENT_REGEX.join(parts[-REFLECTED_MAX_REGEX_PARTS // 2:])))
 
-                    parts = filter(None, regex.split(REFLECTED_REPLACEMENT_REGEX))
+                    parts = filterNone(regex.split(REFLECTED_REPLACEMENT_REGEX))
 
                     if regex.startswith(REFLECTED_REPLACEMENT_REGEX):
                         regex = r"%s%s" % (REFLECTED_BORDER_REGEX, regex[len(REFLECTED_REPLACEMENT_REGEX):])
@@ -3687,7 +3748,7 @@ def safeSQLIdentificatorNaming(name, isTable=False):
 
     retVal = name
 
-    if isinstance(name, basestring):
+    if isinstance(name, six.string_types):
         retVal = getUnicode(name)
         _ = isTable and Backend.getIdentifiedDbms() in (DBMS.MSSQL, DBMS.SYBASE)
 
@@ -3726,7 +3787,7 @@ def unsafeSQLIdentificatorNaming(name):
 
     retVal = name
 
-    if isinstance(name, basestring):
+    if isinstance(name, six.string_types):
         if Backend.getIdentifiedDbms() in (DBMS.MYSQL, DBMS.ACCESS):
             retVal = name.replace("`", "")
         elif Backend.getIdentifiedDbms() in (DBMS.PGSQL, DBMS.DB2, DBMS.SQLITE, DBMS.INFORMIX, DBMS.HSQLDB):
@@ -3757,7 +3818,7 @@ def isNoneValue(value):
     False
     """
 
-    if isinstance(value, basestring):
+    if isinstance(value, six.string_types):
         return value in ("None", "")
     elif isListLike(value):
         return all(isNoneValue(_) for _ in value)
@@ -3776,7 +3837,7 @@ def isNullValue(value):
     False
     """
 
-    return isinstance(value, basestring) and value.upper() == NULL
+    return hasattr(value, "upper") and value.upper() == NULL
 
 def expandMnemonics(mnemonics, parser, args):
     """
@@ -3878,7 +3939,7 @@ def safeCSValue(value):
 
     retVal = value
 
-    if retVal and isinstance(retVal, basestring):
+    if retVal and isinstance(retVal, six.string_types):
         if not (retVal[0] == retVal[-1] == '"'):
             if any(_ in retVal for _ in (conf.get("csvDel", defaults.csvDel), '"', '\n')):
                 retVal = '"%s"' % retVal.replace('"', '""')
@@ -3972,7 +4033,7 @@ def asciifyUrl(url, forceQuote=False):
     u'http://www.xn--uuraj-gxa24d.com'
     """
 
-    parts = urlparse.urlsplit(url)
+    parts = _urllib.parse.urlsplit(url)
     if not parts.scheme or not parts.netloc:
         # apparently not an url
         return url
@@ -3993,10 +4054,10 @@ def asciifyUrl(url, forceQuote=False):
     def quote(s, safe):
         s = s or ''
         # Triggers on non-ascii characters - another option would be:
-        #     urllib.quote(s.replace('%', '')) != s.replace('%', '')
+        #     _urllib.parse.quote(s.replace('%', '')) != s.replace('%', '')
         # which would trigger on all %-characters, e.g. "&".
         if getUnicode(s).encode("ascii", "replace") != s or forceQuote:
-            return urllib.quote(s.encode(UNICODE_ENCODING) if isinstance(s, unicode) else s, safe=safe)
+            return _urllib.parse.quote(s.encode(UNICODE_ENCODING) if isinstance(s, unicode) else s, safe=safe)
         return s
 
     username = quote(parts.username, '')
@@ -4020,7 +4081,7 @@ def asciifyUrl(url, forceQuote=False):
     if port:
         netloc += ':' + str(port)
 
-    return urlparse.urlunsplit([parts.scheme, netloc, path, query, parts.fragment]) or url
+    return _urllib.parse.urlunsplit([parts.scheme, netloc, path, query, parts.fragment]) or url
 
 def isAdminFromPrivileges(privileges):
     """
@@ -4059,9 +4120,9 @@ def findPageForms(content, url, raise_=False, addToTargets=False):
     set([(u'/input.php', 'POST', u'id=1', None, None)])
     """
 
-    class _(StringIO):
+    class _(io.BytesIO):
         def __init__(self, content, url):
-            StringIO.__init__(self, unicodeencode(content, kb.pageEncoding) if isinstance(content, unicode) else content)
+            io.BytesIO.__init__(self, unicodeencode(content, kb.pageEncoding) if isinstance(content, unicode) else content)
             self._url = url
 
         def geturl(self):
@@ -4178,7 +4239,7 @@ def checkSameHost(*urls):
                 value = "http://%s" % value
             return value
 
-        return all(re.sub(r"(?i)\Awww\.", "", urlparse.urlparse(_(url) or "").netloc.split(':')[0]) == re.sub(r"(?i)\Awww\.", "", urlparse.urlparse(_(urls[0]) or "").netloc.split(':')[0]) for url in urls[1:])
+        return all(re.sub(r"(?i)\Awww\.", "", _urllib.parse.urlparse(_(url) or "").netloc.split(':')[0]) == re.sub(r"(?i)\Awww\.", "", _urllib.parse.urlparse(_(urls[0]) or "").netloc.split(':')[0]) for url in urls[1:])
 
 def getHostHeader(url):
     """
@@ -4191,7 +4252,7 @@ def getHostHeader(url):
     retVal = url
 
     if url:
-        retVal = urlparse.urlparse(url).netloc
+        retVal = _urllib.parse.urlparse(url).netloc
 
         if re.search(r"http(s)?://\[.+\]", url, re.I):
             retVal = extractRegexResult(r"http(s)?://\[(?P<result>.+)\]", url)
@@ -4332,7 +4393,7 @@ def decodeHexValue(value, raw=False):
 
     def _(value):
         retVal = value
-        if value and isinstance(value, basestring):
+        if value and isinstance(value, six.string_types):
             if len(value) % 2 != 0:
                 retVal = "%s?" % hexdecode(value[:-1]) if len(value) > 1 else value
                 singleTimeWarnMessage("there was a problem decoding value '%s' from expected hexadecimal form" % value)
@@ -4380,7 +4441,7 @@ def extractExpectedValue(value, expected):
         elif expected == EXPECTED.BOOL:
             if isinstance(value, int):
                 value = bool(value)
-            elif isinstance(value, basestring):
+            elif isinstance(value, six.string_types):
                 value = value.strip().lower()
                 if value in ("true", "false"):
                     value = value == "true"
@@ -4393,7 +4454,7 @@ def extractExpectedValue(value, expected):
                 else:
                     value = None
         elif expected == EXPECTED.INT:
-            if isinstance(value, basestring):
+            if isinstance(value, six.string_types):
                 value = int(value) if value.isdigit() else None
 
     return value
@@ -4404,7 +4465,7 @@ def hashDBWrite(key, value, serialize=False):
     """
 
     if conf.hashDB:
-        _ = '|'.join((str(_) if not isinstance(_, basestring) else _) for _ in (conf.hostname, conf.path.strip('/') if conf.path is not None else conf.port, key, HASHDB_MILESTONE_VALUE))
+        _ = '|'.join((str(_) if not isinstance(_, six.string_types) else _) for _ in (conf.hostname, conf.path.strip('/') if conf.path is not None else conf.port, key, HASHDB_MILESTONE_VALUE))
         conf.hashDB.write(_, value, serialize)
 
 def hashDBRetrieve(key, unserialize=False, checkConf=False):
@@ -4415,10 +4476,10 @@ def hashDBRetrieve(key, unserialize=False, checkConf=False):
     retVal = None
 
     if conf.hashDB:
-        _ = '|'.join((str(_) if not isinstance(_, basestring) else _) for _ in (conf.hostname, conf.path.strip('/') if conf.path is not None else conf.port, key, HASHDB_MILESTONE_VALUE))
+        _ = '|'.join((str(_) if not isinstance(_, six.string_types) else _) for _ in (conf.hostname, conf.path.strip('/') if conf.path is not None else conf.port, key, HASHDB_MILESTONE_VALUE))
         retVal = conf.hashDB.retrieve(_, unserialize) if kb.resumeValues and not (checkConf and any((conf.flushSession, conf.freshQueries))) else None
 
-        if not kb.inferenceMode and not kb.fileReadMode and isinstance(retVal, basestring) and any(_ in retVal for _ in (PARTIAL_VALUE_MARKER, PARTIAL_HEX_VALUE_MARKER)):
+        if not kb.inferenceMode and not kb.fileReadMode and isinstance(retVal, six.string_types) and any(_ in retVal for _ in (PARTIAL_VALUE_MARKER, PARTIAL_HEX_VALUE_MARKER)):
             retVal = None
 
     return retVal
@@ -4437,7 +4498,7 @@ def resetCookieJar(cookieJar):
                 logger.info(infoMsg)
 
                 content = readCachedFileContent(conf.loadCookies)
-                lines = filter(None, (line.strip() for line in content.split("\n") if not line.startswith('#')))
+                lines = filterNone(line.strip() for line in content.split("\n") if not line.startswith('#'))
                 handle, filename = tempfile.mkstemp(prefix=MKSTEMP_PREFIX.COOKIE_JAR)
                 os.close(handle)
 
@@ -4465,7 +4526,7 @@ def resetCookieJar(cookieJar):
                 errMsg = "no valid cookies found"
                 raise SqlmapGenericException(errMsg)
 
-        except cookielib.LoadError as ex:
+        except Exception as ex:
             errMsg = "there was a problem loading "
             errMsg += "cookies file ('%s')" % re.sub(r"(cookies) file '[^']+'", r"\g<1>", getSafeExString(ex))
             raise SqlmapGenericException(errMsg)
@@ -4654,7 +4715,7 @@ def parseRequestFile(reqFile, checkParams=True):
             reqResList = re.finditer(BURP_REQUEST_REGEX, content, re.I | re.S)
 
         for match in reqResList:
-            request = match if isinstance(match, basestring) else match.group(1)
+            request = match if isinstance(match, six.string_types) else match.group(1)
             request = re.sub(r"\A[^\w]+", "", request)
             schemePort = re.search(r"(http[\w]*)\:\/\/.*?\:([\d]+).+?={10,}", request, re.I | re.S)
 
@@ -4746,7 +4807,7 @@ def parseRequestFile(reqFile, checkParams=True):
             data = data.rstrip("\r\n") if data else data
 
             if getPostReq and (params or cookie or not checkParams):
-                if not port and isinstance(scheme, basestring) and scheme.lower() == "https":
+                if not port and hasattr(scheme, "lower") and scheme.lower() == "https":
                     port = "443"
                 elif not scheme and port == "443":
                     scheme = "https"
@@ -4767,14 +4828,7 @@ def parseRequestFile(reqFile, checkParams=True):
                 if not(conf.scope and not re.search(conf.scope, url, re.I)):
                     yield (url, conf.method or method, data, cookie, tuple(headers))
 
-    checkFile(reqFile)
+    content = readCachedFileContent(reqFile)
-    try:
-        with openFile(reqFile, "rb") as f:
-            content = f.read()
-    except (IOError, OSError, MemoryError) as ex:
-        errMsg = "something went wrong while trying "
-        errMsg += "to read the content of file '%s' ('%s')" % (reqFile, getSafeExString(ex))
-        raise SqlmapSystemException(errMsg)
 
     if conf.scope:
         logger.info("using regular expression '%s' for filtering targets" % conf.scope)
@@ -4800,9 +4854,9 @@ def getSafeExString(ex, encoding=None):
         retVal = ex.message
     elif getattr(ex, "msg", None):
         retVal = ex.msg
-    elif isinstance(ex, (list, tuple)) and len(ex) > 1 and isinstance(ex[1], basestring):
+    elif isinstance(ex, (list, tuple)) and len(ex) > 1 and isinstance(ex[1], six.string_types):
         retVal = ex[1]
-    elif isinstance(ex, (list, tuple)) and len(ex) > 0 and isinstance(ex[0], basestring):
+    elif isinstance(ex, (list, tuple)) and len(ex) > 0 and isinstance(ex[0], six.string_types):
         retVal = ex[0]
 
     if retVal is None:
@@ -4816,21 +4870,27 @@ def safeVariableNaming(value):
     """
     Returns escaped safe-representation of a given variable name that can be used in Python evaluated code
 
-    >>> safeVariableNaming("foo bar")
+    >>> safeVariableNaming("class.id")
-    'foo__SAFE__20bar'
+    'EVAL_636c6173732e6964'
     """
 
-    return re.sub(r"[^\w]", lambda match: "%s%02x" % (SAFE_VARIABLE_MARKER, ord(match.group(0))), value)
+    if value in keyword.kwlist or re.search(r"\A[^a-zA-Z]|[^\w]", value):
+        value = "%s%s" % (EVALCODE_ENCODED_PREFIX, value.encode(UNICODE_ENCODING).encode("hex"))
+
+    return value
 
 def unsafeVariableNaming(value):
     """
     Returns unescaped safe-representation of a given variable name
 
-    >>> unsafeVariableNaming("foo__SAFE__20bar")
+    >>> unsafeVariableNaming("EVAL_636c6173732e6964")
-    'foo bar'
+    u'class.id'
     """
 
-    return re.sub(r"%s([0-9a-f]{2})" % SAFE_VARIABLE_MARKER, lambda match: match.group(1).decode("hex"), value)
+    if value.startswith(EVALCODE_ENCODED_PREFIX):
+        value = value[len(EVALCODE_ENCODED_PREFIX):].decode("hex").decode(UNICODE_ENCODING)
+
+    return value
 
 def firstNotNone(*args):
     """
@@ -4848,3 +4908,40 @@ def firstNotNone(*args):
             break
 
     return retVal
+
+def chunkSplitPostData(data):
+    """
+    Convert POST data to chunked transfer-encoded data (Note: splitting done by SQL keywords)
+
+    >>> random.seed(0)
+    >>> chunkSplitPostData("SELECT username,password FROM users")
+    '5;UAqFz\\r\\nSELEC\\r\\n8;sDK4F\\r\\nT userna\\r\\n3;UMp48\\r\\nme,\\r\\n8;3tT3Q\\r\\npassword\\r\\n4;gAL47\\r\\n FRO\\r\\n5;1qXIa\\r\\nM use\\r\\n2;yZPaE\\r\\nrs\\r\\n0\\r\\n\\r\\n'
+    """
+
+    length = len(data)
+    retVal = ""
+    index = 0
+
+    while index < length:
+        chunkSize = randomInt(1)
+
+        if index + chunkSize >= length:
+            chunkSize = length - index
+
+        salt = randomStr(5, alphabet=string.ascii_letters + string.digits)
+
+        while chunkSize:
+            candidate = data[index:index + chunkSize]
+
+            if re.search(r"\b%s\b" % '|'.join(HTTP_CHUNKED_SPLIT_KEYWORDS), candidate, re.I):
+                chunkSize -= 1
+            else:
+                break
+
+        index += chunkSize
+        retVal += "%x;%s\r\n" % (chunkSize, salt)
+        retVal += "%s\r\n" % candidate
+
+    retVal += "0\r\n\r\n"
+
+    return retVal

lib/core/compat.py

@@ -0,0 +1,172 @@
+#!/usr/bin/env python2
+
+"""
+Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
+See the file 'LICENSE' for copying permission
+"""
+
+import binascii
+import os
+import random
+import uuid
+import sys
+
+class WichmannHill(random.Random):
+    """
+    Reference: https://svn.python.org/projects/python/trunk/Lib/random.py
+    """
+
+    VERSION = 1     # used by getstate/setstate
+
+    def seed(self, a=None):
+        """Initialize internal state from hashable object.
+
+        None or no argument seeds from current time or from an operating
+        system specific randomness source if available.
+
+        If a is not None or an int or long, hash(a) is used instead.
+
+        If a is an int or long, a is used directly.  Distinct values between
+        0 and 27814431486575L inclusive are guaranteed to yield distinct
+        internal states (this guarantee is specific to the default
+        Wichmann-Hill generator).
+        """
+
+        if a is None:
+            try:
+                a = int(binascii.hexlify(os.urandom(16)), 16)
+            except NotImplementedError:
+                import time
+                a = int(time.time() * 256) # use fractional seconds
+
+        if not isinstance(a, int):
+            a = hash(a)
+
+        a, x = divmod(a, 30268)
+        a, y = divmod(a, 30306)
+        a, z = divmod(a, 30322)
+        self._seed = int(x)+1, int(y)+1, int(z)+1
+
+        self.gauss_next = None
+
+    def random(self):
+        """Get the next random number in the range [0.0, 1.0)."""
+
+        # Wichman-Hill random number generator.
+        #
+        # Wichmann, B. A. & Hill, I. D. (1982)
+        # Algorithm AS 183:
+        # An efficient and portable pseudo-random number generator
+        # Applied Statistics 31 (1982) 188-190
+        #
+        # see also:
+        #        Correction to Algorithm AS 183
+        #        Applied Statistics 33 (1984) 123
+        #
+        #        McLeod, A. I. (1985)
+        #        A remark on Algorithm AS 183
+        #        Applied Statistics 34 (1985),198-200
+
+        # This part is thread-unsafe:
+        # BEGIN CRITICAL SECTION
+        x, y, z = self._seed
+        x = (171 * x) % 30269
+        y = (172 * y) % 30307
+        z = (170 * z) % 30323
+        self._seed = x, y, z
+        # END CRITICAL SECTION
+
+        # Note:  on a platform using IEEE-754 double arithmetic, this can
+        # never return 0.0 (asserted by Tim; proof too long for a comment).
+        return (x/30269.0 + y/30307.0 + z/30323.0) % 1.0
+
+    def getstate(self):
+        """Return internal state; can be passed to setstate() later."""
+        return self.VERSION, self._seed, self.gauss_next
+
+    def setstate(self, state):
+        """Restore internal state from object returned by getstate()."""
+        version = state[0]
+        if version == 1:
+            version, self._seed, self.gauss_next = state
+        else:
+            raise ValueError("state with version %s passed to "
+                             "Random.setstate() of version %s" %
+                             (version, self.VERSION))
+
+    def jumpahead(self, n):
+        """Act as if n calls to random() were made, but quickly.
+
+        n is an int, greater than or equal to 0.
+
+        Example use:  If you have 2 threads and know that each will
+        consume no more than a million random numbers, create two Random
+        objects r1 and r2, then do
+            r2.setstate(r1.getstate())
+            r2.jumpahead(1000000)
+        Then r1 and r2 will use guaranteed-disjoint segments of the full
+        period.
+        """
+
+        if not n >= 0:
+            raise ValueError("n must be >= 0")
+        x, y, z = self._seed
+        x = int(x * pow(171, n, 30269)) % 30269
+        y = int(y * pow(172, n, 30307)) % 30307
+        z = int(z * pow(170, n, 30323)) % 30323
+        self._seed = x, y, z
+
+    def __whseed(self, x=0, y=0, z=0):
+        """Set the Wichmann-Hill seed from (x, y, z).
+
+        These must be integers in the range [0, 256).
+        """
+
+        if not type(x) == type(y) == type(z) == int:
+            raise TypeError('seeds must be integers')
+        if not (0 <= x < 256 and 0 <= y < 256 and 0 <= z < 256):
+            raise ValueError('seeds must be in range(0, 256)')
+        if 0 == x == y == z:
+            # Initialize from current time
+            import time
+            t = int(time.time() * 256)
+            t = int((t&0xffffff) ^ (t>>24))
+            t, x = divmod(t, 256)
+            t, y = divmod(t, 256)
+            t, z = divmod(t, 256)
+        # Zero is a poor seed, so substitute 1
+        self._seed = (x or 1, y or 1, z or 1)
+
+        self.gauss_next = None
+
+    def whseed(self, a=None):
+        """Seed from hashable object's hash code.
+
+        None or no argument seeds from current time.  It is not guaranteed
+        that objects with distinct hash codes lead to distinct internal
+        states.
+
+        This is obsolete, provided for compatibility with the seed routine
+        used prior to Python 2.1.  Use the .seed() method instead.
+        """
+
+        if a is None:
+            self.__whseed()
+            return
+        a = hash(a)
+        a, x = divmod(a, 256)
+        a, y = divmod(a, 256)
+        a, z = divmod(a, 256)
+        x = (x + a) % 256 or 1
+        y = (y + a) % 256 or 1
+        z = (z + a) % 256 or 1
+        self.__whseed(x, y, z)
+
+# Reference: https://github.com/urllib3/urllib3/blob/master/src/urllib3/filepost.py
+def choose_boundary():
+    return uuid.uuid4().hex
+
+if sys.version_info >= (3, 0):
+    xrange = range
+else:
+    xrange = xrange

lib/core/convert.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
@@ -9,18 +9,15 @@ try:
     import cPickle as pickle
 except:
     import pickle
-finally:
-    import pickle as picklePy
 
 import base64
 import json
 import re
-import StringIO
 import sys
 
 from lib.core.settings import IS_WIN
 from lib.core.settings import UNICODE_ENCODING
-from lib.core.settings import PICKLE_REDUCE_WHITELIST
+from thirdparty import six
 
 def base64decode(value):
     """
@@ -30,7 +27,7 @@ def base64decode(value):
     'foobar'
     """
 
-    return base64.b64decode(value)
+    return base64.b64decode(unicodeencode(value))
 
 def base64encode(value):
     """
@@ -40,7 +37,7 @@ def base64encode(value):
     'Zm9vYmFy'
     """
 
-    return base64.b64encode(value)
+    return base64.b64encode(unicodeencode(value))
 
 def base64pickle(value):
     """
@@ -66,7 +63,7 @@ def base64pickle(value):
 
     return retVal
 
-def base64unpickle(value, unsafe=False):
+def base64unpickle(value):
     """
     Decodes value from Base64 to plain format and deserializes (with pickle) its content
 
@@ -76,26 +73,10 @@ def base64unpickle(value, unsafe=False):
 
     retVal = None
 
-    def _(self):
-        if len(self.stack) > 1:
-            func = self.stack[-2]
-            if func not in PICKLE_REDUCE_WHITELIST:
-                raise Exception("abusing reduce() is bad, Mkay!")
-        self.load_reduce()
-
-    def loads(str):
-        f = StringIO.StringIO(str)
-        if unsafe:
-            unpickler = picklePy.Unpickler(f)
-            unpickler.dispatch[picklePy.REDUCE] = _
-        else:
-            unpickler = pickle.Unpickler(f)
-        return unpickler.load()
-
     try:
-        retVal = loads(base64decode(value))
+        retVal = pickle.loads(base64decode(value))
     except TypeError:
-        retVal = loads(base64decode(bytes(value)))
+        retVal = pickle.loads(base64decode(bytes(value)))
 
     return retVal
 
@@ -108,7 +89,14 @@ def hexdecode(value):
     """
 
     value = value.lower()
-    return (value[2:] if value.startswith("0x") else value).decode("hex")
+    value = value[2:] if value.startswith("0x") else value
+
+    if six.PY2:
+        retVal = value.decode("hex")
+    else:
+        retVal = bytes.fromhex(value)
+
+    return retVal
 
 def hexencode(value, encoding=None):
     """
@@ -118,7 +106,14 @@ def hexencode(value, encoding=None):
     '666f6f626172'
     """
 
-    return unicodeencode(value, encoding).encode("hex")
+    retVal = unicodeencode(value, encoding)
+
+    if six.PY2:
+        retVal = retVal.encode("hex")
+    else:
+        retVal = retVal.hex()
+
+    return retVal
 
 def unicodeencode(value, encoding=None):
     """
@@ -129,11 +124,13 @@ def unicodeencode(value, encoding=None):
     """
 
     retVal = value
-    if isinstance(value, unicode):
+
+    if isinstance(value, six.text_type):
         try:
             retVal = value.encode(encoding or UNICODE_ENCODING)
         except UnicodeEncodeError:
-            retVal = value.encode(UNICODE_ENCODING, "replace")
+            retVal = value.encode(encoding or UNICODE_ENCODING, "replace")
+
     return retVal
 
 def utf8encode(value):
@@ -165,7 +162,7 @@ def htmlunescape(value):
     """
 
     retVal = value
-    if value and isinstance(value, basestring):
+    if value and isinstance(value, six.string_types):
         codes = (("&lt;", '<'), ("&gt;", '>'), ("&quot;", '"'), ("&nbsp;", ' '), ("&amp;", '&'), ("&apos;", "'"))
         retVal = reduce(lambda x, y: x.replace(y[0], y[1]), codes, retVal)
         try:
@@ -183,13 +180,11 @@ def stdoutencode(data):
     retVal = None
 
     try:
-        data = data or ""
+        retVal = unicodeencode(data or "", sys.stdout.encoding)
 
         # Reference: http://bugs.python.org/issue1602
         if IS_WIN:
-            output = data.encode(sys.stdout.encoding, "replace")
+            if '?' in retVal and '?' not in retVal:
-
-            if '?' in output and '?' not in data:
                 warnMsg = "cannot properly display Unicode characters "
                 warnMsg += "inside Windows OS command prompt "
                 warnMsg += "(http://bugs.python.org/issue1602). All "
@@ -199,11 +194,8 @@ def stdoutencode(data):
                 warnMsg += "corresponding output files. "
                 singleTimeWarnMessage(warnMsg)
 
-            retVal = output
-        else:
-            retVal = data.encode(sys.stdout.encoding)
     except:
-        retVal = data.encode(UNICODE_ENCODING) if isinstance(data, unicode) else data
+        retVal = unicodeencode(data or "")
 
     return retVal
 

lib/core/data.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)

lib/core/datatype.py

@@ -1,14 +1,15 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """
 
+import collections
 import copy
 import types
 
-from thirdparty.odict.odict import OrderedDict
+from thirdparty.odict import OrderedDict
 
 class AttribDict(dict):
     """
@@ -140,3 +141,61 @@ class LRUDict(object):
 
     def keys(self):
         return self.cache.keys()
+
+# Reference: https://code.activestate.com/recipes/576694/
+class OrderedSet(collections.MutableSet):
+    def __init__(self, iterable=None):
+        self.end = end = []
+        end += [None, end, end]         # sentinel node for doubly linked list
+        self.map = {}                   # key --> [key, prev, next]
+        if iterable is not None:
+            self |= iterable
+
+    def __len__(self):
+        return len(self.map)
+
+    def __contains__(self, key):
+        return key in self.map
+
+    def add(self, key):
+        if key not in self.map:
+            end = self.end
+            curr = end[1]
+            curr[2] = end[1] = self.map[key] = [key, curr, end]
+
+    def discard(self, key):
+        if key in self.map:
+            key, prev, next = self.map.pop(key)
+            prev[2] = next
+            next[1] = prev
+
+    def __iter__(self):
+        end = self.end
+        curr = end[2]
+        while curr is not end:
+            yield curr[0]
+            curr = curr[2]
+
+    def __reversed__(self):
+        end = self.end
+        curr = end[1]
+        while curr is not end:
+            yield curr[0]
+            curr = curr[1]
+
+    def pop(self, last=True):
+        if not self:
+            raise KeyError('set is empty')
+        key = self.end[1][0] if last else self.end[2][0]
+        self.discard(key)
+        return key
+
+    def __repr__(self):
+        if not self:
+            return '%s()' % (self.__class__.__name__,)
+        return '%s(%r)' % (self.__class__.__name__, list(self))
+
+    def __eq__(self, other):
+        if isinstance(other, OrderedSet):
+            return len(self) == len(other) and list(self) == list(other)
+        return set(self) == set(other)

lib/core/decorators.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)

lib/core/defaults.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)

lib/core/dicts.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)

lib/core/dump.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
@@ -28,6 +28,7 @@ from lib.core.common import randomInt
 from lib.core.common import safeCSValue
 from lib.core.common import unicodeencode
 from lib.core.common import unsafeSQLIdentificatorNaming
+from lib.core.compat import xrange
 from lib.core.data import conf
 from lib.core.data import kb
 from lib.core.data import logger
@@ -50,6 +51,7 @@ from lib.core.settings import UNICODE_ENCODING
 from lib.core.settings import UNSAFE_DUMP_FILEPATH_REPLACEMENT
 from lib.core.settings import VERSION_STRING
 from lib.core.settings import WINDOWS_RESERVED_NAMES
+from thirdparty import six
 from thirdparty.magic import magic
 
 from extra.safe2bin.safe2bin import safechardecode
@@ -133,7 +135,7 @@ class Dump(object):
             if "\n" in _:
                 self._write("%s:\n---\n%s\n---" % (header, _))
             else:
-                self._write("%s: %s" % (header, ("'%s'" % _) if isinstance(data, basestring) else _))
+                self._write("%s: %s" % (header, ("'%s'" % _) if isinstance(data, six.string_types) else _))
         else:
             self._write("%s:\tNone" % header)
 
@@ -142,7 +144,7 @@ class Dump(object):
             try:
                 elements = set(elements)
                 elements = list(elements)
-                elements.sort(key=lambda _: _.lower() if isinstance(_, basestring) else _)
+                elements.sort(key=lambda _: _.lower() if hasattr(_, "lower") else _)
             except:
                 pass
 
@@ -154,7 +156,7 @@ class Dump(object):
             self._write("%s [%d]:" % (header, len(elements)))
 
         for element in elements:
-            if isinstance(element, basestring):
+            if isinstance(element, six.string_types):
                 self._write("[*] %s" % element)
             elif isListLike(element):
                 self._write("[*] " + ", ".join(getUnicode(e) for e in element))
@@ -193,7 +195,7 @@ class Dump(object):
             userSettings = userSettings[0]
 
         users = userSettings.keys()
-        users.sort(key=lambda _: _.lower() if isinstance(_, basestring) else _)
+        users.sort(key=lambda _: _.lower() if hasattr(_, "lower") else _)
 
         if conf.api:
             self._write(userSettings, content_type=content_type)
@@ -287,7 +289,7 @@ class Dump(object):
                     colType = None
 
                     colList = columns.keys()
-                    colList.sort(key=lambda _: _.lower() if isinstance(_, basestring) else _)
+                    colList.sort(key=lambda _: _.lower() if hasattr(_, "lower") else _)
 
                     for column in colList:
                         colType = columns[column]
@@ -379,7 +381,7 @@ class Dump(object):
                     if count is None:
                         count = "Unknown"
 
-                    tables.sort(key=lambda _: _.lower() if isinstance(_, basestring) else _)
+                    tables.sort(key=lambda _: _.lower() if hasattr(_, "lower") else _)
 
                     for table in tables:
                         blank1 = " " * (maxlength1 - len(normalizeUnicode(table) or unicode(table)))

lib/core/enums.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
@@ -231,6 +231,7 @@ class HASHDB_KEYS:
     DBMS = "DBMS"
     DBMS_FORK = "DBMS_FORK"
     CHECK_WAF_RESULT = "CHECK_WAF_RESULT"
+    CHECK_NULL_CONNECTION_RESULT = "CHECK_NULL_CONNECTION_RESULT"
     CONF_TMP_PATH = "CONF_TMP_PATH"
     KB_ABS_FILE_PATHS = "KB_ABS_FILE_PATHS"
     KB_BRUTE_COLUMNS = "KB_BRUTE_COLUMNS"
@@ -252,7 +253,7 @@ class PAYLOAD:
         2: "error-based",
         3: "inline query",
         4: "stacked queries",
-        5: "AND/OR time-based blind",
+        5: "time-based blind",
         6: "UNION query",
     }
 
@@ -377,6 +378,7 @@ class MKSTEMP_PREFIX:
     COOKIE_JAR = "sqlmapcookiejar-"
     BIG_ARRAY = "sqlmapbigarray-"
     SPECIFIC_RESPONSE = "sqlmapresponse-"
+    PREPROCESS = "sqlmappreprocess-"
 
 class TIMEOUT_STATE:
     NORMAL = 0

lib/core/exception.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)

lib/core/log.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)

lib/core/option.py

@@ -1,11 +1,10 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """
 
-import cookielib
 import glob
 import inspect
 import logging
@@ -17,8 +16,6 @@ import sys
 import tempfile
 import threading
 import time
-import urllib2
-import urlparse
 
 import lib.controller.checks
 import lib.core.common
@@ -36,6 +33,7 @@ from lib.core.common import decodeStringEscape
 from lib.core.common import getPublicTypeMembers
 from lib.core.common import getSafeExString
 from lib.core.common import getUnicode
+from lib.core.common import filterNone
 from lib.core.common import findLocalPort
 from lib.core.common import findPageForms
 from lib.core.common import getConsoleWidth
@@ -54,12 +52,14 @@ from lib.core.common import readInput
 from lib.core.common import resetCookieJar
 from lib.core.common import runningAsAdmin
 from lib.core.common import safeExpandUser
+from lib.core.common import safeFilepathEncode
 from lib.core.common import saveConfig
 from lib.core.common import setColor
 from lib.core.common import setOptimize
 from lib.core.common import setPaths
 from lib.core.common import singleTimeWarnMessage
 from lib.core.common import urldecode
+from lib.core.compat import xrange
 from lib.core.data import conf
 from lib.core.data import kb
 from lib.core.data import logger
@@ -67,6 +67,7 @@ from lib.core.data import mergedOptions
 from lib.core.data import queries
 from lib.core.datatype import AttribDict
 from lib.core.datatype import InjectionDict
+from lib.core.datatype import OrderedSet
 from lib.core.defaults import defaults
 from lib.core.dicts import DBMS_DICT
 from lib.core.dicts import DUMP_REPLACEMENTS
@@ -76,6 +77,7 @@ from lib.core.enums import CUSTOM_LOGGING
 from lib.core.enums import DUMP_FORMAT
 from lib.core.enums import HTTP_HEADER
 from lib.core.enums import HTTPMETHOD
+from lib.core.enums import MKSTEMP_PREFIX
 from lib.core.enums import MOBILES
 from lib.core.enums import OPTION_TYPE
 from lib.core.enums import PAYLOAD
@@ -122,7 +124,6 @@ from lib.core.settings import SQLMAP_ENVIRONMENT_PREFIX
 from lib.core.settings import SUPPORTED_DBMS
 from lib.core.settings import SUPPORTED_OS
 from lib.core.settings import TIME_DELAY_CANDIDATES
-from lib.core.settings import UNICODE_ENCODING
 from lib.core.settings import UNION_CHAR_REGEX
 from lib.core.settings import UNKNOWN_DBMS_VERSION
 from lib.core.settings import URI_INJECTABLE_REGEX
@@ -137,6 +138,7 @@ from lib.request.basic import checkCharEncoding
 from lib.request.connect import Connect as Request
 from lib.request.dns import DNSServer
 from lib.request.basicauthhandler import SmartHTTPBasicAuthHandler
+from lib.request.chunkedhandler import ChunkedHandler
 from lib.request.httpshandler import HTTPSHandler
 from lib.request.pkihandler import HTTPSPKIAuthHandler
 from lib.request.rangehandler import HTTPRangeHandler
@@ -147,16 +149,20 @@ from lib.utils.crawler import crawl
 from lib.utils.deps import checkDependencies
 from lib.utils.search import search
 from lib.utils.purge import purge
+from thirdparty import six
 from thirdparty.keepalive import keepalive
 from thirdparty.multipart import multipartpost
-from thirdparty.oset.pyoset import oset
+from thirdparty.six.moves import http_client as _http_client
+from thirdparty.six.moves import http_cookiejar as _http_cookiejar
+from thirdparty.six.moves import urllib as _urllib
 from thirdparty.socks import socks
 from xml.etree.ElementTree import ElementTree
 
-authHandler = urllib2.BaseHandler()
+authHandler = _urllib.request.BaseHandler()
+chunkedHandler = ChunkedHandler()
 httpsHandler = HTTPSHandler()
 keepAliveHandler = keepalive.HTTPHandler()
-proxyHandler = urllib2.ProxyHandler()
+proxyHandler = _urllib.request.ProxyHandler()
 redirectHandler = SmartRedirectHandler()
 rangeHandler = HTTPRangeHandler()
 multipartPostHandler = multipartpost.MultipartPostHandler()
@@ -224,7 +230,7 @@ def _setMultipleTargets():
         errMsg = "the specified list of targets does not exist"
         raise SqlmapFilePathException(errMsg)
 
-    if os.path.isfile(conf.logFile):
+    if checkFile(conf.logFile, False):
         for target in parseRequestFile(conf.logFile):
             url, _, data, _, _ = target
             key = re.sub(r"(\w+=)[^%s ]*" % (conf.paramDel or DEFAULT_GET_POST_DELIMITER), r"\g<1>", "%s %s" % (url, data))
@@ -290,7 +296,7 @@ def _setRequestFromFile():
         conf.requestFile = safeExpandUser(conf.requestFile)
         seen = set()
 
-        if not os.path.isfile(conf.requestFile):
+        if not checkFile(conf.requestFile, False):
             errMsg = "specified HTTP request file '%s' " % conf.requestFile
             errMsg += "does not exist"
             raise SqlmapFilePathException(errMsg)
@@ -307,7 +313,7 @@ def _setRequestFromFile():
     if conf.secondReq:
         conf.secondReq = safeExpandUser(conf.secondReq)
 
-        if not os.path.isfile(conf.secondReq):
+        if not checkFile(conf.secondReq, False):
             errMsg = "specified second-order HTTP request file '%s' " % conf.secondReq
             errMsg += "does not exist"
             raise SqlmapFilePathException(errMsg)
@@ -409,7 +415,7 @@ def _setBulkMultipleTargets():
     infoMsg = "parsing multiple targets list from '%s'" % conf.bulkFile
     logger.info(infoMsg)
 
-    if not os.path.isfile(conf.bulkFile):
+    if not checkFile(conf.bulkFile, False):
         errMsg = "the specified bulk file "
         errMsg += "does not exist"
         raise SqlmapFilePathException(errMsg)
@@ -655,7 +661,7 @@ def _setTechnique():
     validTechniques = sorted(getPublicTypeMembers(PAYLOAD.TECHNIQUE), key=lambda x: x[1])
     validLetters = [_[0][0].upper() for _ in validTechniques]
 
-    if conf.tech and isinstance(conf.tech, basestring):
+    if conf.tech and isinstance(conf.tech, six.string_types):
         _ = []
 
         for letter in conf.tech.upper():
@@ -733,8 +739,8 @@ def _setTamperingFunctions():
         for script in re.split(PARAMETER_SPLITTING_REGEX, conf.tamper):
             found = False
 
-            path = paths.SQLMAP_TAMPER_PATH.encode(sys.getfilesystemencoding() or UNICODE_ENCODING)
+            path = safeFilepathEncode(paths.SQLMAP_TAMPER_PATH)
-            script = script.strip().encode(sys.getfilesystemencoding() or UNICODE_ENCODING)
+            script = safeFilepathEncode(script.strip())
 
             try:
                 if not script:
@@ -769,9 +775,9 @@ def _setTamperingFunctions():
                 sys.path.insert(0, dirname)
 
             try:
-                module = __import__(filename[:-3].encode(sys.getfilesystemencoding() or UNICODE_ENCODING))
+                module = __import__(safeFilepathEncode(filename[:-3]))
             except Exception as ex:
-                raise SqlmapSyntaxException("cannot import tamper module '%s' (%s)" % (filename[:-3], getSafeExString(ex)))
+                raise SqlmapSyntaxException("cannot import tamper module '%s' (%s)" % (getUnicode(filename[:-3]), getSafeExString(ex)))
 
             priority = PRIORITY.NORMAL if not hasattr(module, "__priority__") else module.__priority__
 
@@ -779,7 +785,7 @@ def _setTamperingFunctions():
                 if name == "tamper" and inspect.getargspec(function).args and inspect.getargspec(function).keywords == "kwargs":
                     found = True
                     kb.tamperFunctions.append(function)
-                    function.func_name = module.__name__
+                    function.__name__ = module.__name__
 
                     if check_priority and priority > last_priority:
                         message = "it appears that you might have mixed "
@@ -805,7 +811,7 @@ def _setTamperingFunctions():
                         function()
                     except Exception as ex:
                         errMsg = "error occurred while checking dependencies "
-                        errMsg += "for tamper module '%s' ('%s')" % (filename[:-3], getSafeExString(ex))
+                        errMsg += "for tamper module '%s' ('%s')" % (getUnicode(filename[:-3]), getSafeExString(ex))
                         raise SqlmapGenericException(errMsg)
 
             if not found:
@@ -825,6 +831,80 @@ def _setTamperingFunctions():
             for _, function in priorities:
                 kb.tamperFunctions.append(function)
 
+def _setPreprocessFunctions():
+    """
+    Loads preprocess functions from given script(s)
+    """
+
+    if conf.preprocess:
+        for script in re.split(PARAMETER_SPLITTING_REGEX, conf.preprocess):
+            found = False
+
+            script = safeFilepathEncode(script.strip())
+
+            try:
+                if not script:
+                    continue
+
+                if not os.path.exists(script):
+                    errMsg = "preprocess script '%s' does not exist" % script
+                    raise SqlmapFilePathException(errMsg)
+
+                elif not script.endswith(".py"):
+                    errMsg = "preprocess script '%s' should have an extension '.py'" % script
+                    raise SqlmapSyntaxException(errMsg)
+            except UnicodeDecodeError:
+                errMsg = "invalid character provided in option '--preprocess'"
+                raise SqlmapSyntaxException(errMsg)
+
+            dirname, filename = os.path.split(script)
+            dirname = os.path.abspath(dirname)
+
+            infoMsg = "loading preprocess module '%s'" % filename[:-3]
+            logger.info(infoMsg)
+
+            if not os.path.exists(os.path.join(dirname, "__init__.py")):
+                errMsg = "make sure that there is an empty file '__init__.py' "
+                errMsg += "inside of preprocess scripts directory '%s'" % dirname
+                raise SqlmapGenericException(errMsg)
+
+            if dirname not in sys.path:
+                sys.path.insert(0, dirname)
+
+            try:
+                module = __import__(safeFilepathEncode(filename[:-3]))
+            except Exception as ex:
+                raise SqlmapSyntaxException("cannot import preprocess module '%s' (%s)" % (getUnicode(filename[:-3]), getSafeExString(ex)))
+
+            for name, function in inspect.getmembers(module, inspect.isfunction):
+                if name == "preprocess" and inspect.getargspec(function).args and all(_ in inspect.getargspec(function).args for _ in ("page", "headers", "code")):
+                    found = True
+
+                    kb.preprocessFunctions.append(function)
+                    function.__name__ = module.__name__
+
+                    break
+
+            if not found:
+                errMsg = "missing function 'preprocess(page, headers=None, code=None)' "
+                errMsg += "in preprocess script '%s'" % script
+                raise SqlmapGenericException(errMsg)
+            else:
+                try:
+                    _, _, _ = function("", {}, None)
+                except:
+                    handle, filename = tempfile.mkstemp(prefix=MKSTEMP_PREFIX.PREPROCESS, suffix=".py")
+                    os.close(handle)
+
+                    open(filename, "w+b").write("#!/usr/bin/env\n\ndef preprocess(page, headers=None, code=None):\n    return page, headers, code\n")
+                    open(os.path.join(os.path.dirname(filename), "__init__.py"), "w+b").write("pass")
+
+                    errMsg = "function 'preprocess(page, headers=None, code=None)' "
+                    errMsg += "in preprocess script '%s' " % script
+                    errMsg += "should return a tuple '(page, headers, code)' "
+                    errMsg += "(Note: find template script at '%s')" % filename
+                    raise SqlmapGenericException(errMsg)
+
 def _setWafFunctions():
     """
     Loads WAF/IPS detecting functions from script(s)
@@ -847,9 +927,9 @@ def _setWafFunctions():
             try:
                 if filename[:-3] in sys.modules:
                     del sys.modules[filename[:-3]]
-                module = __import__(filename[:-3].encode(sys.getfilesystemencoding() or UNICODE_ENCODING))
+                module = __import__(safeFilepathEncode(filename[:-3]))
             except ImportError as ex:
-                raise SqlmapSyntaxException("cannot import WAF script '%s' (%s)" % (filename[:-3], getSafeExString(ex)))
+                raise SqlmapSyntaxException("cannot import WAF script '%s' (%s)" % (getUnicode(filename[:-3]), getSafeExString(ex)))
 
             _ = dict(inspect.getmembers(module))
             if "detect" not in _:
@@ -975,7 +1055,7 @@ def _setHTTPHandlers():
         logger.debug(debugMsg)
 
         try:
-            _ = urlparse.urlsplit(conf.proxy)
+            _ = _urllib.parse.urlsplit(conf.proxy)
         except Exception as ex:
             errMsg = "invalid proxy address '%s' ('%s')" % (conf.proxy, getSafeExString(ex))
             raise SqlmapSyntaxException(errMsg)
@@ -1011,10 +1091,14 @@ def _setHTTPHandlers():
         if scheme in (PROXY_TYPE.SOCKS4, PROXY_TYPE.SOCKS5):
             proxyHandler.proxies = {}
 
+            if scheme == PROXY_TYPE.SOCKS4:
+                warnMsg = "SOCKS4 does not support resolving (DNS) names (i.e. causing DNS leakage)"
+                singleTimeWarnMessage(warnMsg)
+
             socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS5 if scheme == PROXY_TYPE.SOCKS5 else socks.PROXY_TYPE_SOCKS4, hostname, port, username=username, password=password)
-            socks.wrapmodule(urllib2)
+            socks.wrapmodule(_http_client)
         else:
-            socks.unwrapmodule(urllib2)
+            socks.unwrapmodule(_http_client)
 
             if conf.proxyCred:
                 # Reference: http://stackoverflow.com/questions/34079/how-to-specify-an-authenticated-proxy-for-a-python-http-connection
@@ -1030,16 +1114,16 @@ def _setHTTPHandlers():
     debugMsg = "creating HTTP requests opener object"
     logger.debug(debugMsg)
 
-    handlers = filter(None, [multipartPostHandler, proxyHandler if proxyHandler.proxies else None, authHandler, redirectHandler, rangeHandler, httpsHandler])
+    handlers = filterNone([multipartPostHandler, proxyHandler if proxyHandler.proxies else None, authHandler, redirectHandler, rangeHandler, chunkedHandler if conf.chunked else None, httpsHandler])
 
     if not conf.dropSetCookie:
         if not conf.loadCookies:
-            conf.cj = cookielib.CookieJar()
+            conf.cj = _http_cookiejar.CookieJar()
         else:
-            conf.cj = cookielib.MozillaCookieJar()
+            conf.cj = _http_cookiejar.MozillaCookieJar()
             resetCookieJar(conf.cj)
 
-        handlers.append(urllib2.HTTPCookieProcessor(conf.cj))
+        handlers.append(_urllib.request.HTTPCookieProcessor(conf.cj))
 
     # Reference: http://www.w3.org/Protocols/rfc2616/rfc2616-sec8.html
     if conf.keepAlive:
@@ -1055,8 +1139,8 @@ def _setHTTPHandlers():
         else:
             handlers.append(keepAliveHandler)
 
-    opener = urllib2.build_opener(*handlers)
+    opener = _urllib.request.build_opener(*handlers)
-    urllib2.install_opener(opener)
+    _urllib.request.install_opener(opener)
 
 def _setSafeVisit():
     """
@@ -1069,14 +1153,14 @@ def _setSafeVisit():
         checkFile(conf.safeReqFile)
 
         raw = readCachedFileContent(conf.safeReqFile)
-        match = re.search(r"\A([A-Z]+) ([^ ]+) HTTP/[0-9.]+\Z", raw[:raw.find('\n')])
+        match = re.search(r"\A([A-Z]+) ([^ ]+) HTTP/[0-9.]+\Z", raw.split('\n')[0].strip())
 
         if match:
             kb.safeReq.method = match.group(1)
             kb.safeReq.url = match.group(2)
             kb.safeReq.headers = {}
 
-            for line in raw[raw.find('\n') + 1:].split('\n'):
+            for line in raw.split('\n')[1:]:
                 line = line.strip()
                 if line and ':' in line:
                     key, value = line.split(':', 1)
@@ -1088,7 +1172,7 @@ def _setSafeVisit():
                             if value.endswith(":443"):
                                 scheme = "https"
                             value = "%s://%s" % (scheme, value)
-                        kb.safeReq.url = urlparse.urljoin(value, kb.safeReq.url)
+                        kb.safeReq.url = _urllib.parse.urljoin(value, kb.safeReq.url)
                 else:
                     break
 
@@ -1211,7 +1295,7 @@ def _setHTTPAuthentication():
         conf.authUsername = aCredRegExp.group(1)
         conf.authPassword = aCredRegExp.group(2)
 
-        kb.passwordMgr = urllib2.HTTPPasswordMgrWithDefaultRealm()
+        kb.passwordMgr = _urllib.request.HTTPPasswordMgrWithDefaultRealm()
 
         _setAuthCred()
 
@@ -1219,7 +1303,7 @@ def _setHTTPAuthentication():
             authHandler = SmartHTTPBasicAuthHandler(kb.passwordMgr)
 
         elif authType == AUTH_TYPE.DIGEST:
-            authHandler = urllib2.HTTPDigestAuthHandler(kb.passwordMgr)
+            authHandler = _urllib.request.HTTPDigestAuthHandler(kb.passwordMgr)
 
         elif authType == AUTH_TYPE.NTLM:
             try:
@@ -1381,7 +1465,7 @@ def _setHostname():
 
     if conf.url:
         try:
-            conf.hostname = urlparse.urlsplit(conf.url).netloc.split(':')[0]
+            conf.hostname = _urllib.parse.urlsplit(conf.url).netloc.split(':')[0]
         except ValueError as ex:
             errMsg = "problem occurred while "
             errMsg += "parsing an URL '%s' ('%s')" % (conf.url, getSafeExString(ex))
@@ -1615,8 +1699,8 @@ def _cleanupOptions():
             re.compile(conf.csrfToken)
 
             if re.escape(conf.csrfToken) != conf.csrfToken:
-                message = "provided value for option '--csrf-token' is a regular expression? [Y/n] "
+                message = "provided value for option '--csrf-token' is a regular expression? [y/N] "
-                if not readInput(message, default='Y', boolean=True):
+                if not readInput(message, default='N', boolean=True):
                     conf.csrfToken = re.escape(conf.csrfToken)
         except re.error:
             conf.csrfToken = re.escape(conf.csrfToken)
@@ -1656,7 +1740,7 @@ def _cleanupOptions():
     if conf.csvDel:
         conf.csvDel = decodeStringEscape(conf.csvDel)
 
-    if conf.torPort and isinstance(conf.torPort, basestring) and conf.torPort.isdigit():
+    if conf.torPort and hasattr(conf.torPort, "isdigit") and conf.torPort.isdigit():
         conf.torPort = int(conf.torPort)
 
     if conf.torType:
@@ -1705,8 +1789,8 @@ def _cleanupEnvironment():
     Cleanup environment (e.g. from leftovers after --sqlmap-shell).
     """
 
-    if issubclass(urllib2.socket.socket, socks.socksocket):
+    if issubclass(_http_client.socket.socket, socks.socksocket):
-        socks.unwrapmodule(urllib2)
+        socks.unwrapmodule(_http_client)
 
     if hasattr(socket, "_ready"):
         socket._ready.clear()
@@ -1803,6 +1887,7 @@ def _setKnowledgeBaseAttributes(flushAll=True):
     kb.connErrorChoice = None
     kb.connErrorCounter = 0
     kb.cookieEncodeChoice = None
+    kb.copyExecTest = None
     kb.counters = {}
     kb.customInjectionMark = CUSTOM_INJECTION_MARK_CHAR
     kb.data = AttribDict()
@@ -1937,9 +2022,10 @@ def _setKnowledgeBaseAttributes(flushAll=True):
         kb.headerPaths = {}
         kb.keywords = set(getFileItems(paths.SQL_KEYWORDS))
         kb.passwordMgr = None
+        kb.preprocessFunctions = []
         kb.skipVulnHost = None
         kb.tamperFunctions = []
-        kb.targets = oset()
+        kb.targets = OrderedSet()
         kb.testedParams = set()
         kb.userAgents = None
         kb.vainRun = True
@@ -2232,7 +2318,11 @@ def _setTorSocksProxySettings():
 
     # SOCKS5 to prevent DNS leaks (http://en.wikipedia.org/wiki/Tor_%28anonymity_network%29)
     socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS5 if conf.torType == PROXY_TYPE.SOCKS5 else socks.PROXY_TYPE_SOCKS4, LOCALHOST, port)
-    socks.wrapmodule(urllib2)
+    socks.wrapmodule(_http_client)
+
+def _setHttpChunked():
+    if conf.chunked and conf.data:
+        _http_client.HTTPConnection._set_content_length = lambda self, a, b: None
 
 def _checkWebSocket():
     if conf.url and (conf.url.startswith("ws:/") or conf.url.startswith("wss:/")):
@@ -2321,6 +2411,10 @@ def _basicOptionValidation():
         errMsg = "switch '--dump' is incompatible with switch '--search'"
         raise SqlmapSyntaxException(errMsg)
 
+    if conf.chunked and not any((conf.data, conf.requestFile, conf.forms)):
+        errMsg = "switch '--chunked' requires usage of (POST) options/switches '--data', '-r' or '--forms'"
+        raise SqlmapSyntaxException(errMsg)
+
     if conf.api and not conf.configFile:
         errMsg = "switch '--api' requires usage of option '-c'"
         raise SqlmapSyntaxException(errMsg)
@@ -2485,7 +2579,7 @@ def _basicOptionValidation():
         errMsg = "option '--crack' should be used as a standalone"
         raise SqlmapSyntaxException(errMsg)
 
-    if isinstance(conf.uCols, basestring):
+    if isinstance(conf.uCols, six.string_types):
         if not conf.uCols.isdigit() and ("-" not in conf.uCols or len(conf.uCols.split("-")) != 2):
             errMsg = "value for option '--union-cols' must be a range with hyphon "
             errMsg += "(e.g. 1-10) or integer value (e.g. 5)"
@@ -2549,10 +2643,12 @@ def init():
     _setMultipleTargets()
     _listTamperingFunctions()
     _setTamperingFunctions()
+    _setPreprocessFunctions()
     _setWafFunctions()
     _setTrafficOutputFP()
     _setupHTTPCollector()
     _resolveCrossReferences()
+    _setHttpChunked()
     _checkWebSocket()
 
     parseTargetDirect()

lib/core/optiondict.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
@@ -61,6 +61,7 @@ optDict = {
         "csrfToken": "string",
         "csrfUrl": "string",
         "forceSSL": "boolean",
+        "chunked": "boolean",
         "hpp": "boolean",
         "evalCode": "string",
     },
@@ -192,7 +193,6 @@ optDict = {
     },
 
     "General": {
-        # "xmlFile": "string",
         "trafficFile": "string",
         "batch": "boolean",
         "binaryFields": "string",
@@ -211,6 +211,7 @@ optDict = {
         "hexConvert": "boolean",
         "outputDir": "string",
         "parseErrors": "boolean",
+        "preprocess": "string",
         "repair": "boolean",
         "saveConfig": "string",
         "scope": "string",

lib/core/patch.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
@@ -6,9 +6,9 @@ See the file 'LICENSE' for copying permission
 """
 
 import codecs
-import httplib
 
 from lib.core.settings import IS_WIN
+from thirdparty.six.moves import http_client as _http_client
 
 def dirtyPatches():
     """
@@ -16,7 +16,7 @@ def dirtyPatches():
     """
 
     # accept overly long result lines (e.g. SQLi results in HTTP header responses)
-    httplib._MAXLINE = 1 * 1024 * 1024
+    _http_client._MAXLINE = 1 * 1024 * 1024
 
     # add support for inet_pton() on Windows OS
     if IS_WIN:
@@ -24,3 +24,11 @@ def dirtyPatches():
 
     # Reference: https://github.com/nodejs/node/issues/12786#issuecomment-298652440
     codecs.register(lambda name: codecs.lookup("utf-8") if name == "cp65001" else None)
+
+    # Reference: http://bugs.python.org/issue17849
+    if hasattr(_http_client, "LineAndFileWrapper"):
+        def _(self, *args):
+            return self._readline()
+
+        _http_client.LineAndFileWrapper._readline = _http_client.LineAndFileWrapper.readline
+        _http_client.LineAndFileWrapper.readline = _

lib/core/profiling.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)

lib/core/readlineng.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)

lib/core/replication.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)

lib/core/revision.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
@@ -44,9 +44,12 @@ def getRevisionNumber():
             break
 
     if not retVal:
-        process = subprocess.Popen("git rev-parse --verify HEAD", shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+        try:
-        stdout, _ = process.communicate()
+            process = subprocess.Popen("git rev-parse --verify HEAD", shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
-        match = re.search(r"(?i)[0-9a-f]{32}", stdout or "")
+            stdout, _ = process.communicate()
-        retVal = match.group(0) if match else None
+            match = re.search(r"(?i)[0-9a-f]{32}", stdout or "")
+            retVal = match.group(0) if match else None
+        except:
+            pass
 
     return retVal[:7] if retVal else None

lib/core/session.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)

lib/core/settings.py

@@ -1,25 +1,23 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """
 
+import codecs
 import os
 import random
 import re
-import subprocess
 import string
 import sys
-import types
 
-from lib.core.datatype import AttribDict
 from lib.core.enums import DBMS
 from lib.core.enums import DBMS_DIRECTORY_NAME
 from lib.core.enums import OS
 
 # sqlmap version (<major>.<minor>.<month>.<monthly commit>)
-VERSION = "1.3.3.0"
+VERSION = "1.3.4.0"
 TYPE = "dev" if VERSION.count('.') > 2 and VERSION.split('.')[-1] != '0' else "stable"
 TYPE_COLORS = {"dev": 33, "stable": 90, "pip": 34}
 VERSION_STRING = "sqlmap/%s#%s" % ('.'.join(VERSION.split('.')[:-1]) if VERSION.count('.') > 2 and VERSION.split('.')[-1] == '0' else VERSION, TYPE)
@@ -100,7 +98,7 @@ MAX_CONSECUTIVE_CONNECTION_ERRORS = 15
 PRECONNECT_CANDIDATE_TIMEOUT = 10
 
 # Servers known to cause issue with pre-connection mechanism (because of lack of multi-threaded support)
-PRECONNECT_INCOMPATIBLE_SERVERS = ("SimpleHTTP",)
+PRECONNECT_INCOMPATIBLE_SERVERS = ("SimpleHTTP", "BaseHTTP")
 
 # Maximum sleep time in "Murphy" (testing) mode
 MAX_MURPHY_SLEEP_TIME = 3
@@ -218,19 +216,20 @@ DUMMY_USER_PREFIX = "__dummy__"
 DEFAULT_PAGE_ENCODING = "iso-8859-1"
 
 try:
-    unicode(DEFAULT_PAGE_ENCODING, DEFAULT_PAGE_ENCODING)
+    codecs.lookup(DEFAULT_PAGE_ENCODING)
 except LookupError:
     DEFAULT_PAGE_ENCODING = "utf8"
 
+# Marker for program piped input
+STDIN_PIPE_DASH = '-'
+
 # URL used in dummy runs
 DUMMY_URL = "http://foo/bar?id=1"
 
-# System variables
-IS_WIN = subprocess.mswindows
-
 # The name of the operating system dependent module imported. The following names have currently been registered: 'posix', 'nt', 'mac', 'os2', 'ce', 'java', 'riscos'
 PLATFORM = os.name
 PYVERSION = sys.version.split()[0]
+IS_WIN = PLATFORM == "nt"
 
 # DBMS system databases
 MSSQL_SYSTEM_DBS = ("Northwind", "master", "model", "msdb", "pubs", "tempdb")
@@ -338,10 +337,11 @@ ERROR_PARSING_REGEXES = (
     r"<b>[^<]*(fatal|error|warning|exception)[^<]*</b>:?\s*(?P<result>[^<]+)",
     r"(?m)^\s*(fatal|error|warning|exception):?\s*(?P<result>[^\n]+?)$",
     r"(?P<result>[^\n>]*SQL Syntax[^\n<]+)",
-    r"<li>Error Type:<br>(?P<result>.+?)</li>",
+    r"(?s)<li>Error Type:<br>(?P<result>.+?)</li>",
     r"CDbCommand (?P<result>[^<>\n]*SQL[^<>\n]+)",
     r"error '[0-9a-f]{8}'((<[^>]+>)|\s)+(?P<result>[^<>]+)",
-    r"\[[^\n\]]+(ODBC|JDBC)[^\n\]]+\](\[[^\]]+\])?(?P<result>[^\n]+(in query expression|\(SQL| at /[^ ]+pdo)[^\n<]+)"
+    r"\[[^\n\]]+(ODBC|JDBC)[^\n\]]+\](\[[^\]]+\])?(?P<result>[^\n]+(in query expression|\(SQL| at /[^ ]+pdo)[^\n<]+)",
+    r"(?P<result>query error: SELECT[^<>]+)"
 )
 
 # Regular expression used for parsing charset info from meta html headers
@@ -378,7 +378,7 @@ URI_HTTP_HEADER = "URI"
 URI_INJECTABLE_REGEX = r"//[^/]*/([^\.*?]+)\Z"
 
 # Regex used for masking sensitive data
-SENSITIVE_DATA_REGEX = r"(\s|=)(?P<result>[^\s=]*%s[^\s]*)\s"
+SENSITIVE_DATA_REGEX = r"(\s|=)(?P<result>[^\s=]*\b%s\b[^\s]*)\s"
 
 # Options to explicitly mask in anonymous (unhandled exception) reports (along with anything carrying the <hostname> inside)
 SENSITIVE_OPTIONS = ("hostname", "answers", "data", "dnsDomain", "googleDork", "authCred", "proxyCred", "tbl", "db", "col", "user", "cookie", "proxy", "fileRead", "fileWrite", "fileDest", "testParameter", "authCred")
@@ -444,7 +444,7 @@ HASH_MOD_ITEM_DISPLAY = 11
 HASH_EMPTY_PASSWORD_MARKER = "<empty>"
 
 # Maximum integer value
-MAX_INT = sys.maxint
+MAX_INT = sys.maxsize
 
 # Replacement for unsafe characters in dump table filenames
 UNSAFE_DUMP_FILEPATH_REPLACEMENT = '_'
@@ -509,8 +509,6 @@ HTML_TITLE_REGEX = r"<title>(?P<result>[^<]+)</title>"
 # Table used for Base64 conversion in WordPress hash cracking routine
 ITOA64 = "./0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz"
 
-PICKLE_REDUCE_WHITELIST = (types.BooleanType, types.DictType, types.FloatType, types.IntType, types.ListType, types.LongType, types.NoneType, types.StringType, types.TupleType, types.UnicodeType, types.XRangeType, type(AttribDict()), type(set()))
-
 # Chars used to quickly distinguish if the user provided tainted parameter values
 DUMMY_SQL_INJECTION_CHARS = ";()'"
 
@@ -761,8 +759,8 @@ METASPLOIT_SESSION_TIMEOUT = 120
 # Reference: http://www.postgresql.org/docs/9.0/static/catalog-pg-largeobject.html
 LOBLKSIZE = 2048
 
-# Suffix used to mark variables having keyword names
+# Prefix used to mark special variables (e.g. keywords, having special chars, etc.)
-EVALCODE_KEYWORD_SUFFIX = "_KEYWORD"
+EVALCODE_ENCODED_PREFIX = "EVAL_"
 
 # Reference: http://www.cookiecentral.com/faq/#3.5
 NETSCAPE_FORMAT_HEADER_COOKIES = "# Netscape HTTP Cookie File."
@@ -791,6 +789,9 @@ KB_CHARS_BOUNDARY_CHAR = 'q'
 # Letters of lower frequency used in kb.chars
 KB_CHARS_LOW_FREQUENCY_ALPHABET = "zqxjkvbp"
 
+# SQL keywords used for splitting in HTTP chunked transfer encoded requests (switch --chunk)
+HTTP_CHUNKED_SPLIT_KEYWORDS = ("SELECT", "UPDATE", "INSERT", "FROM", "LOAD_FILE", "UNION", "information_schema", "sysdatabases", "msysaccessobjects", "msysqueries", "sysmodules")
+
 # CSS style used in HTML dump format
 HTML_DUMP_CSS_STYLE = """<style>
 table{

lib/core/shell.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)

lib/core/subprocessng.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
@@ -85,7 +85,7 @@ class Popen(subprocess.Popen):
         getattr(self, which).close()
         setattr(self, which, None)
 
-    if subprocess.mswindows:
+    if IS_WIN:
         def send(self, input):
             if not self.stdin:
                 return None

lib/core/target.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
@@ -12,7 +12,6 @@ import subprocess
 import sys
 import tempfile
 import time
-import urlparse
 
 from lib.core.common import Backend
 from lib.core.common import getSafeExString
@@ -27,6 +26,7 @@ from lib.core.common import randomStr
 from lib.core.common import readInput
 from lib.core.common import resetCookieJar
 from lib.core.common import urldecode
+from lib.core.compat import xrange
 from lib.core.data import conf
 from lib.core.data import kb
 from lib.core.data import logger
@@ -73,7 +73,8 @@ from lib.core.settings import URI_INJECTABLE_REGEX
 from lib.core.settings import USER_AGENT_ALIASES
 from lib.core.settings import XML_RECOGNITION_REGEX
 from lib.utils.hashdb import HashDB
-from thirdparty.odict.odict import OrderedDict
+from thirdparty.odict import OrderedDict
+from thirdparty.six.moves import urllib as _urllib
 
 def _setRequestParams():
     """
@@ -212,7 +213,7 @@ def _setRequestParams():
                 if not (kb.processUserMarks and kb.customInjectionMark in conf.data):
                     conf.data = getattr(conf.data, UNENCODED_ORIGINAL_VALUE, conf.data)
                     conf.data = conf.data.replace(kb.customInjectionMark, ASTERISK_MARKER)
-                    conf.data = re.sub(r"(?si)((Content-Disposition[^\n]+?name\s*=\s*[\"']?(?P<name>[^\"'\r\n]+)[\"']?).+?)(((\r)?\n)+--)", functools.partial(process, repl=r"\g<1>%s\g<4>" % kb.customInjectionMark), conf.data)
+                    conf.data = re.sub(r"(?si)((Content-Disposition[^\n]+?name\s*=\s*[\"']?(?P<name>[^\"'\r\n]+)[\"']?).+?)((%s)+--)" % ("\r\n" if "\r\n" in conf.data else '\n'), functools.partial(process, repl=r"\g<1>%s\g<4>" % kb.customInjectionMark), conf.data)
 
                 kb.postHint = POST_HINT.MULTIPART
 
@@ -276,7 +277,7 @@ def _setRequestParams():
 
             if not kb.processUserMarks:
                 if place == PLACE.URI:
-                    query = urlparse.urlsplit(value).query
+                    query = _urllib.parse.urlsplit(value).query
                     if query:
                         parameters = conf.parameters[PLACE.GET] = query
                         paramDict = paramToDict(PLACE.GET, parameters)
@@ -393,7 +394,7 @@ def _setRequestParams():
         raise SqlmapGenericException(errMsg)
 
     if conf.csrfToken:
-        if not any(re.search(conf.csrfToken, ' '.join(_), re.I) for _ in (conf.paramDict.get(PLACE.GET, {}), conf.paramDict.get(PLACE.POST, {}))) and not re.search(r"\b%s\b" % re.escape(conf.csrfToken), conf.data or "") and conf.csrfToken not in set(_[0].lower() for _ in conf.httpHeaders) and conf.csrfToken not in conf.paramDict.get(PLACE.COOKIE, {}):
+        if not any(re.search(conf.csrfToken, ' '.join(_), re.I) for _ in (conf.paramDict.get(PLACE.GET, {}), conf.paramDict.get(PLACE.POST, {}))) and not re.search(r"\b%s\b" % conf.csrfToken, conf.data or "") and conf.csrfToken not in set(_[0].lower() for _ in conf.httpHeaders) and conf.csrfToken not in conf.paramDict.get(PLACE.COOKIE, {}):
             errMsg = "anti-CSRF token parameter '%s' not " % conf.csrfToken._original
             errMsg += "found in provided GET, POST, Cookie or header values"
             raise SqlmapGenericException(errMsg)

lib/core/testing.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)

lib/core/threads.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
@@ -8,11 +8,12 @@ See the file 'LICENSE' for copying permission
 from __future__ import print_function
 
 import difflib
-import random
 import threading
 import time
 import traceback
 
+from lib.core.compat import WichmannHill
+from lib.core.compat import xrange
 from lib.core.data import conf
 from lib.core.data import kb
 from lib.core.data import logger
@@ -57,7 +58,7 @@ class _ThreadData(threading.local):
         self.lastRequestMsg = None
         self.lastRequestUID = 0
         self.lastRedirectURL = None
-        self.random = random.WichmannHill()
+        self.random = WichmannHill()
         self.resumed = False
         self.retriesCount = 0
         self.seqMatcher = difflib.SequenceMatcher(None)

lib/core/unescaper.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)

lib/core/update.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)

lib/core/wordlist.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)

lib/parse/__init__.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)

lib/parse/banner.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)

lib/parse/cmdline.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
@@ -22,6 +22,7 @@ from lib.core.common import checkSystemEncoding
 from lib.core.common import dataToStdout
 from lib.core.common import expandMnemonics
 from lib.core.common import getUnicode
+from lib.core.compat import xrange
 from lib.core.data import cmdLineOptions
 from lib.core.data import conf
 from lib.core.data import logger
@@ -215,6 +216,9 @@ def cmdLineParser(argv=None):
         request.add_option("--force-ssl", dest="forceSSL", action="store_true",
                            help="Force usage of SSL/HTTPS")
 
+        request.add_option("--chunked", dest="chunked", action="store_true",
+                           help="Use HTTP chunked transfer encoded (POST) requests")
+
         request.add_option("--hpp", dest="hpp", action="store_true",
                            help="Use HTTP parameter pollution method")
 
@@ -595,6 +599,9 @@ def cmdLineParser(argv=None):
         general.add_option("--parse-errors", dest="parseErrors", action="store_true",
                            help="Parse and display DBMS error messages from responses")
 
+        general.add_option("--preprocess", dest="preprocess",
+                             help="Use given script(s) for preprocessing of response data")
+
         general.add_option("--repair", dest="repair", action="store_true",
                            help="Redump entries having unknown character marker (%s)" % INFERENCE_UNKNOWN_CHAR)
 

lib/parse/configfile.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)

lib/parse/handler.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)

lib/parse/headers.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)

lib/parse/html.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)

lib/parse/payloads.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
@@ -11,6 +11,7 @@ import re
 from xml.etree import ElementTree as et
 
 from lib.core.common import getSafeExString
+from lib.core.compat import xrange
 from lib.core.data import conf
 from lib.core.data import paths
 from lib.core.datatype import AttribDict
@@ -24,8 +25,8 @@ def cleanupVals(text, tag):
     if tag in ("clause", "where"):
         text = text.split(',')
 
-    if isinstance(text, basestring):
+    if hasattr(text, "isdigit") and text.isdigit():
-        text = int(text) if text.isdigit() else text
+        text = int(text)
 
     elif isinstance(text, list):
         count = 0

lib/parse/sitemap.py

@@ -1,19 +1,19 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """
 
-import httplib
 import re
 
 from lib.core.common import readInput
 from lib.core.data import kb
 from lib.core.data import logger
+from lib.core.datatype import OrderedSet
 from lib.core.exception import SqlmapSyntaxException
 from lib.request.connect import Connect as Request
-from thirdparty.oset.pyoset import oset
+from thirdparty.six.moves import http_client as _http_client
 
 abortedFlag = None
 
@@ -26,11 +26,11 @@ def parseSitemap(url, retVal=None):
     try:
         if retVal is None:
             abortedFlag = False
-            retVal = oset()
+            retVal = OrderedSet()
 
         try:
             content = Request.getPage(url=url, raise404=True)[0] if not abortedFlag else ""
-        except httplib.InvalidURL:
+        except _http_client.InvalidURL:
             errMsg = "invalid URL given for sitemap ('%s')" % url
             raise SqlmapSyntaxException(errMsg)
 

lib/request/__init__.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)

lib/request/basic.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
@@ -7,15 +7,16 @@ See the file 'LICENSE' for copying permission
 
 import codecs
 import gzip
+import io
 import logging
 import re
-import StringIO
 import struct
 import zlib
 
 from lib.core.common import Backend
 from lib.core.common import extractErrorMessage
 from lib.core.common import extractRegexResult
+from lib.core.common import filterNone
 from lib.core.common import getPublicTypeMembers
 from lib.core.common import getSafeExString
 from lib.core.common import getUnicode
@@ -47,7 +48,7 @@ from lib.parse.headers import headersParser
 from lib.parse.html import htmlParser
 from lib.utils.htmlentities import htmlEntities
 from thirdparty.chardet import detect
-from thirdparty.odict.odict import OrderedDict
+from thirdparty.odict import OrderedDict
 
 def forgeHeaders(items=None, base=None):
     """
@@ -100,7 +101,7 @@ def forgeHeaders(items=None, base=None):
 
                 if ("%s=" % getUnicode(cookie.name)) in getUnicode(headers[HTTP_HEADER.COOKIE]):
                     if conf.loadCookies:
-                        conf.httpHeaders = filter(None, ((item if item[0] != HTTP_HEADER.COOKIE else None) for item in conf.httpHeaders))
+                        conf.httpHeaders = filterNone((item if item[0] != HTTP_HEADER.COOKIE else None) for item in conf.httpHeaders)
                     elif kb.mergeCookies is None:
                         message = "you provided a HTTP %s header value. " % HTTP_HEADER.COOKIE
                         message += "The target URL provided its own cookies within "
@@ -257,12 +258,12 @@ def decodePage(page, contentEncoding, contentType):
     if not page or (conf.nullConnection and len(page) < 2):
         return getUnicode(page)
 
-    if isinstance(contentEncoding, basestring) and contentEncoding:
+    if hasattr(contentEncoding, "lower"):
         contentEncoding = contentEncoding.lower()
     else:
         contentEncoding = ""
 
-    if isinstance(contentType, basestring) and contentType:
+    if hasattr(contentType, "lower"):
         contentType = contentType.lower()
     else:
         contentType = ""
@@ -273,9 +274,9 @@ def decodePage(page, contentEncoding, contentType):
 
         try:
             if contentEncoding == "deflate":
-                data = StringIO.StringIO(zlib.decompress(page, -15))  # Reference: http://stackoverflow.com/questions/1089662/python-inflate-and-deflate-implementations
+                data = io.BytesIO(zlib.decompress(page, -15))  # Reference: http://stackoverflow.com/questions/1089662/python-inflate-and-deflate-implementations
             else:
-                data = gzip.GzipFile("", "rb", 9, StringIO.StringIO(page))
+                data = gzip.GzipFile("", "rb", 9, io.BytesIO(page))
                 size = struct.unpack("<l", page[-4:])[0]  # Reference: http://pydoc.org/get.cgi/usr/local/lib/python2.5/gzip.py
                 if size > MAX_CONNECTION_TOTAL_SIZE:
                     raise Exception("size too large")

lib/request/basicauthhandler.py

@@ -1,19 +1,19 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """
 
-import urllib2
+from thirdparty.six.moves import urllib as _urllib
 
-class SmartHTTPBasicAuthHandler(urllib2.HTTPBasicAuthHandler):
+class SmartHTTPBasicAuthHandler(_urllib.request.HTTPBasicAuthHandler):
     """
     Reference: http://selenic.com/hg/rev/6c51a5056020
     Fix for a: http://bugs.python.org/issue8797
     """
     def __init__(self, *args, **kwargs):
-        urllib2.HTTPBasicAuthHandler.__init__(self, *args, **kwargs)
+        _urllib.request.HTTPBasicAuthHandler.__init__(self, *args, **kwargs)
         self.retried_req = set()
         self.retried_count = 0
 
@@ -30,8 +30,8 @@ class SmartHTTPBasicAuthHandler(urllib2.HTTPBasicAuthHandler):
             self.retried_count = 0
         else:
             if self.retried_count > 5:
-                raise urllib2.HTTPError(req.get_full_url(), 401, "basic auth failed", headers, None)
+                raise _urllib.error.HTTPError(req.get_full_url(), 401, "basic auth failed", headers, None)
             else:
                 self.retried_count += 1
 
-        return urllib2.HTTPBasicAuthHandler.http_error_auth_reqed(self, auth_header, host, req, headers)
+        return _urllib.request.HTTPBasicAuthHandler.http_error_auth_reqed(self, auth_header, host, req, headers)

lib/request/chunkedhandler.py

@@ -0,0 +1,43 @@
+#!/usr/bin/env python2
+
+"""
+Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
+See the file 'LICENSE' for copying permission
+"""
+
+from lib.core.data import conf
+from thirdparty.six.moves import urllib as _urllib
+
+class ChunkedHandler(_urllib.request.HTTPHandler):
+    """
+    Ensures that HTTPHandler is working properly in case of Chunked Transfer-Encoding
+    """
+
+    def _http_request(self, request):
+        host = request.get_host()
+        if not host:
+            raise _urllib.error.URLError("no host given")
+
+        if request.has_data():  # POST
+            data = request.get_data()
+            if not request.has_header("Content-type"):
+                request.add_unredirected_header(
+                    "Content-type",
+                    "application/x-www-form-urlencoded")
+            if not request.has_header("Content-length") and not conf.chunked:
+                request.add_unredirected_header(
+                    "Content-length", "%d" % len(data))
+
+        sel_host = host
+        if request.has_proxy():
+            sel_host = _urllib.parse.urlsplit(request.get_selector()).netloc
+
+        if not request.has_header("Host"):
+            request.add_unredirected_header("Host", sel_host)
+        for name, value in self.parent.addheaders:
+            name = name.capitalize()
+            if not request.has_header(name):
+                request.add_unredirected_header(name, value)
+        return request
+
+    http_request = _http_request

lib/request/comparison.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)

lib/request/connect.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
@@ -6,9 +6,6 @@ See the file 'LICENSE' for copying permission
 """
 
 import binascii
-import compiler
-import httplib
-import keyword
 import logging
 import re
 import socket
@@ -16,9 +13,6 @@ import string
 import struct
 import time
 import traceback
-import urllib
-import urllib2
-import urlparse
 
 try:
     import websocket
@@ -32,11 +26,13 @@ from lib.core.agent import agent
 from lib.core.common import asciifyUrl
 from lib.core.common import calculateDeltaSeconds
 from lib.core.common import checkSameHost
+from lib.core.common import chunkSplitPostData
 from lib.core.common import clearConsoleLine
 from lib.core.common import dataToStdout
 from lib.core.common import escapeJsonValue
 from lib.core.common import evaluateCode
 from lib.core.common import extractRegexResult
+from lib.core.common import filterNone
 from lib.core.common import findMultipartPostBoundary
 from lib.core.common import getCurrentThreadData
 from lib.core.common import getHeader
@@ -62,6 +58,7 @@ from lib.core.common import unicodeencode
 from lib.core.common import unsafeVariableNaming
 from lib.core.common import urldecode
 from lib.core.common import urlencode
+from lib.core.compat import xrange
 from lib.core.data import conf
 from lib.core.data import kb
 from lib.core.data import logger
@@ -92,7 +89,7 @@ from lib.core.settings import DEFAULT_CONTENT_TYPE
 from lib.core.settings import DEFAULT_COOKIE_DELIMITER
 from lib.core.settings import DEFAULT_GET_POST_DELIMITER
 from lib.core.settings import DEFAULT_USER_AGENT
-from lib.core.settings import EVALCODE_KEYWORD_SUFFIX
+from lib.core.settings import EVALCODE_ENCODED_PREFIX
 from lib.core.settings import HTTP_ACCEPT_HEADER_VALUE
 from lib.core.settings import HTTP_ACCEPT_ENCODING_HEADER_VALUE
 from lib.core.settings import MAX_CONNECTION_CHUNK_SIZE
@@ -123,7 +120,10 @@ from lib.request.basic import processResponse
 from lib.request.direct import direct
 from lib.request.comparison import comparison
 from lib.request.methodrequest import MethodRequest
-from thirdparty.odict.odict import OrderedDict
+from thirdparty import six
+from thirdparty.odict import OrderedDict
+from thirdparty.six.moves import http_client as _http_client
+from thirdparty.six.moves import urllib as _urllib
 from thirdparty.socks.socks import ProxyError
 
 class Connect(object):
@@ -272,14 +272,19 @@ class Connect(object):
         checking = kwargs.get("checking", False)
         skipRead = kwargs.get("skipRead", False)
         finalCode = kwargs.get("finalCode", False)
+        chunked = kwargs.get("chunked", False) or conf.chunked
 
         if multipart:
             post = multipart
 
+        if chunked and post:
+            post = _urllib.parse.unquote(post)
+            post = chunkSplitPostData(post)
+
         websocket_ = url.lower().startswith("ws")
 
-        if not urlparse.urlsplit(url).netloc:
+        if not _urllib.parse.urlsplit(url).netloc:
-            url = urlparse.urljoin(conf.url, url)
+            url = _urllib.parse.urljoin(conf.url, url)
 
         # flag to know if we are dealing with the same target host
         target = checkSameHost(url, conf.url)
@@ -300,7 +305,7 @@ class Connect(object):
         code = None
         status = None
 
-        _ = urlparse.urlsplit(url)
+        _ = _urllib.parse.urlsplit(url)
         requestMsg = u"HTTP request [#%d]:\r\n%s " % (threadData.lastRequestUID, method or (HTTPMETHOD.POST if post is not None else HTTPMETHOD.GET))
         requestMsg += getUnicode(("%s%s" % (_.path or "/", ("?%s" % _.query) if _.query else "")) if not any((refreshing, crawling, checking)) else url)
         responseMsg = u"HTTP response "
@@ -328,7 +333,7 @@ class Connect(object):
                 pass
 
             elif target:
-                if conf.forceSSL and urlparse.urlparse(url).scheme != "https":
+                if conf.forceSSL and _urllib.parse.urlparse(url).scheme != "https":
                     url = re.sub(r"(?i)\Ahttp:", "https:", url)
                     url = re.sub(r"(?i):80/", ":443/", url)
 
@@ -353,7 +358,7 @@ class Connect(object):
                 url = "%s?%s" % (url, get)
                 requestMsg += "?%s" % get
 
-            requestMsg += " %s" % httplib.HTTPConnection._http_vsn_str
+            requestMsg += " %s" % _http_client.HTTPConnection._http_vsn_str
 
             # Prepare HTTP headers
             headers = forgeHeaders({HTTP_HEADER.COOKIE: cookie, HTTP_HEADER.USER_AGENT: ua, HTTP_HEADER.REFERER: referer, HTTP_HEADER.HOST: host}, base=None if target else {})
@@ -398,6 +403,9 @@ class Connect(object):
             if conf.keepAlive:
                 headers[HTTP_HEADER.CONNECTION] = "keep-alive"
             
+            if chunked:
+                headers[HTTP_HEADER.TRANSFER_ENCODING] = "chunked"
+
             if auxHeaders:
                 headers = forgeHeaders(auxHeaders, headers)
 
@@ -419,7 +427,7 @@ class Connect(object):
                 page = ws.recv()
                 ws.close()
                 code = ws.status
-                status = httplib.responses[code]
+                status = _http_client.responses[code]
 
                 class _(dict):
                     pass
@@ -427,7 +435,7 @@ class Connect(object):
                 responseHeaders = _(ws.getheaders())
                 responseHeaders.headers = ["%s: %s\r\n" % (_[0].capitalize(), _[1]) for _ in responseHeaders.items()]
 
-                requestHeaders += "\r\n".join(["%s: %s" % (getUnicode(key.capitalize() if isinstance(key, basestring) else key), getUnicode(value)) for (key, value) in responseHeaders.items()])
+                requestHeaders += "\r\n".join(["%s: %s" % (getUnicode(key.capitalize() if hasattr(key, "capitalize") else key), getUnicode(value)) for (key, value) in responseHeaders.items()])
                 requestMsg += "\r\n%s" % requestHeaders
 
                 if post is not None:
@@ -444,11 +452,11 @@ class Connect(object):
                     req = MethodRequest(url, post, headers)
                     req.set_method(method)
                 elif url is not None:
-                    req = urllib2.Request(url, post, headers)
+                    req = _urllib.request.Request(url, post, headers)
                 else:
                     return None, None, None
 
-                requestHeaders += "\r\n".join(["%s: %s" % (getUnicode(key.capitalize() if isinstance(key, basestring) else key), getUnicode(value)) for (key, value) in req.header_items()])
+                requestHeaders += "\r\n".join(["%s: %s" % (getUnicode(key.capitalize() if hasattr(key, "capitalize") else key), getUnicode(value)) for (key, value) in req.header_items()])
 
                 if not getRequestHeader(req, HTTP_HEADER.COOKIE) and conf.cj:
                     conf.cj._policy._now = conf.cj._now = int(time.time())
@@ -456,7 +464,7 @@ class Connect(object):
                     requestHeaders += "\r\n%s" % ("Cookie: %s" % ";".join("%s=%s" % (getUnicode(cookie.name), getUnicode(cookie.value)) for cookie in cookies))
 
                 if post is not None:
-                    if not getRequestHeader(req, HTTP_HEADER.CONTENT_LENGTH):
+                    if not getRequestHeader(req, HTTP_HEADER.CONTENT_LENGTH) and not chunked:
                         requestHeaders += "\r\n%s: %d" % (string.capwords(HTTP_HEADER.CONTENT_LENGTH), len(post))
 
                 if not getRequestHeader(req, HTTP_HEADER.CONNECTION):
@@ -467,7 +475,8 @@ class Connect(object):
                 if post is not None:
                     requestMsg += "\r\n\r\n%s" % getUnicode(post)
 
-                requestMsg += "\r\n"
+                if not chunked:
+                    requestMsg += "\r\n"
 
                 if not multipart:
                     threadData.lastRequestMsg = requestMsg
@@ -482,7 +491,7 @@ class Connect(object):
                             for char in (r"\r", r"\n"):
                                 cookie.value = re.sub(r"(%s)([^ \t])" % char, r"\g<1>\t\g<2>", cookie.value)
 
-                conn = urllib2.urlopen(req)
+                conn = _urllib.request.urlopen(req)
 
                 if not kb.authHeader and getRequestHeader(req, HTTP_HEADER.AUTHORIZATION) and (conf.authType or "").lower() == AUTH_TYPE.BASIC.lower():
                     kb.authHeader = getRequestHeader(req, HTTP_HEADER.AUTHORIZATION)
@@ -538,7 +547,7 @@ class Connect(object):
                         if re.search(r"\Ahttps?://", refresh, re.I):
                             url = refresh
                         else:
-                            url = urlparse.urljoin(url, refresh)
+                            url = _urllib.parse.urljoin(url, refresh)
 
                         threadData.lastRedirectMsg = (threadData.lastRequestUID, page)
                         kwargs["refreshing"] = True
@@ -570,7 +579,7 @@ class Connect(object):
             else:
                 raise
 
-        except urllib2.HTTPError as ex:
+        except _urllib.error.HTTPError as ex:
             page = None
             responseHeaders = None
 
@@ -592,7 +601,7 @@ class Connect(object):
             except:
                 pass
             finally:
-                page = page if isinstance(page, unicode) else getUnicode(page)
+                page = getUnicode(page)
 
             code = ex.code
             status = getSafeExString(ex)
@@ -604,7 +613,7 @@ class Connect(object):
             responseMsg += "[#%d] (%s %s):\r\n" % (threadData.lastRequestUID, code, status)
 
             if responseHeaders:
-                logHeaders = "\r\n".join(["%s: %s" % (getUnicode(key.capitalize() if isinstance(key, basestring) else key), getUnicode(value)) for (key, value) in responseHeaders.items()])
+                logHeaders = getUnicode("".join(responseHeaders.headers).strip())
 
             logHTTPTraffic(requestMsg, "%s%s\r\n\r\n%s" % (responseMsg, logHeaders, (page or "")[:MAX_CONNECTION_CHUNK_SIZE]), start, time.time())
 
@@ -619,22 +628,22 @@ class Connect(object):
                 logger.log(CUSTOM_LOGGING.TRAFFIC_IN, responseMsg)
 
             if ex.code != conf.ignoreCode:
-                if ex.code == httplib.UNAUTHORIZED:
+                if ex.code == _http_client.UNAUTHORIZED:
                     errMsg = "not authorized, try to provide right HTTP "
                     errMsg += "authentication type and valid credentials (%d)" % code
                     raise SqlmapConnectionException(errMsg)
-                elif ex.code == httplib.NOT_FOUND:
+                elif ex.code == _http_client.NOT_FOUND:
                     if raise404:
                         errMsg = "page not found (%d)" % code
                         raise SqlmapConnectionException(errMsg)
                     else:
                         debugMsg = "page not found (%d)" % code
                         singleTimeLogMessage(debugMsg, logging.DEBUG)
-                elif ex.code == httplib.GATEWAY_TIMEOUT:
+                elif ex.code == _http_client.GATEWAY_TIMEOUT:
                     if ignoreTimeout:
                         return None if not conf.ignoreTimeouts else "", None, None
                     else:
-                        warnMsg = "unable to connect to the target URL (%d - %s)" % (ex.code, httplib.responses[ex.code])
+                        warnMsg = "unable to connect to the target URL (%d - %s)" % (ex.code, _http_client.responses[ex.code])
                         if threadData.retriesCount < conf.retries and not kb.threadException:
                             warnMsg += ". sqlmap is going to retry the request"
                             logger.critical(warnMsg)
@@ -648,7 +657,7 @@ class Connect(object):
                     debugMsg = "got HTTP error code: %d (%s)" % (code, status)
                     logger.debug(debugMsg)
 
-        except (urllib2.URLError, socket.error, socket.timeout, httplib.HTTPException, struct.error, binascii.Error, ProxyError, SqlmapCompressionException, WebSocketException, TypeError, ValueError, OverflowError):
+        except (_urllib.error.URLError, socket.error, socket.timeout, _http_client.HTTPException, struct.error, binascii.Error, ProxyError, SqlmapCompressionException, WebSocketException, TypeError, ValueError, OverflowError):
             tbMsg = traceback.format_exc()
 
             if checking:
@@ -739,17 +748,29 @@ class Connect(object):
                 raise SqlmapConnectionException(warnMsg)
 
         finally:
-            if isinstance(page, basestring) and not isinstance(page, unicode):
+            if isinstance(page, six.binary_type):
                 if HTTP_HEADER.CONTENT_TYPE in (responseHeaders or {}) and not re.search(TEXT_CONTENT_TYPE_REGEX, responseHeaders[HTTP_HEADER.CONTENT_TYPE]):
-                    page = unicode(page, errors="ignore")
+                    page = six.text_type(page, errors="ignore")
                 else:
                     page = getUnicode(page)
+
+            for function in kb.preprocessFunctions:
+                try:
+                    page, responseHeaders, code = function(page, responseHeaders, code)
+                except Exception as ex:
+                    errMsg = "error occurred while running preprocess "
+                    errMsg += "function '%s' ('%s')" % (function.__name__, getSafeExString(ex))
+                    raise SqlmapGenericException(errMsg)
+
+            threadData.lastPage = page
+            threadData.lastCode = code
+
             socket.setdefaulttimeout(conf.timeout)
 
         processResponse(page, responseHeaders, status)
 
         if conn and getattr(conn, "redurl", None):
-            _ = urlparse.urlsplit(conn.redurl)
+            _ = _urllib.parse.urlsplit(conn.redurl)
             _ = ("%s%s" % (_.path or "/", ("?%s" % _.query) if _.query else ""))
             requestMsg = re.sub(r"(\n[A-Z]+ ).+?( HTTP/\d)", r"\g<1>%s\g<2>" % getUnicode(_).replace("\\", "\\\\"), requestMsg, 1)
 
@@ -758,12 +779,12 @@ class Connect(object):
                 requestMsg = re.sub(r"(?i)Content-length: \d+\n", "", requestMsg)
                 requestMsg = re.sub(r"(?s)\n\n.+", "\n", requestMsg)
 
-            responseMsg += "[#%d] (%d %s):\r\n" % (threadData.lastRequestUID, conn.code, status)
+            responseMsg += "[#%d] (%s %s):\r\n" % (threadData.lastRequestUID, conn.code, status)
         else:
             responseMsg += "[#%d] (%s %s):\r\n" % (threadData.lastRequestUID, code, status)
 
         if responseHeaders:
-            logHeaders = "\r\n".join(["%s: %s" % (getUnicode(key.capitalize() if isinstance(key, basestring) else key), getUnicode(value)) for (key, value) in responseHeaders.items()])
+            logHeaders = getUnicode("".join(responseHeaders.headers).strip())
 
         if not skipLogTraffic:
             logHTTPTraffic(requestMsg, "%s%s\r\n\r\n%s" % (responseMsg, logHeaders, (page or "")[:MAX_CONNECTION_CHUNK_SIZE]), start, time.time())
@@ -837,11 +858,11 @@ class Connect(object):
                         payload = function(payload=payload, headers=auxHeaders, delimiter=delimiter, hints=hints)
                     except Exception as ex:
                         errMsg = "error occurred while running tamper "
-                        errMsg += "function '%s' ('%s')" % (function.func_name, getSafeExString(ex))
+                        errMsg += "function '%s' ('%s')" % (function.__name__, getSafeExString(ex))
                         raise SqlmapGenericException(errMsg)
 
-                    if not isinstance(payload, basestring):
+                    if not isinstance(payload, six.string_types):
-                        errMsg = "tamper function '%s' returns " % function.func_name
+                        errMsg = "tamper function '%s' returns " % function.__name__
                         errMsg += "invalid payload type ('%s')" % type(payload)
                         raise SqlmapValueException(errMsg)
 
@@ -965,6 +986,10 @@ class Connect(object):
         if conf.csrfToken:
             def _adjustParameter(paramString, parameter, newValue):
                 retVal = paramString
+
+                if urlencode(parameter) in paramString:
+                    parameter = urlencode(parameter)
+
                 match = re.search(r"%s=[^&]*" % re.escape(parameter), paramString, re.I)
                 if match:
                     retVal = re.sub("(?i)%s" % re.escape(match.group(0)), ("%s=%s" % (parameter, newValue)).replace('\\', r'\\'), paramString)
@@ -972,10 +997,13 @@ class Connect(object):
                     match = re.search(r"(%s[\"']:[\"'])([^\"']+)" % re.escape(parameter), paramString, re.I)
                     if match:
                         retVal = re.sub("(?i)%s" % re.escape(match.group(0)), "%s%s" % (match.group(1), newValue), paramString)
+
                 return retVal
 
             token = AttribDict()
             page, headers, code = Connect.getPage(url=conf.csrfUrl or conf.url, data=conf.data if conf.csrfUrl == conf.url else None, method=conf.method if conf.csrfUrl == conf.url else None, cookie=conf.parameters.get(PLACE.COOKIE), direct=True, silent=True, ua=conf.parameters.get(PLACE.USER_AGENT), referer=conf.parameters.get(PLACE.REFERER), host=conf.parameters.get(PLACE.HOST))
+            page = urldecode(page)  # for anti-CSRF tokens with special characters in their name (e.g. 'foo:bar=...')
+
             match = re.search(r"(?i)<input[^>]+\bname=[\"']?(?P<name>%s)\b[^>]*\bvalue=[\"']?(?P<value>[^>'\"]*)" % conf.csrfToken, page or "", re.I)
 
             if not match:
@@ -998,7 +1026,7 @@ class Connect(object):
                     token.value = "".join(chr(int(_)) for _ in match.group(1).replace(' ', "").split(','))
 
             if not token:
-                if conf.csrfUrl and conf.csrfToken and conf.csrfUrl != conf.url and code == httplib.OK:
+                if conf.csrfUrl and conf.csrfToken and conf.csrfUrl != conf.url and code == _http_client.OK:
                     if headers and "text/plain" in headers.get(HTTP_HEADER.CONTENT_TYPE, ""):
                         token.name = conf.csrfToken
                         token.value = page
@@ -1062,14 +1090,13 @@ class Connect(object):
             delimiter = conf.paramDel or DEFAULT_GET_POST_DELIMITER
             variables = {"uri": uri, "lastPage": threadData.lastPage, "_locals": locals()}
             originals = {}
-            keywords = keyword.kwlist
 
             if not get and PLACE.URI in conf.parameters:
-                query = urlparse.urlsplit(uri).query or ""
+                query = _urllib.parse.urlsplit(uri).query or ""
             else:
                 query = None
 
-            for item in filter(None, (get, post if not kb.postHint else None, query)):
+            for item in filterNone((get, post if not kb.postHint else None, query)):
                 for part in item.split(delimiter):
                     if '=' in part:
                         name, value = part.split('=', 1)
@@ -1077,8 +1104,6 @@ class Connect(object):
                         if safeVariableNaming(name) != name:
                             conf.evalCode = re.sub(r"\b%s\b" % re.escape(name), safeVariableNaming(name), conf.evalCode)
                             name = safeVariableNaming(name)
-                        elif name in keywords:
-                            name = "%s%s" % (name, EVALCODE_KEYWORD_SUFFIX)
                         value = urldecode(value, convall=True, spaceplus=(item == post and kb.postSpaceToPlus))
                         variables[name] = value
 
@@ -1090,31 +1115,29 @@ class Connect(object):
                         if safeVariableNaming(name) != name:
                             conf.evalCode = re.sub(r"\b%s\b" % re.escape(name), safeVariableNaming(name), conf.evalCode)
                             name = safeVariableNaming(name)
-                        elif name in keywords:
-                            name = "%s%s" % (name, EVALCODE_KEYWORD_SUFFIX)
                         value = urldecode(value, convall=True)
                         variables[name] = value
 
             while True:
                 try:
-                    compiler.parse(unicodeencode(conf.evalCode.replace(';', '\n')))
+                    compile(unicodeencode(conf.evalCode.replace(';', '\n')), "", "exec")
                 except SyntaxError as ex:
                     if ex.text:
                         original = replacement = ex.text.strip()
+
                         if '=' in original:
                             name, value = original.split('=', 1)
                             name = name.strip()
                             if safeVariableNaming(name) != name:
                                 replacement = re.sub(r"\b%s\b" % re.escape(name), safeVariableNaming(name), replacement)
-                            elif name in keywords:
-                                replacement = re.sub(r"\b%s\b" % re.escape(name), "%s%s" % (name, EVALCODE_KEYWORD_SUFFIX), replacement)
                         else:
                             for _ in re.findall(r"[A-Za-z_]+", original)[::-1]:
-                                if _ in keywords:
+                                if safeVariableNaming(_) != _:
-                                    replacement = replacement.replace(_, "%s%s" % (_, EVALCODE_KEYWORD_SUFFIX))
+                                    replacement = replacement.replace(_, safeVariableNaming(_))
                                     break
+
                         if original == replacement:
-                            conf.evalCode = conf.evalCode.replace(EVALCODE_KEYWORD_SUFFIX, "")
+                            conf.evalCode = conf.evalCode.replace(EVALCODE_ENCODED_PREFIX, "")
                             break
                         else:
                             conf.evalCode = conf.evalCode.replace(getUnicode(ex.text.strip(), UNICODE_ENCODING), replacement)
@@ -1127,11 +1150,6 @@ class Connect(object):
             evaluateCode(conf.evalCode, variables)
 
             for variable in list(variables.keys()):
-                if variable.endswith(EVALCODE_KEYWORD_SUFFIX):
-                    value = variables[variable]
-                    del variables[variable]
-                    variables[variable.replace(EVALCODE_KEYWORD_SUFFIX, "")] = value
-
                 if unsafeVariableNaming(variable) != variable:
                     value = variables[variable]
                     del variables[variable]
@@ -1141,7 +1159,7 @@ class Connect(object):
 
             for name, value in variables.items():
                 if name != "__builtins__" and originals.get(name, "") != value:
-                    if isinstance(value, (basestring, int)):
+                    if isinstance(value, (int, six.string_types)):
                         found = False
                         value = getUnicode(value, UNICODE_ENCODING)
 
@@ -1284,7 +1302,7 @@ class Connect(object):
 
         if conf.secondUrl:
             page, headers, code = Connect.getPage(url=conf.secondUrl, cookie=cookie, ua=ua, silent=silent, auxHeaders=auxHeaders, response=response, raise404=False, ignoreTimeout=timeBasedCompare, refreshing=True)
-        elif kb.secondReq and IPS_WAF_CHECK_PAYLOAD not in urllib.unquote(value or ""):
+        elif kb.secondReq and IPS_WAF_CHECK_PAYLOAD not in _urllib.parse.unquote(value or ""):
             def _(value):
                 if kb.customInjectionMark in (value or ""):
                     if payload is None:
@@ -1295,10 +1313,9 @@ class Connect(object):
             page, headers, code = Connect.getPage(url=_(kb.secondReq[0]), post=_(kb.secondReq[2]), method=kb.secondReq[1], cookie=kb.secondReq[3], silent=silent, auxHeaders=dict(auxHeaders, **dict(kb.secondReq[4])), response=response, raise404=False, ignoreTimeout=timeBasedCompare, refreshing=True)
 
         threadData.lastQueryDuration = calculateDeltaSeconds(start)
-        threadData.lastPage = page
-        threadData.lastCode = code
 
-        kb.originalCode = kb.originalCode or code
+        kb.originalCode = code if kb.originalCode is None else kb.originalCode
+        kb.originalPage = page if kb.originalPage is None else kb.originalPage
 
         if kb.testMode:
             kb.testQueryCount += 1

lib/request/direct.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)

lib/request/dns.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)

lib/request/httpshandler.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
@@ -6,17 +6,18 @@ See the file 'LICENSE' for copying permission
 """
 
 import distutils.version
-import httplib
 import re
 import socket
-import urllib2
 
+from lib.core.common import filterNone
 from lib.core.common import getSafeExString
 from lib.core.data import conf
 from lib.core.data import kb
 from lib.core.data import logger
 from lib.core.exception import SqlmapConnectionException
 from lib.core.settings import PYVERSION
+from thirdparty.six.moves import http_client as _http_client
+from thirdparty.six.moves import urllib as _urllib
 
 ssl = None
 try:
@@ -25,9 +26,9 @@ try:
 except ImportError:
     pass
 
-_protocols = filter(None, (getattr(ssl, _, None) for _ in ("PROTOCOL_TLSv1_2", "PROTOCOL_TLSv1_1", "PROTOCOL_TLSv1", "PROTOCOL_SSLv3", "PROTOCOL_SSLv23", "PROTOCOL_SSLv2")))
+_protocols = filterNone(getattr(ssl, _, None) for _ in ("PROTOCOL_TLSv1_2", "PROTOCOL_TLSv1_1", "PROTOCOL_TLSv1", "PROTOCOL_SSLv3", "PROTOCOL_SSLv23", "PROTOCOL_SSLv2"))
 
-class HTTPSConnection(httplib.HTTPSConnection):
+class HTTPSConnection(_http_client.HTTPSConnection):
     """
     Connection class that enables usage of newer SSL protocols.
 
@@ -35,7 +36,7 @@ class HTTPSConnection(httplib.HTTPSConnection):
     """
 
     def __init__(self, *args, **kwargs):
-        httplib.HTTPSConnection.__init__(self, *args, **kwargs)
+        _http_client.HTTPSConnection.__init__(self, *args, **kwargs)
 
     def connect(self):
         def create_sock():
@@ -63,7 +64,7 @@ class HTTPSConnection(httplib.HTTPSConnection):
                         break
                     else:
                         sock.close()
-                except (ssl.SSLError, socket.error, httplib.BadStatusLine) as ex:
+                except (ssl.SSLError, socket.error, _http_client.BadStatusLine) as ex:
                     self._tunnel_host = None
                     logger.debug("SSL connection error occurred ('%s')" % getSafeExString(ex))
 
@@ -83,7 +84,7 @@ class HTTPSConnection(httplib.HTTPSConnection):
                         break
                     else:
                         sock.close()
-                except (ssl.SSLError, socket.error, httplib.BadStatusLine) as ex:
+                except (ssl.SSLError, socket.error, _http_client.BadStatusLine) as ex:
                     self._tunnel_host = None
                     logger.debug("SSL connection error occurred ('%s')" % getSafeExString(ex))
 
@@ -94,14 +95,6 @@ class HTTPSConnection(httplib.HTTPSConnection):
                 errMsg += " (please retry with Python >= 2.7.9)"
             raise SqlmapConnectionException(errMsg)
 
-class HTTPSHandler(urllib2.HTTPSHandler):
+class HTTPSHandler(_urllib.request.HTTPSHandler):
     def https_open(self, req):
-        return self.do_open(HTTPSConnection if ssl else httplib.HTTPSConnection, req)
+        return self.do_open(HTTPSConnection if ssl else _http_client.HTTPSConnection, req)
-
-# Bug fix (http://bugs.python.org/issue17849)
-
-def _(self, *args):
-    return self._readline()
-
-httplib.LineAndFileWrapper._readline = httplib.LineAndFileWrapper.readline
-httplib.LineAndFileWrapper.readline = _

lib/request/inject.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
@@ -17,6 +17,7 @@ from lib.core.common import calculateDeltaSeconds
 from lib.core.common import cleanQuery
 from lib.core.common import expandAsteriskForColumns
 from lib.core.common import extractExpectedValue
+from lib.core.common import filterNone
 from lib.core.common import getPublicTypeMembers
 from lib.core.common import getTechniqueData
 from lib.core.common import hashDBRetrieve
@@ -31,6 +32,7 @@ from lib.core.common import pushValue
 from lib.core.common import randomStr
 from lib.core.common import readInput
 from lib.core.common import singleTimeWarnMessage
+from lib.core.compat import xrange
 from lib.core.data import conf
 from lib.core.data import kb
 from lib.core.data import logger
@@ -59,6 +61,7 @@ from lib.techniques.dns.test import dnsTest
 from lib.techniques.dns.use import dnsUse
 from lib.techniques.error.use import errorUse
 from lib.techniques.union.use import unionUse
+from thirdparty import six
 
 def _goDns(payload, expression):
     value = None
@@ -334,7 +337,7 @@ def _goUnion(expression, unpack=True, dump=False):
 
     output = unionUse(expression, unpack=unpack, dump=dump)
 
-    if isinstance(output, basestring):
+    if isinstance(output, six.string_types):
         output = parseUnionPage(output)
 
     return output
@@ -429,7 +432,7 @@ def getValue(expression, blind=True, union=True, error=True, time=True, fromUser
                     found = (value is not None) or (value is None and expectingNone) or count >= MAX_TECHNIQUES_PER_VALUE
 
                 if found and conf.dnsDomain:
-                    _ = "".join(filter(None, (key if isTechniqueAvailable(value) else None for key, value in {'E': PAYLOAD.TECHNIQUE.ERROR, 'Q': PAYLOAD.TECHNIQUE.QUERY, 'U': PAYLOAD.TECHNIQUE.UNION}.items())))
+                    _ = "".join(filterNone(key if isTechniqueAvailable(value) else None for key, value in {'E': PAYLOAD.TECHNIQUE.ERROR, 'Q': PAYLOAD.TECHNIQUE.QUERY, 'U': PAYLOAD.TECHNIQUE.UNION}.items()))
                     warnMsg = "option '--dns-domain' will be ignored "
                     warnMsg += "as faster techniques are usable "
                     warnMsg += "(%s) " % _

lib/request/methodrequest.py

@@ -1,19 +1,19 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """
 
-import urllib2
+from thirdparty.six.moves import urllib as _urllib
 
-class MethodRequest(urllib2.Request):
+class MethodRequest(_urllib.request.Request):
     """
-    Used to create HEAD/PUT/DELETE/... requests with urllib2
+    Used to create HEAD/PUT/DELETE/... requests with urllib
     """
 
     def set_method(self, method):
         self.method = method.upper()
 
     def get_method(self):
-        return getattr(self, 'method', urllib2.Request.get_method(self))
+        return getattr(self, 'method', _urllib.request.Request.get_method(self))

lib/request/pkihandler.py

@@ -1,20 +1,19 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """
 
-import httplib
-import urllib2
-
 from lib.core.data import conf
 from lib.core.common import getSafeExString
 from lib.core.exception import SqlmapConnectionException
+from thirdparty.six.moves import http_client as _http_client
+from thirdparty.six.moves import urllib as _urllib
 
-class HTTPSPKIAuthHandler(urllib2.HTTPSHandler):
+class HTTPSPKIAuthHandler(_urllib.request.HTTPSHandler):
     def __init__(self, auth_file):
-        urllib2.HTTPSHandler.__init__(self)
+        _urllib.request.HTTPSHandler.__init__(self)
         self.auth_file = auth_file
 
     def https_open(self, req):
@@ -23,7 +22,7 @@ class HTTPSPKIAuthHandler(urllib2.HTTPSHandler):
     def getConnection(self, host, timeout=None):
         try:
             # Reference: https://docs.python.org/2/library/ssl.html#ssl.SSLContext.load_cert_chain
-            return httplib.HTTPSConnection(host, cert_file=self.auth_file, key_file=self.auth_file, timeout=conf.timeout)
+            return _http_client.HTTPSConnection(host, cert_file=self.auth_file, key_file=self.auth_file, timeout=conf.timeout)
         except IOError as ex:
             errMsg = "error occurred while using key "
             errMsg += "file '%s' ('%s')" % (self.auth_file, getSafeExString(ex))

lib/request/rangehandler.py

@@ -1,45 +1,23 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """
 
-import urllib
-import urllib2
-
 from lib.core.exception import SqlmapConnectionException
+from thirdparty.six.moves import urllib as _urllib
 
-class HTTPRangeHandler(urllib2.BaseHandler):
+class HTTPRangeHandler(_urllib.request.BaseHandler):
     """
     Handler that enables HTTP Range headers.
 
     Reference: http://stackoverflow.com/questions/1971240/python-seek-on-remote-file
-
-    This was extremely simple. The Range header is a HTTP feature to
-    begin with so all this class does is tell urllib2 that the
-    "206 Partial Content" response from the HTTP server is what we
-    expected.
-
-    Example:
-        import urllib2
-        import byterange
-
-        range_handler = range.HTTPRangeHandler()
-        opener = urllib2.build_opener(range_handler)
-
-        # install it
-        urllib2.install_opener(opener)
-
-        # create Request and set Range header
-        req = urllib2.Request('https://www.python.org/')
-        req.header['Range'] = 'bytes=30-50'
-        f = urllib2.urlopen(req)
     """
 
     def http_error_206(self, req, fp, code, msg, hdrs):
         # 206 Partial Content Response
-        r = urllib.addinfourl(fp, hdrs, req.get_full_url())
+        r = _urllib.response.addinfourl(fp, hdrs, req.get_full_url())
         r.code = code
         r.msg = msg
         return r

lib/request/redirecthandler.py

@@ -1,16 +1,13 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """
 
+import io
 import time
 import types
-import urllib2
-import urlparse
-
-from StringIO import StringIO
 
 from lib.core.data import conf
 from lib.core.data import kb
@@ -33,8 +30,9 @@ from lib.core.settings import MAX_TOTAL_REDIRECTIONS
 from lib.core.threads import getCurrentThreadData
 from lib.request.basic import decodePage
 from lib.request.basic import parseResponse
+from thirdparty.six.moves import urllib as _urllib
 
-class SmartRedirectHandler(urllib2.HTTPRedirectHandler):
+class SmartRedirectHandler(_urllib.request.HTTPRedirectHandler):
     def _get_header_redirect(self, headers):
         retVal = None
 
@@ -67,7 +65,7 @@ class SmartRedirectHandler(urllib2.HTTPRedirectHandler):
 
     def _redirect_request(self, req, fp, code, msg, headers, newurl):
         newurl = newurl.replace(' ', '%20')
-        return urllib2.Request(newurl, data=req.data, headers=req.headers, origin_req_host=req.get_origin_req_host())
+        return _urllib.request.Request(newurl, data=req.data, headers=req.headers, origin_req_host=req.get_origin_req_host())
 
     def http_error_302(self, req, fp, code, msg, headers):
         start = time.time()
@@ -97,7 +95,7 @@ class SmartRedirectHandler(urllib2.HTTPRedirectHandler):
         redirectMsg += "[#%d] (%d %s):\r\n" % (threadData.lastRequestUID, code, getUnicode(msg))
 
         if headers:
-            logHeaders = "\r\n".join("%s: %s" % (getUnicode(key.capitalize() if isinstance(key, basestring) else key), getUnicode(value)) for (key, value) in headers.items())
+            logHeaders = "\r\n".join("%s: %s" % (getUnicode(key.capitalize() if hasattr(key, "capitalize") else key), getUnicode(value)) for (key, value) in headers.items())
         else:
             logHeaders = ""
 
@@ -110,8 +108,8 @@ class SmartRedirectHandler(urllib2.HTTPRedirectHandler):
 
         if redurl:
             try:
-                if not urlparse.urlsplit(redurl).netloc:
+                if not _urllib.parse.urlsplit(redurl).netloc:
-                    redurl = urlparse.urljoin(req.get_full_url(), redurl)
+                    redurl = _urllib.parse.urljoin(req.get_full_url(), redurl)
 
                 self._infinite_loop_check(req)
                 self._ask_redirect_choice(code, redurl, req.get_method())
@@ -140,8 +138,8 @@ class SmartRedirectHandler(urllib2.HTTPRedirectHandler):
                 req.headers[HTTP_HEADER.COOKIE] = delimiter.join("%s=%s" % (key, cookies[key]) for key in cookies)
 
             try:
-                result = urllib2.HTTPRedirectHandler.http_error_302(self, req, fp, code, msg, headers)
+                result = _urllib.request.HTTPRedirectHandler.http_error_302(self, req, fp, code, msg, headers)
-            except urllib2.HTTPError as ex:
+            except _urllib.error.HTTPError as ex:
                 result = ex
 
                 # Dirty hack for http://bugs.python.org/issue15701
@@ -165,7 +163,7 @@ class SmartRedirectHandler(urllib2.HTTPRedirectHandler):
             except:
                 redurl = None
                 result = fp
-                fp.read = StringIO("").read
+                fp.read = io.BytesIO("").read
         else:
             result = fp
 

lib/request/templates.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)

lib/takeover/__init__.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)

lib/takeover/abstraction.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 """
 Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
@@ -44,7 +44,10 @@ class Abstraction(Web, UDF, XP_cmdshell):
         XP_cmdshell.__init__(self)
 
     def execCmd(self, cmd, silent=False):
-        if self.webBackdoorUrl and not isStackingAvailable():
+        if Backend.isDbms(DBMS.PGSQL) and self.checkCopyExec():
+            self.copyExecCmd(cmd)
+
+        elif self.webBackdoorUrl and not isStackingAvailable():
             self.webBackdoorRunCmd(cmd)
 
         elif Backend.getIdentifiedDbms() in (DBMS.MYSQL, DBMS.PGSQL):
@@ -60,7 +63,10 @@ class Abstraction(Web, UDF, XP_cmdshell):
     def evalCmd(self, cmd, first=None, last=None):
         retVal = None
 
-        if self.webBackdoorUrl and not isStackingAvailable():
+        if Backend.isDbms(DBMS.PGSQL) and self.checkCopyExec():
+            retVal = self.copyExecCmd(cmd)
+
+        elif self.webBackdoorUrl and not isStackingAvailable():
             retVal = self.webBackdoorRunCmd(cmd)
 
         elif Backend.getIdentifiedDbms() in (DBMS.MYSQL, DBMS.PGSQL):
@@ -103,14 +109,19 @@ class Abstraction(Web, UDF, XP_cmdshell):
             logger.info(infoMsg)
 
         else:
-            if Backend.getIdentifiedDbms() in (DBMS.MYSQL, DBMS.PGSQL):
+            if Backend.isDbms(DBMS.PGSQL) and self.checkCopyExec():
-                infoMsg = "going to use injected sys_eval and sys_exec "
+                infoMsg = "going to use 'COPY ... FROM PROGRAM ...' "
-                infoMsg += "user-defined functions for operating system "
+                infoMsg += "command execution"
+                logger.info(infoMsg)
+
+            elif Backend.getIdentifiedDbms() in (DBMS.MYSQL, DBMS.PGSQL):
+                infoMsg = "going to use injected user-defined functions "
+                infoMsg += "'sys_eval' and 'sys_exec' for operating system "
                 infoMsg += "command execution"
                 logger.info(infoMsg)
 
             elif Backend.isDbms(DBMS.MSSQL):
-                infoMsg = "going to use xp_cmdshell extended procedure for "
+                infoMsg = "going to use extended procedure 'xp_cmdshell' for "
                 infoMsg += "operating system command execution"
                 logger.info(infoMsg)
 
@@ -200,7 +211,9 @@ class Abstraction(Web, UDF, XP_cmdshell):
 
                 logger.warn(warnMsg)
 
-            if Backend.getIdentifiedDbms() in (DBMS.MYSQL, DBMS.PGSQL):
+            if any((conf.osCmd, conf.osShell)) and Backend.isDbms(DBMS.PGSQL) and self.checkCopyExec():
+                success = True
+            elif Backend.getIdentifiedDbms() in (DBMS.MYSQL, DBMS.PGSQL):
                 success = self.udfInjectSys()
 
                 if success is not True: