NationalSecurityAgency/PACE-python
1
0 Fork 0
mirror of https://github.com/NationalSecurityAgency/PACE-python.git synced 2026-11-07 06:05:58 -10:00
Code Issues Packages Projects Releases Wiki Activity

Initial commit

Browse Source
This commit is contained in:
Scott Ruoti committed 2016-11-01 13:24:49 -04:00
commit e82995d032
174 files changed
+83193
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

.gitignore
+10
View File
@@ -0,0 +1,10 @@
# global IDE, Maven, Python, ignores
target/
.project
.classpath
.settings
.pydevproject
*.pyc
.idea
*.iml
LICENSE.txt
+10
View File
@@ -0,0 +1,10 @@
Copyright (c) 2016, MIT Lincoln Laboratory
All rights reserved.
Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
README.md
+295
View File
@@ -0,0 +1,295 @@
DISTRIBUTION STATEMENT A. Approved for public release: distribution unlimited.
This material is based upon work supported by the Department of Defense under Air Force Contract
No. FA8721-05-C-0002 and/or FA8702-15-D-0001. Any opinions, findings, conclusions or
recommendations expressed in this material are those of the author(s) and do not
necessarily reflect the views of the Department of Defense.
© 2015 Massachusetts Institute of Technology.
MIT Proprietary, Subject to FAR52.227-11 Patent Rights - Ownership by the contractor (May 2014)
The software/firmware is provided to you on an As-Is basis
Delivered to the US Government with Unlimited Rights, as defined in DFARS Part 252.227-7013
or 7014 (Feb 2014). Notwithstanding any copyright notice, U.S. Government rights in this
work are defined by DFARS 252.227-7013 or DFARS 252.227-7014 as detailed above. Use of
this work other than as specifically authorized by the U.S. Government may violate any
copyrights that exist in this work.
# Proactively-secure Accumulo with Cryptographic Enforcement (PACE)
The PACE library's goal is to add cryptographic protections to the Accumulo
database. This currently takes the form of a client-side library containing
cryptographic protections against a malicious Accumulo server, such as a
compromised server machine or system administrator. This README serves as
high-level documentation for the entire library; more low-level information can
be found in the documentation for each subpackage.
## Installation
Once you have the [Dependencies](### Dependencies) installed, you can install PACE by running
`python setup.py install` in the main `pace-<version>` directory. You may
need to provide sudo access.
It is recommended to run `nosetests` in the main folder when the installation
is complete, to ensure that everything installed correctly. They typically run
in 300-500 seconds. If it takes significantly longer, see the section on [Configuring Fastmath](### Configuring Fastmath) below.
To connect to a live Accumulo instance, you will need to run Accumulo with a proxy. See the section on [Running the Thrift Proxy](### Running the Thrift Proxy) for more details.
### Dependencies
PACE depends on the following user installations. These must be installed
manually on your system:
- [python 2.x](https://www.python.org/)
- [GCC with C++ support](https://gcc.gnu.org)
- [MPFR](http://www.mpfr.org/)
- [GMP](https://gmplib.org/)
- [pycryptopp 0.6.0.39+](https://pypi.python.org/pypi/pycryptopp)
- [pycrypto 2.7a1](https://www.dlitz.net/software/pycrypto/)
- [thrift](https://thrift.apache.org/download)
PACE also depends on the following Python packages. If you have a repository
source that Python can access by default, then the PACE installer will install
these automatically. If not, you will need to install them by hand.
- [nosetests](https://nose.readthedocs.org/en/latest/), version 1.3.3+.
- [pyaccumulo](https://pypi.python.org/pypi/pyaccumulo), version 1.5.0.6+.
- [Beaker](https://pypi.python.org/pypi/Beaker), version 1.7.0+.
- [enum34](https://pypi.python.org/pypi/enum34), version 1.0.4+.
- [distribute](https://pypi.python.org/pypi/distribute), version 0.6.14+.
### Configuring Fastmath
One of the dependencies of PACE can be configured to use "fastmath," which
means that it will use GNU's Multiple Precision arithmetic library (GMP) in
place of Python's standard (slower) arithmetic functions. It is recommended
(though not required) to use fastmath, as it greatly speeds up many of PACE's
operations.
If the tests are running very slowly (more than 1000 seconds), then fastmath
is not running, and there is
probably a configuration problem with `pycrypto` and `gmp`. To fix it, first
note the location of `gmp.h`. On Ubuntu, it is likely in `/usr/include/gmp.h`.
On CentOS, it is probably in `/opt/gmp/include/gmp.h`. If you can't find it,
do `sudo find / -name "gmp.h"`. Call the location above the include folder
`<gmp-home>`. For instance, in CentOS, `<gmp-home>` is `/opt/gmp`.
Inside the `pace-<version>` directory, navigate to
`dependencies/shared/src/pycrypto`. Run
```bash
./configure --includedir=<gmp-home>/include --with-gmp
```
Open up src/configure.h and make sure the following definitions are set:
```
#define HAVE_DECL_MPZ_POWM 0
#define HAVE_DECL_MPZ_POWM_SEC 0
#define HAVE_LIBGMP 1
```
Warning: Make sure you do this _after_ you run the previous `configure`
command, otherwise fastmath may not work correctly!
Now, edit the `setup.py` file in the `pycrypto` directory and search for
"`_fastmath`". Make sure the `Extension` declaration looks like this:
```
# _fastmath (uses GNU mp library)
Extension("Crypto.PublicKey._fastmath",
include_dirs=['<gmp-home>/include', 'src/', '/usr/include'],
library_dirs=['<gmp-home>/lib'],
libraries=['gmp'],
sources=["src/_fastmath.c"]),
```
Finally, rebuild and install pycrypto with
```bash
sudo python setup.py build
sudo python setup.py install
```
If it worked, then the build command should have printed
```
building 'Crypto.PublicKey._fastmath' extension
```
and you can now enter python and try to import fastmath by doing:
```bash
python
>>> from Crypto.PublicKey import _fastmath
```
If the build command gives the following warning
```
warning: GMP or MPIR library not found; Not building Crypto.PublicKey._fastmath.
```
then try completely removing pycrypto by doing the following three steps:
1. Remove the `pycrypto` folder from the pace dependencies folder.
2. Remove the `egg-info` file from python's list of packages by running the
following command. Replace `<version>` with your python version, either
2.6 or 2.7.
```
sudo rm /usr/lib64/python<version>/site-packages/pycrypto-2.7a2-py<version>.egg-info
```
3. Remove the `Crypto` folder from the same location:
```
sudo rm -r /usr/lib64/python<version>/site-packages/Crypto
```
Then try completely reinstalling pycrypto. When running PACE, you may get a
warning about the version of `libgmp`. This is a bug in pycrypto that should
not affect the use of PACE.
### Common Installation Issues
If the installer fails, or the tests do not pass, try these steps.
1. Any of the three following messages
```
******************************************************************************
An error occured while trying to compile with the C extension enabled
Attempting to build without the extension now
******************************************************************************
```
```
fatal error: Python.h: No such file or directory
```
```
ERROR: Failure: ImportError (No module named Signature)
```
mean that your Python installation is broken. Try updating it and
then re-running `./install.sh`.
Using `apt-get`, you can update by doing `apt-get update` then
`apt-get install -f` and then finally `apt-get install python2.7-dev`.
Using `yum`, do `yum clean all` followed by `yum update python-devel`.
2. If you get a permission denied error, such as
```
[Errno 13] Permission denied: '/usr/local/lib/python2.7/dist-packages/test-easy-install-8988.pth'
```
try rerunning with 'sudo' or as root.
3. If, while the installer is running, you see a flood of messages like
```
[Errno 13] Permission denied: '/usr/local/lib/python2.7/dist-packages/test-easy-install-8988.pth'
```
and then when running the tests, you get errors with `pycryptopp` such as
```
ERROR: Failure: ImportError (No module named pycryptopp.publickey)
```
then your `g++` installation is likely broken. Try updating it and then
rerunning `./install.sh`.
Using `apt-get`, you can update by doing `apt-get update` then
`apt-get install -f` and then finally `apt-get install g++`.
Using `yum`, do `yum clean all` followed by `yum update g++`.
4. If you compiled GMP from source, it is recommended that you add the
following line to your `~/.bashrc` file (and use `source ~/.bashrc` to
make the changes take effect). Replace `$GMP_HOME` with your main GMP install location.
```
export LD_LIBRARY_PATH=$GMP_HOME/lib:$LD_LIBRARY_PATH
```
## Running the Thrift Proxy
For PACE to use a live Accumulo instance, you must run Accumulo with a
proxy so that thrift will
function properly. Assuming your main Accumulo folder is $ACCUMULO_HOME,
edit the `$ACCUMULO_HOME/proxy/proxy.properties` file to contain the correct
information. Notably, make sure `instance` is set to the name of your
Accumulo instance, make sure `zookeepers` points to the host and port of
your zookeeper server, and note the port number (the demo PACE code uses
port 42424, but you can set this to any unused port). Make sure your
Accumulo instance is running, then run the Accumulo
proxy in the terminal by running
```bash
$ACCUMULO_HOME/bin/accumulo proxy -p $ACCUMULO_HOME/proxy/proxy.properties &
disown
```
(Disowning the process at the end means that the process will no longer be
halted when your current terminal window closes. If you have other running
jobs in this terminal that you do not want to disown, look up the jobID of
the thrift proxy by typing `jobs -l` and then typing `disown <jobID>`).
You can check to make sure the proxy is running correctly by opening
python and typing the following, changing your Accumulo settings as
necessary:
```bash
python
>>> from pyaccumulo import Accumulo
>>> conn = Accumulo(user='root', password='secret', host='pacetest-client',
... port=42424)
```
## Subpackages
In this section, we document the subpackages contained within this project. With
the exception of `common`, each of these subpackages also has a
README.md file associated with it, that goes in to more detail about its
capabilities and use. This file aims to provide a preliminary roadmap for users
to determine what to read about in more depth.
### ads
The `ads` subpackage contains implementations of Authenticated Data Structures,
or ADSs. These are data structures that contain extra metadata allowing
untrusted servers hosting them to return query results that can be verified by
cross-referencing with a small, verified value from a trusted user. We use these
to implement a prototype of such a scheme in Accumulo, using Authenticated Skip
Lists.
Note that this is one way of cryptographically verifying the results of queries
received from Accumulo, similar to the `signatures` subpackage described below.
However, this is currenty still prototype-level; while digital signatures can
only provide cell-level, not query-level, integrity checks, our implementation
of them is much more mature and therefore currently recommended over our ADS
library.
### common
The `common` subpackage contains several utility files common to all other
subpackages of PACE. In particular, the files in it are as follows:
- `common_utils`: contains utility functions for generating test data and
interacting with Accumulo instances.
- `fakeconn`: defines a class `FakeConnection` used for testing
Accumulo-dependent code without a live Accumulo instance.
- `pacetest`: defines `PACETestCase`, a subclass of the `TestCase` class from
Python's `unittest` package, with several convenient default fields and a
`generate_elems()` function for randomly generating test data.
### encryption
The `encryption` package contains ways to cryptographically enforce the
confidentiality of data stored in Accumulo. We have two main ways of doing this:
- Field-level encryption: a flexible library for encrypting different fields of
Accumulo cells in different ways, allowing the user to determine what to
encrypt and how based on security and performance needs.
- Cryptographically enforced access control: a means of
cryptographically enforcing Accumulo's cell-level security feature, preventing
a malicious server from leaking data to innocent users who should not have
access to it.
### pki
The `pki` package defines an interface for a key management system for the
cryptographically enforced access control mentioned above, and implements this
interface using Accumulo to store keys.
### signatures
The `signatures` package contains ways to cryptographically enforce the
integrity of data stored in Accumulo. This uses digital signatures to sign the
entirety of each Accumulo cell, allowing clients performing queries to guarantee
that each result from the query is correct and has not been tampered with.
## License
Copyright (c) 2016, MIT Lincoln Laboratory
All rights reserved.
Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
pace.egg-info/PKG-INFO
+293
View File
@@ -0,0 +1,293 @@
Metadata-Version: 1.0
Name: pace
Version: 1.3
Summary: Proactively-Secure Accumulo with Cryptographic Enforcement
Home-page: UNKNOWN
Author: MIT Lincoln Laboratory
Author-email: [email protected]
License: UNKNOWN
Description: DISTRIBUTION STATEMENT A. Approved for public release: distribution unlimited.
This material is based upon work supported by the Department of Defense under Air Force Contract
No. FA8721-05-C-0002 and/or FA8702-15-D-0001. Any opinions, findings, conclusions or
recommendations expressed in this material are those of the author(s) and do not
necessarily reflect the views of the Department of Defense.
&copy; 2015 Massachusetts Institute of Technology.
MIT Proprietary, Subject to FAR52.227-11 Patent Rights - Ownership by the contractor (May 2014)
The software/firmware is provided to you on an As-Is basis
Delivered to the US Government with Unlimited Rights, as defined in DFARS Part 252.227-7013
or 7014 (Feb 2014). Notwithstanding any copyright notice, U.S. Government rights in this
work are defined by DFARS 252.227-7013 or DFARS 252.227-7014 as detailed above. Use of
this work other than as specifically authorized by the U.S. Government may violate any
copyrights that exist in this work.
# Proactively-secure Accumulo with Cryptographic Enforcement (PACE)
The PACE library's goal is to add cryptographic protections to the Accumulo
database. This currently takes the form of a client-side library containing
cryptographic protections against a malicious Accumulo server, such as a
compromised server machine or system administrator. This README serves as
high-level documentation for the entire library; more low-level information can
be found in the documentation for each subpackage.
## Installation
Once you have the [Dependencies](### Dependencies) installed, you can install PACE by running
`python setup.py install` in the main `pace-<version>` directory. You may
need to provide sudo access.
It is recommended to run `nosetests` in the main folder when the installation
is complete, to ensure that everything installed correctly. They typically run
in 300-500 seconds. If it takes significantly longer, see the section on [Configuring Fastmath](### Configuring Fastmath) below.
To connect to a live Accumulo instance, you will need to run Accumulo with a proxy. See the section on [Running the Thrift Proxy](### Running the Thrift Proxy) for more details.
### Dependencies
PACE depends on the following user installations. These must be installed
manually on your system:
- [python 2.x](https://www.python.org/)
- [GCC with C++ support](https://gcc.gnu.org)
- [MPFR](http://www.mpfr.org/)
- [GMP](https://gmplib.org/)
- [pycryptopp 0.6.0.39+](https://pypi.python.org/pypi/pycryptopp)
- [pycrypto 2.7a1](https://www.dlitz.net/software/pycrypto/)
- [thrift](https://thrift.apache.org/download)
PACE also depends on the following Python packages. If you have a repository
source that Python can access by default, then the PACE installer will install
these automatically. If not, you will need to install them by hand.
- [nosetests](https://nose.readthedocs.org/en/latest/), version 1.3.3+.
- [pyaccumulo](https://pypi.python.org/pypi/pyaccumulo), version 1.5.0.6+.
- [Beaker](https://pypi.python.org/pypi/Beaker), version 1.7.0+.
- [enum34](https://pypi.python.org/pypi/enum34), version 1.0.4+.
- [distribute](https://pypi.python.org/pypi/distribute), version 0.6.14+.
### Configuring Fastmath
One of the dependencies of PACE can be configured to use "fastmath," which
means that it will use GNU's Multiple Precision arithmetic library (GMP) in
place of Python's standard (slower) arithmetic functions. It is recommended
(though not required) to use fastmath, as it greatly speeds up many of PACE's
operations.
If the tests are running very slowly (more than 1000 seconds), then fastmath
is not running, and there is
probably a configuration problem with `pycrypto` and `gmp`. To fix it, first
note the location of `gmp.h`. On Ubuntu, it is likely in `/usr/include/gmp.h`.
On CentOS, it is probably in `/opt/gmp/include/gmp.h`. If you can't find it,
do `sudo find / -name "gmp.h"`. Call the location above the include folder
`<gmp-home>`. For instance, in CentOS, `<gmp-home>` is `/opt/gmp`.
Inside the `pace-<version>` directory, navigate to
`dependencies/shared/src/pycrypto`. Run
```bash
./configure --includedir=<gmp-home>/include --with-gmp
```
Open up src/configure.h and make sure the following definitions are set:
```
#define HAVE_DECL_MPZ_POWM 0
#define HAVE_DECL_MPZ_POWM_SEC 0
#define HAVE_LIBGMP 1
```
Warning: Make sure you do this _after_ you run the previous `configure`
command, otherwise fastmath may not work correctly!
Now, edit the `setup.py` file in the `pycrypto` directory and search for
"`_fastmath`". Make sure the `Extension` declaration looks like this:
```
# _fastmath (uses GNU mp library)
Extension("Crypto.PublicKey._fastmath",
include_dirs=['<gmp-home>/include', 'src/', '/usr/include'],
library_dirs=['<gmp-home>/lib'],
libraries=['gmp'],
sources=["src/_fastmath.c"]),
```
Finally, rebuild and install pycrypto with
```bash
sudo python setup.py build
sudo python setup.py install
```
If it worked, then the build command should have printed
```
building 'Crypto.PublicKey._fastmath' extension
```
and you can now enter python and try to import fastmath by doing:
```bash
python
>>> from Crypto.PublicKey import _fastmath
```
If the build command gives the following warning
```
warning: GMP or MPIR library not found; Not building Crypto.PublicKey._fastmath.
```
then try completely removing pycrypto by doing the following three steps:
1. Remove the `pycrypto` folder from the pace dependencies folder.
2. Remove the `egg-info` file from python's list of packages by running the
following command. Replace `<version>` with your python version, either
2.6 or 2.7.
```
sudo rm /usr/lib64/python<version>/site-packages/pycrypto-2.7a2-py<version>.egg-info
```
3. Remove the `Crypto` folder from the same location:
```
sudo rm -r /usr/lib64/python<version>/site-packages/Crypto
```
Then try completely reinstalling pycrypto. When running PACE, you may get a
warning about the version of `libgmp`. This is a bug in pycrypto that should
not affect the use of PACE.
### Common Installation Issues
If the installer fails, or the tests do not pass, try these steps.
1. Any of the three following messages
```
******************************************************************************
An error occured while trying to compile with the C extension enabled
Attempting to build without the extension now
******************************************************************************
```
```
fatal error: Python.h: No such file or directory
```
```
ERROR: Failure: ImportError (No module named Signature)
```
mean that your Python installation is broken. Try updating it and
then re-running `./install.sh`.
Using `apt-get`, you can update by doing `apt-get update` then
`apt-get install -f` and then finally `apt-get install python2.7-dev`.
Using `yum`, do `yum clean all` followed by `yum update python-devel`.
2. If you get a permission denied error, such as
```
[Errno 13] Permission denied: '/usr/local/lib/python2.7/dist-packages/test-easy-install-8988.pth'
```
try rerunning with 'sudo' or as root.
3. If, while the installer is running, you see a flood of messages like
```
[Errno 13] Permission denied: '/usr/local/lib/python2.7/dist-packages/test-easy-install-8988.pth'
```
and then when running the tests, you get errors with `pycryptopp` such as
```
ERROR: Failure: ImportError (No module named pycryptopp.publickey)
```
then your `g++` installation is likely broken. Try updating it and then
rerunning `./install.sh`.
Using `apt-get`, you can update by doing `apt-get update` then
`apt-get install -f` and then finally `apt-get install g++`.
Using `yum`, do `yum clean all` followed by `yum update g++`.
4. If you compiled GMP from source, it is recommended that you add the
following line to your `~/.bashrc` file (and use `source ~/.bashrc` to
make the changes take effect). Replace `$GMP_HOME` with your main GMP install location.
```
export LD_LIBRARY_PATH=$GMP_HOME/lib:$LD_LIBRARY_PATH
```
## Running the Thrift Proxy
For PACE to use a live Accumulo instance, you must run Accumulo with a
proxy so that thrift will
function properly. Assuming your main Accumulo folder is $ACCUMULO_HOME,
edit the `$ACCUMULO_HOME/proxy/proxy.properties` file to contain the correct
information. Notably, make sure `instance` is set to the name of your
Accumulo instance, make sure `zookeepers` points to the host and port of
your zookeeper server, and note the port number (the demo PACE code uses
port 42424, but you can set this to any unused port). Make sure your
Accumulo instance is running, then run the Accumulo
proxy in the terminal by running
```bash
$ACCUMULO_HOME/bin/accumulo proxy -p $ACCUMULO_HOME/proxy/proxy.properties &
disown
```
(Disowning the process at the end means that the process will no longer be
halted when your current terminal window closes. If you have other running
jobs in this terminal that you do not want to disown, look up the jobID of
the thrift proxy by typing `jobs -l` and then typing `disown <jobID>`).
You can check to make sure the proxy is running correctly by opening
python and typing the following, changing your Accumulo settings as
necessary:
```bash
python
>>> from pyaccumulo import Accumulo
>>> conn = Accumulo(user='root', password='secret', host='pacetest-client',
... port=42424)
```
## Subpackages
In this section, we document the subpackages contained within this project. With
the exception of `common`, each of these subpackages also has a
README.md file associated with it, that goes in to more detail about its
capabilities and use. This file aims to provide a preliminary roadmap for users
to determine what to read about in more depth.
### ads
The `ads` subpackage contains implementations of Authenticated Data Structures,
or ADSs. These are data structures that contain extra metadata allowing
untrusted servers hosting them to return query results that can be verified by
cross-referencing with a small, verified value from a trusted user. We use these
to implement a prototype of such a scheme in Accumulo, using Authenticated Skip
Lists.
Note that this is one way of cryptographically verifying the results of queries
received from Accumulo, similar to the `signatures` subpackage described below.
However, this is currenty still prototype-level; while digital signatures can
only provide cell-level, not query-level, integrity checks, our implementation
of them is much more mature and therefore currently recommended over our ADS
library.
### common
The `common` subpackage contains several utility files common to all other
subpackages of PACE. In particular, the files in it are as follows:
- `common_utils`: contains utility functions for generating test data and
interacting with Accumulo instances.
- `fakeconn`: defines a class `FakeConnection` used for testing
Accumulo-dependent code without a live Accumulo instance.
- `pacetest`: defines `PACETestCase`, a subclass of the `TestCase` class from
Python's `unittest` package, with several convenient default fields and a
`generate_elems()` function for randomly generating test data.
### encryption
The `encryption` package contains ways to cryptographically enforce the
confidentiality of data stored in Accumulo. We have two main ways of doing this:
- Field-level encryption: a flexible library for encrypting different fields of
Accumulo cells in different ways, allowing the user to determine what to
encrypt and how based on security and performance needs.
- Cryptographically enforced access control: a means of
cryptographically enforcing Accumulo's cell-level security feature, preventing
a malicious server from leaking data to innocent users who should not have
access to it.
### pki
The `pki` package defines an interface for a key management system for the
cryptographically enforced access control mentioned above, and implements this
interface using Accumulo to store keys.
### signatures
The `signatures` package contains ways to cryptographically enforce the
integrity of data stored in Accumulo. This uses digital signatures to sign the
entirety of each Accumulo cell, allowing clients performing queries to guarantee
that each result from the query is correct and has not been tampered with.
Platform: UNKNOWN
pace.egg-info/SOURCES.txt
+171
View File
@@ -0,0 +1,171 @@
README.md
setup.py
pace/__init__.py
pace.egg-info/PKG-INFO
pace.egg-info/SOURCES.txt
pace.egg-info/dependency_links.txt
pace.egg-info/requires.txt
pace.egg-info/top_level.txt
pace/ads/README.md
pace/ads/__init__.py
pace/ads/merkle/__init__.py
pace/ads/merkle/benchmark.py
pace/ads/merkle/empty_node.py
pace/ads/merkle/eq.py
pace/ads/merkle/hash_node.py
pace/ads/merkle/mht.py
pace/ads/merkle/mht_node.py
pace/ads/merkle/mht_node_test.py
pace/ads/merkle/mht_server.py
pace/ads/merkle/mht_server_test.py
pace/ads/merkle/mht_test.py
pace/ads/merkle/mht_utils.py
pace/ads/merkle/vo.py
pace/ads/merkle/vo_node.py
pace/ads/merkle/vo_test.py
pace/ads/skiplist/.authskiplist.py.swp
pace/ads/skiplist/.skiplist.py.swp
pace/ads/skiplist/README.md
pace/ads/skiplist/__init__.py
pace/ads/skiplist/authnode.py
pace/ads/skiplist/authskiplist.py
pace/ads/skiplist/authskiplist_test.py
pace/ads/skiplist/coin.py
pace/ads/skiplist/coin_test.py
pace/ads/skiplist/elemclass.py
pace/ads/skiplist/elemclass_test.py
pace/ads/skiplist/skiplist.py
pace/ads/skiplist/skiplist_server.py
pace/ads/skiplist/skiplist_server_test.py
pace/ads/skiplist/skiplist_test.py
pace/ads/skiplist/skiplistnode.py
pace/ads/skiplist/skiplistvo.py
pace/ads/skiplist/skiplistvo_test.py
pace/ads/skiplist/vonode.py
pace/ads/skiplist/accumulo/.demo.py.swp
pace/ads/skiplist/accumulo/__init__.py
pace/ads/skiplist/accumulo/embeddednode.py
pace/ads/skiplist/accumulo/embeddedskiplist.py
pace/ads/skiplist/accumulo/embeddedskiplist_test.py
pace/common/__init__.py
pace/common/common_utils.py
pace/common/fakeconn.py
pace/common/pacetest.py
pace/encryption/.demo_commands.txt.swp
pace/encryption/.enc_mutation_test.py.swp
pace/encryption/.encrypt_test.py.swp
pace/encryption/.encryption_pki.py.swp
pace/encryption/AES_encrypt.py
pace/encryption/AES_encrypt_test.py
pace/encryption/DEMO.md
pace/encryption/README.md
pace/encryption/__init__.py
pace/encryption/abstract_encrypt.py
pace/encryption/abstract_encrypt_test.py
pace/encryption/acc_encrypt.py
pace/encryption/acc_encrypt_test.py
pace/encryption/benchmarker.py
pace/encryption/demo.py
pace/encryption/enc_classes.py
pace/encryption/enc_mutation.py
pace/encryption/enc_mutation_test.py
pace/encryption/enc_utils.py
pace/encryption/encrypt_test.py
pace/encryption/encryption_exceptions.py
pace/encryption/encryption_pki.py
pace/encryption/perf_benchmarking.py
pace/encryption/vars.py
pace/encryption/config/config_examp.cfg
pace/encryption/config/ceabac/VIS_AES_CBC.cfg
pace/encryption/config/ceabac/VIS_AES_CFB.cfb
pace/encryption/config/ceabac/VIS_AES_CTR.cfg
pace/encryption/config/ceabac/VIS_AES_GCM.cfg
pace/encryption/config/ceabac/VIS_AES_OFB.cfg
pace/encryption/config/different_schemas_ceabac/ceabac_cq_value.cfg
pace/encryption/config/different_schemas_ceabac/ceabac_row_cf_cq_value.cfg
pace/encryption/config/different_schemas_ceabac/ceabac_row_cfcq_value.cfg
pace/encryption/config/different_schemas_ceabac/ceabac_row_cfcqvalue.cfg
pace/encryption/config/different_schemas_ceabac/ceabac_row_value.cfg
pace/encryption/config/different_schemas_ceabac/ceabac_value.cfg
pace/encryption/config/different_schemas_non_ceabac/cq_value.cfg
pace/encryption/config/different_schemas_non_ceabac/row_cf_cq_value.cfg
pace/encryption/config/different_schemas_non_ceabac/row_cfcq_value.cfg
pace/encryption/config/different_schemas_non_ceabac/row_cfcqvalue.cfg
pace/encryption/config/different_schemas_non_ceabac/row_value.cfg
pace/encryption/config/different_schemas_non_ceabac/value.cfg
pace/encryption/config/mixed_schemas/SSE_auth_VIS.cfg
pace/encryption/config/mixed_schemas/det_SSE_VIS.cfg
pace/encryption/config/mixed_schemas/det_sse_auth.cfg
pace/encryption/config/non-ceabac/AES_CBC.cfg
pace/encryption/config/non-ceabac/AES_CFB.cfg
pace/encryption/config/non-ceabac/AES_CTR.cfg
pace/encryption/config/non-ceabac/AES_GCM.cfg
pace/encryption/config/non-ceabac/AES_OFB.cfg
pace/encryption/config/non-ceabac/AES_SIV.cfg
pace/encryption/visibility/__init__.py
pace/encryption/visibility/secret_vis_tree.py
pace/encryption/visibility/secret_vis_tree_test.py
pace/encryption/visibility/vis_encrypt.py
pace/encryption/visibility/vis_parser.py
pace/encryption/visibility/vis_parser_test.py
pace/pki/.frontend.py.swp
pace/pki/README.md
pace/pki/__init__.py
pace/pki/abstractpki.py
pace/pki/accumulo_keystore.py
pace/pki/attrusermap.py
pace/pki/attrusermap_test.py
pace/pki/dictpki.py
pace/pki/frontend.py
pace/pki/key_wrap_utils.py
pace/pki/key_wrap_utils_test.py
pace/pki/keygen.py
pace/pki/keygen_test.py
pace/pki/keystore.py
pace/pki/keystore_test.py
pace/pki/user1_privkey.pem
pace/pki/user1_pubkey.pem
pace/pki/user2_privkey.pem
pace/pki/user2_pubkey.pem
pace/pki/user3_privkey.pem
pace/pki/user3_pubkey.pem
pace/pki/user_info.cfg
pace/pki/userattrmap.py
pace/signatures/.signaturepki.py.swp
pace/signatures/DEMO.md
pace/signatures/README.md
pace/signatures/__init__.py
pace/signatures/acc_sig.py
pace/signatures/benchmark_results.csv
pace/signatures/benchmarker.py
pace/signatures/default_file.txt
pace/signatures/demo.py
pace/signatures/full_benchmark_out.csv
pace/signatures/manualtruststore.py
pace/signatures/noisy_default_file.txt
pace/signatures/sign.py
pace/signatures/sign_test.py
pace/signatures/sign_utils.py
pace/signatures/sign_utils_old.py
pace/signatures/signaturepki.py
pace/signatures/signconfig.py
pace/signatures/test.sh
pace/signatures/test_main.py
pace/signatures/truststore_test.py
pace/signatures/vars.py
pace/signatures/verify.py
pace/signatures/cfg/batch_test.cfg
pace/signatures/cfg/stream_test.cfg
pace/signatures/cfg/value_test.cfg
pace/signatures/keys/bad_store_1
pace/signatures/keys/bad_store_2
pace/signatures/keys/bad_store_3
pace/signatures/keys/bad_store_4
pace/signatures/keys/bad_store_4b
pace/signatures/keys/test_ecdsa_privkey.pem
pace/signatures/keys/test_ecdsa_pubkey.pem
pace/signatures/keys/test_privkey.rsa
pace/signatures/keys/test_privkey_3072.rsa
pace/signatures/keys/test_pubkey.rsa
pace/signatures/keys/test_pubkey_3072.rsa
pace/signatures/keys/test_store
pace.egg-info/dependency_links.txt
+1
View File
@@ -0,0 +1 @@
pace.egg-info/requires.txt
+5
View File
@@ -0,0 +1,5 @@
Beaker>=1.7.0
enum34>=1.0.4
nose>=1.1.2
pyaccumulo>=1.5.0.6
distribute>=0.6.14
pace.egg-info/top_level.txt
+1
View File
@@ -0,0 +1 @@
pace
pace/__init__.py
View File
Whitespace-only changes.
pace/ads/README.md
+39
View File
@@ -0,0 +1,39 @@
DISTRIBUTION STATEMENT A. Approved for public release: distribution unlimited.
This material is based upon work supported by the Department of Defense under Air Force Contract
No. FA8721-05-C-0002 and/or FA8702-15-D-0001. Any opinions, findings, conclusions or
recommendations expressed in this material are those of the author(s) and do not
necessarily reflect the views of the Department of Defense.
&copy; 2015 Massachusetts Institute of Technology.
MIT Proprietary, Subject to FAR52.227-11 Patent Rights - Ownership by the contractor (May 2014)
The software/firmware is provided to you on an As-Is basis
Delivered to the US Government with Unlimited Rights, as defined in DFARS Part 252.227-7013
or 7014 (Feb 2014). Notwithstanding any copyright notice, U.S. Government rights in this
work are defined by DFARS 252.227-7013 or DFARS 252.227-7014 as detailed above. Use of
this work other than as specifically authorized by the U.S. Government may violate any
copyrights that exist in this work.
# Authenticated Data Structures
Authenticated Data Structures, or ADSs, give users the ability to verify the
correctness of queries from an untrusted server given only a small
(constant-size), signed value from a trusted data owner. In looking to
implement this functionality in Accumulo, we investigated two such data
structures: Merkle Hash Trees, and Authenticated Skip Lists.
We include implementations of both data structures here. However, Merkle Hash
Trees as implemented here are more suited for static data sets, which is at odds
with Accumulo's typically high ingest rates. We implemented Authenticated Skip
Lists as an efficient option for performing rapid insert operations. For this
reason, there are two directories here:
- `merkle`: an implementation of Merkle Hash Trees. This is solely a tree
library, and does not interface with Accumulo at all. As such, we include no
README in this directory.
- `skiplist`: an implementation of skip lists, authenticated skip lists, and an
embedding of the latter into Accumulo. More information about this subpackage
can be found in its directory's README.
pace/ads/__init__.py
View File
Whitespace-only changes.
pace/ads/merkle/__init__.py
View File
Whitespace-only changes.
pace/ads/merkle/benchmark.py
+118
View File
@@ -0,0 +1,118 @@
## **************
## Copyright 2014 MIT Lincoln Laboratory
## Project: PACE
## Authors: ZS
## Description: Benchmarks for the merkle tree library
## Modifications:
## Date Name Modification
## ---- ---- ------------
## 08 Aug 2014 ZS Original file
## **************
import random
from pace.ads.merkle.mht import MHT, MHTInsertionException
from pace.ads.merkle.mht_utils import MHTUtils
from pace.ads.merkle.mht_node import MHTNode
from pace.ads.merkle.vo import VO
class MHTBenchmark(object):
def __init__(self,
mht_sizes=[100,500,1000,5000,100000], #,100000],
vo_size_fractions=[10000,1000,100,50,20,10,5,2],
input_chunks=[1,2,4,8,16]):
self.mht_sizes = mht_sizes
self.vo_size_fractions = vo_size_fractions
self.input_chunks = input_chunks
def benchmark(self):
for size in self.mht_sizes:
for num_chunks in self.input_chunks:
elems = [random.randint(1, 99999999)
for i in range(0, size/num_chunks - 2)]
elems.sort()
elems = [0] + elems + [100000000]
## mht1 - iterated single-element insert
## mht2 - gestalt batch insert
mht1 = MHT.new(elems)
mht2 = MHT.new(elems)
for i in range(1, num_chunks):
elems = [random.randint(1, 99999999)
for i in range(1, size/num_chunks)]
mht1.batch_insert(elems)
mht2._gestalt_batch_insert(min(elems), max(elems), elems)
this_output = []
for frac in self.vo_size_fractions:
if frac >= size:
continue
vo_size = size / frac
start = random.choice(range(1, size - vo_size - 1))
end = start + vo_size
vo1 = mht1.range_query(start, end)
vo2 = mht2.range_query(start, end)
svo1 = vo1.serialize()
svo2 = vo2.serialize()
ser_length1 = len(svo1)
ser_length2 = len(svo2)
this_output.append((vo_size, ser_length1, ser_length2))
## Output format: (size, num_chunks, this_output) where:
##
## size - the (approximate) total number of elements in the MHT
## actual_size - the length of the MHT's sorted_elems field
## num_chunks - the number of chunks in which the elements are
## inserted into the MHT
## this_output - the output from running range queries on the
## MHT built with those properties.
yield ((size, len(mht1.sorted_elems), num_chunks, this_output))
@staticmethod
def stringify_benchmark_output(output):
lines = []
for size, actual_size, num_chunks, runs in output:
lines.append('MHT of size %d (%d) split into %d chunks:' %(size, actual_size, num_chunks))
intermediate = []
for vo_size, iter_length, batch_length in runs:
intermediate.append(
'VO size: %d\n\t\tbits per element (sequential insert): %f\n\t\tbits per element (gestalt insert): %f'
%(vo_size, iter_length/float(vo_size), batch_length/float(vo_size)))
lines = lines + ['\t' + line for line in intermediate]
return '\n'.join(lines)
@staticmethod
def print_benchmark_output(output):
for size, actual_size, num_chunks, runs in output:
print 'MHT of size %d (%d) split into %d chunks:' %(size, actual_size, num_chunks)
intermediate = []
for vo_size, iter_length, batch_length in runs:
print '\t%s' %(
'VO size: %d\n\t\tbits per element (sequential insert): %f\n\t\tbits per element (gestalt insert): %f'
%(vo_size, iter_length/float(vo_size), batch_length/float(vo_size)))
def run(self):
try:
MHTBenchmark.print_benchmark_output(self.benchmark())
except MHTInsertionException as e:
print 'ERROR!!!'
print e.msg
raise e
def go():
MHTBenchmark().run()
pace/ads/merkle/empty_node.py
+29
View File
@@ -0,0 +1,29 @@
## **************
## Copyright 2014 MIT Lincoln Laboratory
## Project: PACE
## Authors: ZS
## Description: Empty node class for VOs
## Modifications:
## Date Name Modification
## ---- ---- ------------
## 28 Jul 2014 ZS Original file
## **************
from pace.ads.merkle.eq import EqMixin
class EmptyNode(EqMixin):
def __init__(self):
pass
def set_parent(self, parent):
pass
def serialize(self, depth=0):
return 'None'
@staticmethod
def deserialize(s):
if s == 'None':
return EmptyNode()
else:
raise Error('Error in deserialization:\nExpected "None", got %s' %s)
pace/ads/merkle/eq.py
+11
View File
@@ -0,0 +1,11 @@
## Code taken from StackOverflow:
## http://stackoverflow.com/questions/390250/elegant-ways-to-support-equivalence-equality-in-python-classes
## 15 Oct 2014: changed '__neq__' to '__ne__' (CS)
class EqMixin(object):
def __eq__(self, other):
return (isinstance(other, self.__class__)
and self.__dict__ == other.__dict__)
def __ne__(self, other):
return not self.__eq__(other)
pace/ads/merkle/hash_node.py
+28
View File
@@ -0,0 +1,28 @@
## **************
## Copyright 2014 MIT Lincoln Laboratory
## Project: PACE
## Authors: ZS
## Description: Hash node class for VOs
## Modifications:
## Date Name Modification
## ---- ---- ------------
## 28 Jul 2014 ZS Original file
## **************
from base64 import b64encode, b64decode
from pace.ads.merkle.eq import EqMixin
class HashNode(EqMixin):
def __init__(self, hval):
self.hval = hval
def set_parent(self, parent):
pass
def serialize(self, depth=0):
return b64encode(self.hval)
@staticmethod
def deserialize(s):
return b64decode(s)
pace/ads/merkle/mht.py
+370
View File
@@ -0,0 +1,370 @@
## **************
## Copyright 2014 MIT Lincoln Laboratory
## Project: PACE
## Authors: ZS
## Description: Merkle hash tree library
## Modifications:
## Date Name Modification
## ---- ---- ------------
## 17 Jul 2014 ZS Original file
## **************
import bisect
from pace.ads.merkle.mht_utils import MHTUtils
from pace.ads.merkle.mht_node import MHTNode
from pace.ads.merkle.vo import VO
class MHTInsertionException(Exception):
def __init__(self, msg):
self.msg = msg
class MHT(object):
""" Binary Merkle hash tree class. Provides a verified data structure
that can prove that an element is (or isn't) contained within it.
Instance variables:
elems - a dict of elements mapped to their corresponding leaf node
in the hash tree
sorted_elems - the original (sorted) list of elements
root - the top level node of the hash tree, whoseh hash value depends
on all of the elements.
"""
def __init__(self, elems, sorted_elems, root):
self.elems = elems
self.sorted_elems = sorted_elems
self.root = root
@staticmethod
def new(elems):
""" Assumes elems is sorted, nonempty, and contains no
repeated elements.
"""
sorted_elems = elems
work = [MHTNode.leaf(elem) for elem in elems]
elem_dict = dict(zip(elems, work))
next_level = []
# I am going out of my way not to implement this recursively,
# because python
while work:
for i in range(0, len(work), 2):
if i+1 >= len(work):
next_level.append(work[i])
else:
left = work[i]
right = work[i+1]
new_hash = MHTUtils.merge_hashes(left.hval, right.hval)
new_node = MHTNode.node(new_hash, left, right)
left.parent = new_node
right.parent = new_node
next_level.append(new_node)
if len(next_level) == 1:
return MHT(elems=elem_dict, sorted_elems=elems,
root=next_level[0])
work = next_level
next_level = []
@staticmethod
def partial_insert(sorted_elems, elem):
""" Insert an element into a list of leaves. Helper function for
both insert() and batch_insert()
Arguments:
sorted_elems - a sorted list of elements in the MHT
elem - the element to insert into leaves
Returns:
new_leaves - a list of leaves in an MHT that contains elem
inserted into the correct location
i - the index into which elem was inserted.
"""
i = bisect.bisect_left(sorted_elems, elem)
return (sorted_elems[:i] + [elem] + sorted_elems[i:], i)
def insert(self, elem):
""" Insertion of a single element into a Merkle hash tree.
NB: this function does *not* guarantee anything about the
balancedness of the resulting tree.
Arguments:
self - the MHT to be modified
elem - the element to be inserted
Returns:
vo - a verification object that proves that the range into which
elem was inserted was in fact empty before insertion
Assumes:
- There are at least two 'boundary' elements in the MHT
- One boundary element is less than everything else that will
be inserted (including elem), and the other is greater than
everything else (including elem).
"""
## Insertion algorithm:
## 1) Find where the element belongs in the leaf list.
## 2) Choose a neighbor to pair it with, and make a new node
## that is the parent of those two leaves.
## 3) Replace that leaf's spot in the old tree with that node.
## 4) Move up the tree and recompute the hash values at each
## node, constructing a VO on the way up.
new_elems, i = MHT.partial_insert(self.sorted_elems, elem)
vo = VO.new(self.sorted_elems[i-1], [], self.sorted_elems[i], self.root)
self._batch_single_insert(elem, i, new_elems)
return vo
def _batch_single_insert(self, elem, i, new_elems):
""" Insertion of a single element into a Merkle hash tree without
a corresponding VO, as part of a batch insert.
NB: this function does *not* guarantee anything about the
balancedness of the resulting tree.
Arguments:
self - the MHT to be modified
elem - the element to be inserted
i - the index at which to insert elem
new_elems - the new list of elements
Returns:
i - the index at which elem was inserted into the list
Assumes:
- There are at least two 'boundary' elements in the MHT
- One boundary element is less than everything else that will
be inserted (including elem), and the other is greater than
everything else (including elem).
Optimization Opportunities:
- We might be able to get away with waiting to update the tree
until the end of the batch insert, then doing it all at once;
it seems like there's a lot of unnecessary work being done here.
"""
if i-1 < 0 or i+1 > len(self.sorted_elems):
raise MHTInsertionException(
'Element to be inserted %d does not occur within range of MHT (from %d to %d)\nGiven index %d in a %d-element list' %(elem, self.sorted_elems[0], self.sorted_elems[-1], i, len(self.sorted_elems)))
new_leaf = MHTNode.leaf(elem)
self.sorted_elems = new_elems
self.elems[elem] = new_leaf
# Sibling choice needs to be deterministic, so we choose the
# left sibling
sibling_leaf = self.elems[self.sorted_elems[i-1]]
MHTNode.insert(new_leaf, sibling_leaf)
def batch_insert(self, elems):
""" Insert a series of elements into a tree all at once.
Arguments:
elems - an iterable sequence of elements to insert
Optimization opportunities:
- Requiring that the input list be sorted might provide some
interesting ways to optimize insertion? Not sure how realistic
it is in practice, though.
"""
for elem in elems:
new_elems, i = MHT.partial_insert(self.sorted_elems, elem)
self._batch_single_insert(elem, i, new_elems)
@staticmethod
def batch_list_insert(elem, sorted_elems):
""" Step 1/3 of batch insertion.
The first precomputing phase of a batch insert. Determines the list
of elements to ultimately insert into the MHT.
Arguments:
elem - the next element to insert in this phase
sorted_elems - the elements in the range to be inserted into
so far
"""
bisect.insort_left(sorted_elems, elem)
@staticmethod
def batch_node_insert(sorted_elems):
""" Step 2/3 of batch insertion.
Makes the entire new subtree for an insert out of the sorted list
of elements to exist within that range. Currently guaranteed to be
as balanced as possible, since it just makes a new MHT.
Arguments:
sorted_elems - the list of elements within the insertion range,
in sorted order.
Returns:
mht - the entire new subtree to be added to the main MHT
"""
return MHT.new(sorted_elems)
def batch_update(self, subtree, lca):
""" Step 3/3 of batch insertion.
Modifies the MHT to reflect the changes made during the previous
two steps.
Arguments:
subtree - the MHT representing the tree to be placed at the least
common ancestor of the insertion range.
lca - the root of the subtree of the original MHT being modified
"""
if lca.parent:
# LCA isn't the root, so make sure the new subtree has the
# right parent
subtree.root.parent = lca.parent
# Set the LCA's parent child to the new subtree instead of
# the LCA
if lca.parent.left is lca:
lca.parent.left = subtree.root
elif lca.parent.right is lca:
lca.parent.right = subtree.root
else:
# if the lca has no parent, it was the root of the tree
self.root = subtree.root
MHTNode.update_parents(subtree.root.parent)
self.elems = dict(self.elems.items() + subtree.elems.items())
self.sorted_elems = self.root.subtree_elems()
def _gestalt_batch_insert(self, left, right, new_elems):
""" Run all three phases of batch update in a row. Note that this is
mostly for testing/comparison purposes: the three phases are
separated for a reason! In particular, this function
is NOT safe for concurrent access.
Arguments:
left, right - the left and right bounds of the range to be inserted
into
new_elems - the list of new elements to be inserted
"""
lefti = bisect.bisect_left(self.sorted_elems, left)
righti = bisect.bisect_right(self.sorted_elems, right)
## can't currently handle the case where there are no elements in the
## given range
assert lefti < righti
lca = MHTNode.least_common_ancestor(
self.elems[self.sorted_elems[lefti]],
self.elems[self.sorted_elems[righti-1]])
range_elems = lca.subtree_elems()
for elem in new_elems:
MHT.batch_list_insert(elem, range_elems)
subtree = MHT.batch_node_insert(range_elems)
self.batch_update(subtree, lca)
def contains(self, elem):
""" Arguments:
self - the MHT to be queried
elem - the elem whose membership in the MHT is to be determined
Returns:
If elem is not in the MHT, returns false.
If elem is in the MHT, returns a list of hashes that can be combined
with elem's hash to compute the root hash of the tree for the
client to verify that the item actually is present.
NB: if self is a singleton MHT, this will return the empty list.
Therefore, `if mht.contains(x)` is an inadequate test, since it
will evaluate to false if None is returned or if x is the singleton
value within mht.
"""
if elem not in self.elems:
return None
node = self.elems[elem]
proof = []
parent = node.parent
while parent:
if node is parent.left:
proof.append((True, parent.right.hval))
elif node is parent.right:
proof.append((False, parent.left.hval))
node = parent
parent = node.parent
return proof
def range_query(self, lower, upper):
""" Return a verification object for all elements between lower
and upper (inclusive) in the tree.
Arguments:
lower - lower bound for the range query
upper - upper bound for the range query
Returns:
Verification object for the range (never None)
Assume the MHT contains at least two elements and that the
least & greatest elements stored are known to the client.
"""
lbound, li, ri, rbound = MHTNode.find_boundary(self.sorted_elems,
lower,
upper)
vo_elems = self.sorted_elems[li:ri+1]
return VO.new(lbound, vo_elems, rbound, self.root)
def valid(self):
""" Verifies that self is a valid Merkle hash tree
"""
for elem, leaf in self.elems.items():
if (leaf.hval != MHTUtils.hash(elem)):
raise InvalidMerkleTree(
"Each leaf's hash value must be its element's hash.")
if leaf.left or leaf.right:
raise InvalidMerkleTree(
'Each leaf must have no children')
if leaf.elem is None:
raise InvalidMerkleTree(
'Each leaf must have an element')
elif leaf is not self.elems[leaf.elem]:
raise InvalidMerkleTree(
'Each leaf element must point to itself')
if leaf._root_of() is not self.root:
raise InvalidMerkleTree(
'The root of each leaf must be the main root.')
if self.root.parent:
raise InvalidMerkleTree(
'The root must have no parent')
## Make sure the nodes are well-formed
self.root.valid()
pace/ads/merkle/mht_node.py
+228
View File
@@ -0,0 +1,228 @@
## **************
## Copyright 2014 MIT Lincoln Laboratory
## Project: PACE
## Authors: ZS
## Description: Inner nodes in MHTs
## Modifications:
## Date Name Modification
## ---- ---- ------------
## 18 Jul 2014 ZS Original file
## **************
import bisect
from pace.ads.merkle.mht_utils import MHTUtils
from pace.ads.merkle.eq import EqMixin
class MHTNode(object):
""" Class for nodes in a (binary) Merkle hash tree.
Instance variables:
hval - the hash value at this node. always exists.
left, right - the left and right children of this node.
If they are both None, this is a leaf.
They are always either both filled in or both None.
elem - If this is a leaf (i.e. if left & right are None), this
will point to the element the leaf corresponds to. Otherwise,
this is None.
parent - the parent of this node. if None, this is the root.
"""
def __init__(self, hval, left=None, right=None, elem=None):
self.hval = hval
self.left = left
self.right = right
self.parent = None
self.elem = elem
def __eq__(self, n2):
""" NB: Equality ignores hash functions and only makes sure they
both either have parents or don't
"""
if (self.hval == n2.hval
and self.left == n2.left
and self.right == n2.right
and self.elem == n2.elem
and ((self.parent and n2.parent) or
((not self.parent) and (not n2.parent)))):
return True
else:
return False
@classmethod
def leaf(cls, elem, hval=None):
""" Helper function for creating a leaf MHTNode
Arguments:
elem - the element this leaf points to
hval (optional) - the hash value of elem, if it's already been
computed.
"""
if hval:
h = hval
else:
h = MHTUtils.hash(elem)
return cls(h, elem=elem)
@classmethod
def node(cls, hval, left, right):
""" A helper function for defining non-leaf MHTNodes
Arguments:
hval - the hash value of this node
left - the left subtree
right - the right subtree
"""
node = cls(hval, left, right)
if isinstance(left, cls):
left.parent = node
if isinstance(right, cls):
right.parent = node
return node
@classmethod
def insert(cls, new_leaf, sibling_leaf):
old_parent = sibling_leaf.parent
new_node = cls.node(
MHTUtils.merge_hashes(sibling_leaf.hval, new_leaf.hval),
sibling_leaf,
new_leaf)
new_node.parent = old_parent
if new_node.parent is None:
return False
if new_node.parent.left is sibling_leaf:
new_node.parent.left = new_node
elif new_node.parent.right is sibling_leaf:
new_node.parent.right = new_node
MHTNode.update_parents(new_node.parent)
return True
@staticmethod
def update_parents(node):
current_node = node
while current_node:
current_node.hval = MHTUtils.merge_hashes(current_node.left.hval,
current_node.right.hval)
current_node = current_node.parent
@staticmethod
def find_boundary(elems, left_bound, right_bound):
""" Arguments:
elems - list of elements to find the left boundary of
left_bound - the least (according to <=) element that should
be included in the result set.
right_bound - the greatest (according to <=) element that should
be included in the result set.
Returns:
left_boundary - the rightmost element in elems that is less than
left_bound
left_index - the index of the least element in elems that
is >= left_bound
right_index - the index of the greatest element in elems that
is >= left_bound
right_boundary - the leftmost element in elems that is greater than
right_bound
"""
left_i = bisect.bisect_left(elems, left_bound)
right_i = bisect.bisect_right(elems, right_bound)
return elems[left_i-1], left_i, right_i-1, elems[right_i]
def valid(self):
if self.left and self.right:
if self.elem:
raise InvalidMerkleTree('Only leaves may point to elements')
if ((self.left.parent is not self) or
(self.right.parent is not self)):
raise InvalidMerkleTree(
'A node must be the parent of its children.')
if not self.hval:
raise InvalidMerkleTree(
'Each node must have a hash value')
# Children must be valid
self.left.valid()
self.right.valid()
# Your hash value must be the merge of your childrens' hash values
if (self.hval !=
MHTUtils.merge_hashes(self.left.hval, self.right.hval)):
raise InvalidMerkleTree(
"Each node's hash value must be the merge of its children's\
hash values.")
elif self.left or self.right:
# Children are all-or-nothing
raise InvalidMerkleTree('Each node must have zero or two children')
else:
if self.elem is None:
raise InvalidMerkleTree('Leaves must point to their elements.')
if (MHTUtils.hash(self.elem) !=
self.hval):
raise InvalidMerkleTree('Incorrect hash value stored at leaf')
if self.parent:
if not ((self.parent.left is self) or (self.parent.right is self)):
raise InvalidMerkleTree("You must be one of your parents' children")
@staticmethod
def least_common_ancestor(left, right):
current, current_history = left.parent, [left]
other, other_history = right, [right]
while current.parent or other.parent:
if current in other_history:
return current
current_history.append(current)
if other.parent is None:
current = current.parent
else:
tmp = current
current = other.parent
other = tmp
tmp = other_history
other_history = current_history
current_history = tmp
if current in other_history:
return current
return None
def _root_of(self):
""" Not particularly useful; just in place for debugging.
"""
current = self
while current.parent:
current = current.parent
return current
def subtree_elems(self):
""" Return all the elements in this subtree, in order.
"""
if self.elem is not None:
return [self.elem]
else:
return self.left.subtree_elems() + self.right.subtree_elems()
class InvalidMerkleTree(Exception):
def __init__(self, msg):
self.msg = msg
pace/ads/merkle/mht_node_test.py
+119
View File
@@ -0,0 +1,119 @@
## **************
## Copyright 2014 MIT Lincoln Laboratory
## Project: PACE
## Authors: ZS
## Description: Unit tests for MHT nodes
## Modifications:
## Date Name Modification
## ---- ---- ------------
## 21 Jul 2014 ZS Original file
## **************
import os
import sys
this_dir = os.path.dirname(os.path.abspath(__file__))
base_dir = os.path.join(this_dir, '../../..')
sys.path.append(base_dir)
from hashlib import sha256
from unittest import TestCase
import time
import random
from pace.ads.merkle.mht_node import MHTNode
from pace.ads.merkle.mht import MHT
from pace.common.pacetest import PACETestCase
class MerkleTests(PACETestCase):
def setUp(self):
random.seed(int(time.time()))
def test_leaf(self):
for i in range(0, self.num_iters):
elem = random.randint(0, 1000000)
l = MHTNode.leaf(elem)
self.assertEqual(l.elem, elem,
'leaf element is not what was expected')
self.assertEqual(l.parent, None, 'new nodes should have no parent')
self.assertEqual(l.left, None, 'leaves have no left child')
self.assertEqual(l.right, None, 'leaves have no right child')
hval = sha256(bytes(elem)).digest()
l2 = MHTNode.leaf(elem, hval=hval)
self.assertEqual(l.hval, hval,
'Wrong hval being written to leaf')
def test_node(self):
lchild = MHTNode.leaf(5)
rchild = MHTNode.leaf(6)
for i in range(0, self.num_iters):
elem = random.randint(0, 1000000)
hval = sha256(bytes(elem)).digest()
n = MHTNode.node(hval, lchild, rchild)
self.assertEqual(n.hval, hval,
'Wrong hval written to node')
self.assertEqual(n.left, lchild, 'Wrong left child in node')
self.assertEqual(n.right, rchild, 'Wrong right child in node')
self.assertEqual(n.elem, None, 'Nodes should contain no element')
def test_find_boundary(self):
elems = [0, 2, 4, 6, 8, 10, 12, 14, 16]
lower, li, ri, upper = MHTNode.find_boundary(elems, 5, 11)
self.assertEqual(lower, 4, 'Invalid lower boundary returned')
self.assertEqual(elems[li], 6,
'Invalid first element of range returned')
self.assertEqual(upper, 12,
'Invalid upper boundary returned: %d' %upper)
self.assertEqual(elems[ri], 10,
'Invalid first element of range returned')
lower, li, ri, upper = MHTNode.find_boundary(elems, 6, 10)
self.assertEqual(lower, 4, 'Invalid lower boundary returned')
self.assertEqual(elems[li], 6,
'Invalid first element of range returned')
self.assertEqual(upper, 12, 'Invalid upper boundary returned')
self.assertEqual(elems[ri], 10,
'Invalid first element of range returned')
def test_lca(self):
elems = [random.randint(0, 100000000) for i in range(0, self.size)]
elems.sort()
mht = MHT.new(elems)
for i in range(0, self.num_iters):
leaf1 = mht.elems[random.choice(mht.sorted_elems)]
leaf2 = mht.elems[random.choice(mht.sorted_elems)]
while leaf2 is leaf1:
leaf2 = mht.elems[random.choice(mht.sorted_elems)]
lca = MHTNode.least_common_ancestor(leaf1, leaf2)
current = leaf1
parents1 = []
while current is not lca:
self.assertTrue(current)
parents1.append(current)
current = current.parent
current = leaf2
while current is not lca:
self.assertTrue(current)
self.assertTrue(current not in parents1)
current = current.parent
pace/ads/merkle/mht_server.py
+213
View File
@@ -0,0 +1,213 @@
## **************
## Copyright 2014 MIT Lincoln Laboratory
## Project: PACE
## Authors: ZS
## Description: MHT server/data publisher code
## Modifications:
## Date Name Modification
## ---- ---- ------------
## 21 Jul 2014 ZS Original file
## **************
import socket
import logging
from SocketServer import StreamRequestHandler, TCPServer, ThreadingMixIn
from threading import Thread
from base64 import b64encode, b64decode
from pace.ads.merkle.mht import MHT
from pace.ads.merkle.vo import VO
def _recv(sfile, loc=''):
return sfile.readline().strip()
def _recvi(sfile, i, loc=''):
return [_recv(sfile, loc) for x in range(0, i)]
def _send(wfile, msg, loc=''):
total_msg = msg + '\n'
wfile.write(total_msg)
wfile.flush()
def _sendi(wfile, msgs, loc=''):
for msg in msgs:
_send(wfile, msg, loc)
def _batch_iterate(rfile):
while True:
x = _recv(rfile)
if x == 'FINISHED_BATCH_INSERT':
return
else:
yield int(x)
def _recv_iterate(rfile, name):
while True:
x = _recv(rfile)
if x == name:
return
else:
yield x
class MHTHandler(StreamRequestHandler):
def handle(self):
# Initialize the MHT
# Can have a persistent MHT once batch insertion has been
# implemented.
elems = [int(elem) for elem in _recv_iterate(self.rfile, 'END_ELEMS')]
mht = MHT.new(elems)
while True:
task = self.rfile.readline().strip()
if task == 'QUERY':
raw_elem = _recv(self.rfile, 'handler')
elem = int(raw_elem)
proof = mht.contains(elem)
sproof = MHTHandler.serialize_query_result(proof)
_send(self.wfile, sproof, 'handler')
elif task == 'RANGE_QUERY':
raw_lower, raw_upper = _recvi(self.rfile, 2, 'handler')
lower, upper = int(raw_lower), int(raw_upper)
vo = mht.range_query(lower, upper)
_send(self.wfile, vo.serialize(), 'handler')
elif task == 'BATCH_INSERT':
mht.batch_insert(_batch_iterate(self.rfile))
elif task == 'PING':
_send(self.wfile, 'PONG')
elif task == 'QUIT':
return
@staticmethod
def serialize_query_result(qr):
""" Serialization protocol for query result lists.
The result of a query is one of the following:
None - return the string 'None'
list of (flag, hval) pairs - transform each boolean flag into its
string with the str() method, then base 64 encode each hash
value. Put each pair together with the ';' delimiter, then
join the entire list with the ',' delimiter.
"""
if qr is None:
return 'None'
return ','.join(['%s;%s' %(str(f), b64encode(h)) for f, h in qr])
class SocketMHTServer(ThreadingMixIn, TCPServer):
""" An MHT server that serves the MHT over a socket, freeing
the client from having to store anything more than a
handle on the server to send & receive information to & from.
(this will become more interesting once it starts to serve the same
MHT to multiple clients)
"""
pass
class SocketMHTClient(object):
def __init__(self, sock):
self.sock = sock
self.rfile = sock.makefile('r')
self.wfile = sock.makefile('w')
def __enter__(self):
return self
def __exit__(self, type, value, tb):
_send(self.wfile, 'QUIT')
self.rfile.close()
self.wfile.close()
self.sock.close()
@staticmethod
def deserialize_query_result(qr):
if qr == 'None':
return None
return [(f == 'True', b64decode(h))
for f, h in [s.split(';') for s in qr.split(',')]]
@staticmethod
def new(elems, host='localhost', port=9999):
""" Creates a connection to a server and initializes it. Assumes
that the server's MHT has not already been initialized.
"""
s = SocketMHTClient.connect(host, port)
s.initialize(elems)
return s
def initialize(self, elems):
""" Initialize the server being connected to with the elements
elems. Should be called exactly once per server.
"""
_sendi(self.wfile, [str(elem) for elem in elems])
_send(self.wfile, 'END_ELEMS')
@staticmethod
def connect(host='localhost', port=9999):
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect((host, port))
return SocketMHTClient(sock)
def query(self, elem):
""" Query to see whether an element is in the MHT.
Arguments:
self - the MHT object
elem - the object to query
Returns:
None if elem was not contained in the MHT, or a proof
that it was contained in the tree.
"""
_send(self.wfile, 'QUERY')
_send(self.wfile, str(elem), 'client')
proof = _recv(self.rfile, 'client')
return SocketMHTClient.deserialize_query_result(proof)
def range_query(self, lower, upper):
""" Query a range of elements beteween lower and upper (inclusive)
Arguments:
self - the MHT object
lower - the (inclusive) lower bound of the range
upper - the (inclusive) upper bound of the range
Returns:
vo - a verification object (as in vo.py) for the given range
"""
_send(self.wfile, 'RANGE_QUERY')
_sendi(self.wfile, [str(lower), str(upper)])
vo = _recv(self.rfile, 'client')
return VO.deserialize(vo)
def batch_insert(self, expected_initial_root, elems, least, greatest):
vo = self.range_query(least, greatest)
vo.verify(least, greatest, expected_initial_root)
_send(self.wfile, 'BATCH_INSERT')
for elem in elems:
_send(self.wfile, str(elem))
vo.insert(elem)
_send(self.wfile, 'FINISHED_BATCH_INSERT')
return vo.root.hval
def ping(self):
""" Ping a server. Mostly for testing/benchmarking purposes; assumes
it can't respond until it's finished with its previous task.
"""
_send(self.wfile, 'PING')
_recv(self.rfile)
pace/ads/merkle/mht_server_test.py
+175
View File
@@ -0,0 +1,175 @@
## **************
## Copyright 2014 MIT Lincoln Laboratory
## Project: PACE
## Authors: ZS
## Description: Unit tests for MHT servers
## Modifications:
## Date Name Modification
## ---- ---- ------------
## 25 Jul 2014 ZS Original file
## **************
import os
import sys
this_dir = os.path.dirname(os.path.abspath(__file__))
base_dir = os.path.join(this_dir, '../../..')
sys.path.append(base_dir)
from unittest import TestCase
import random
import socket
import time
from threading import Thread
from pace.ads.merkle.mht import MHT
from pace.ads.merkle.mht_utils import MHTUtils
from pace.ads.merkle.mht_server import SocketMHTServer, MHTHandler, SocketMHTClient
from pace.common.pacetest import PACETestCase
class MHTServerTest(PACETestCase):
@staticmethod
def run_server(server):
server.serve_forever()
def setUp(self):
# Need the extra randomness to make sure it doesn't generate the
# same seed on different tests.
random.seed(int(time.time()) + random.randint(0, 1000))
self.host = 'localhost'
self.port = random.randint(9000, 9999)
# Try 10 times to get a valid port; if not, will fail. If this
# fails, check to make sure ports 9000-9999 are not in use.
for _ in xrange(10):
try:
self.server = SocketMHTServer((self.host, self.port),
MHTHandler)
break
except socket.error:
self.port = random.randint(9000, 9999)
self.thread = Thread(group=None, target=MHTServerTest.run_server,
name=None, args=[self.server], kwargs={})
self.thread.daemon = True
self.thread.start()
print 'Starting test',
self.start_time = time.time()
def tearDown(self):
self.end_time = time.time()
print 'Test finished. Time taken: %s' %str(self.end_time - self.start_time)
self.server.shutdown()
self.thread.join()
def test_remote_query(self):
print 'single element query'
for i in range(0, self.num_iters):
elems = [random.randint(0, 1000000000)
for i in range(0, self.size)]
elems.sort()
mht = MHT.new(elems)
with SocketMHTClient.new(elems, self.host, self.port) as client:
for j in range(0, self.num_iters):
elem = random.choice(elems)
good_result = mht.contains(elem)
test_result = client.query(elem)
self.assertEqual(good_result, test_result,
'Unequal proofs returned')
def test_remote_range_query(self):
print 'range query'
total_time = 0
for i in range(0, self.num_iters):
elems = [random.randint(0, 1000000000)
for i in range(0, self.size)]
elems.sort()
mht = MHT.new(elems)
start = time.time()
with SocketMHTClient.new(elems, self.host, self.port) as client:
## Get valid lower & upper bounds with enough room between
## them (about a third of the list) for a substantial range
## query
lower = random.choice(elems[1:self.size/3])
upper = random.choice(elems[2*self.size/3:-1])
self.assertEqual(mht.range_query(lower, upper),
client.range_query(lower, upper))
end = time.time()
total_time = total_time + (end - start)
print 'Average time taken: %s' %str(total_time / self.num_iters)
def test_remote_batch_insert(self):
print 'batch insert'
total_time = 0
other_time = 0
complete_time = 0
loop_time = 0
init_time = time.time()
for i in range(0, self.num_iters):
loop_start = time.time()
start = time.time()
elems = [random.randint(0, 100000000) for i in range(0, self.size)]
elems.sort()
mht = MHT.new(elems)
old_root = mht.root.hval
self.assertEqual(len(mht.sorted_elems), self.size)
## Generate a small number (in this case, n/10) of random elements
## between the minimum and maximum elements (not including the
## boundary elements) to insert.
new_elems = [random.randint(elems[1], elems[-2])
for i in range(0, self.size/10)]
min_ne = min(new_elems)
max_ne = max(new_elems)
vo = mht.range_query(min_ne, max_ne)
for elem in new_elems:
vo.insert(elem)
mht.batch_insert(new_elems)
end = time.time()
start = time.time()
with SocketMHTClient.new(elems, self.host, self.port) as client:
end = time.time()
start = time.time()
new_root = client.batch_insert(old_root, new_elems,
min_ne, max_ne)
client.ping()
end = time.time()
total_time = total_time + (end - start)
start = time.time()
self.assertEqual(new_root, vo.root.hval)
self.assertEqual(mht.root.hval, new_root)
self.assertEqual(len(mht.sorted_elems),
self.size + (self.size/10))
for elem in new_elems:
proof = client.query(elem)
self.assertEqual(proof, mht.contains(elem))
self.assertTrue(MHTUtils.verify(new_root, elem, proof),
'Returned proof does not verify that \
inserted element is in the tree')
end = time.time()
other_time = other_time + (end - start)
loop_end = time.time()
loop_time = loop_time + (loop_end - loop_start)
end_time = time.time()
print 'Total batch insert time: %s' %str(end_time - init_time)
print 'Time for batch inserts: %s' %str(total_time)
print 'And time to do test queries: %s' %str(other_time)
print 'Time in loop: %s' %str(loop_time)
print 'Time accounted for: %s' %str(total_time + other_time)
pace/ads/merkle/mht_test.py
+268
View File
@@ -0,0 +1,268 @@
## **************
## Copyright 2014 MIT Lincoln Laboratory
## Project: PACE
## Authors: ZS
## Description: Unit tests for MHT library
## Modifications:
## Date Name Modification
## ---- ---- ------------
## 17 Jul 2014 ZS Original file
## **************
import os
import sys
this_dir = os.path.dirname(os.path.abspath(__file__))
base_dir = os.path.join(this_dir, '../../..')
sys.path.append(base_dir)
from unittest import TestCase
import time
import random
from pace.ads.merkle.mht import MHT, MHTInsertionException
from pace.ads.merkle.mht_utils import MHTUtils
from pace.common.pacetest import PACETestCase
class MerkleTests(PACETestCase):
def setUp(self):
random.seed(int(time.time()))
def test_valid(self):
""" Ensure that we only create valid hash trees.
"""
for i in range(0, self.num_iters):
elems = [random.randint(0, 1000000000)
for i in range(0, self.size/2)]
elems.sort()
mht = MHT.new(elems)
try:
mht.valid()
except InvalidMerkleTree as e:
self.assertTrue(False, 'Error: %s' %e.msg)
def test_membership(self):
""" Ensure that every element of the hash tree is reflected as in it.
"""
for i in range(0, self.num_iters):
elems = [random.randint(0, 100000000) for i in range(0, 100)]
elems.sort()
mht = MHT.new(elems)
for elem in elems:
self.assertTrue(mht.contains(elem),
'MHT claims not to have element %d' %elem)
def test_non_membership(self):
""" Ensure that non-elements of the MHT are not in it
"""
for i in range(0, self.num_iters):
elems = [random.randint(0, 100000000) for i in range(0, 100)]
elems.sort()
mht = MHT.new(elems)
self.assertEqual(mht.contains(-1), None,
"MHT thinks it contains -1 (it doesn't)")
self.assertEqual(mht.contains(100000001), None,
"MHT thinks it contains 100000001 (it doesn't)")
def test_proofs(self):
""" Ensure that the returned membership proofs are valid.
"""
for i in range(0, self.num_iters):
elems = [random.randint(0, 100000000) for i in range(0, 100)]
elems.sort()
mht = MHT.new(elems)
for elem in elems:
proof = mht.contains(elem)
self.assertTrue(MHTUtils.verify(mht.root.hval, elem, proof),
'Returned proof does not verify that %d is in \
the tree' %elem)
def test_partial_insert(self):
for i in range(0, self.num_iters):
elems = [random.randint(0, 100000000) for i in range(0, 100)]
elems.sort()
index = random.choice(range(1, 99))
left = elems[index-1]
right = elems[index]
elem = random.choice(range(left, right))
new_elems, i = MHT.partial_insert(elems, elem)
self.assertEqual(index, i, 'Error: element inserted at wrong index')
self.assertEqual(new_elems,
elems[:index] + [elem] + elems[index:],
'Error: failure to insert element into list')
def test_left_insert_fails(self):
for i in range(0, self.num_iters):
elems = [random.randint(0, 100000000) for i in range(0, 100)]
elems.sort()
elems = elems[1:]
index = random.choice(range(1, 99))
left = elems[index-1]
right = elems[index]
mht = MHT.new(elems)
elem = 0
try:
mht.insert(elem)
except MHTInsertionException as e:
return
raise Error('Not supposed to succeed on an out-of-range insert.')
def test_valid_insert(self):
for i in range(0, self.num_iters):
elems = [random.randint(0, 100000000) for i in range(0, 100)]
elems.sort()
mht = MHT.new(elems)
root = mht.root.hval
index = random.choice(range(1, 99))
left = elems[index-1]
right = elems[index]
elem = random.choice(range(left, right))
vo = mht.insert(elem)
mht.valid()
vo.verify(left+1, right-1, root)
def test_insert_contains(self):
for i in range(0, self.num_iters):
elems = [random.randint(0, 100000000) for i in range(0, 100)]
elems.sort()
mht = MHT.new(elems)
root = mht.root.hval
index = random.choice(range(1, 99))
left = elems[index-1]
right = elems[index]
elem = random.choice(range(left, right))
vo = mht.insert(elem)
proof = mht.contains(elem)
self.assertTrue(MHTUtils.verify(mht.root.hval, elem, proof),
'Returned proof does not verify that inserted \
element is in the tree')
def test_batch_insert(self):
for i in range(0, self.num_iters):
elems = [random.randint(0, 100000000) for i in range(0, 300)]
elems.sort()
mht = MHT.new(elems)
root = mht.root.hval
new_elems = [random.randint(elems[0], elems[-1])
for i in range(0, 50)]
mht.batch_insert(new_elems)
for elem in new_elems:
proof = mht.contains(elem)
self.assertTrue(MHTUtils.verify(mht.root.hval, elem, proof),
'Returned proof does not verify that inserted \
element is in the tree')
def test_gestalt_batch_insert(self):
for i in range(0, self.num_iters):
elems = [random.randint(0, 100000000) for i in range(0, 300)]
elems.sort()
mht = MHT.new(elems)
control_mht = MHT.new(elems)
root = mht.root.hval
left = random.choice(elems[1:self.size/3])
right = random.choice(elems[2*self.size/3:-1])
new_elems = [random.randint(left, right)
for i in range(0, 50)]
mht._gestalt_batch_insert(left, right, new_elems)
mht.valid() ## make sure it's valid
control_mht.batch_insert(new_elems)
self.assertEqual(mht.sorted_elems, control_mht.sorted_elems)
for elem in new_elems:
proof = mht.contains(elem)
self.assertTrue(proof is not None,
'Added element not in tree')
self.assertTrue(MHTUtils.verify(mht.root.hval, elem, proof))
def test_gestalt_batch_insert_again(self):
for i in range(0, self.num_iters):
elems = [random.randint(1, 99999999)
for i in range(0, 98)]
elems.sort()
elems = [0] + elems + [100000000]
mht = MHT.new(elems)
for i in range(1, 10):
elems = [random.randint(1, 99999999)
for i in range(1, 100)]
mht._gestalt_batch_insert(min(elems), max(elems), elems)
mht.valid()
def test_batch_insert_vo(self):
for i in range(0, self.num_iters):
elems = [random.randint(0, 100000000) for i in range(0, 300)]
elems.sort()
mht = MHT.new(elems)
root = mht.root.hval
new_elems = [random.randint(elems[0], elems[-1])
for i in range(0, 50)]
vo = mht.range_query(min(new_elems), max(new_elems))
mht.batch_insert(new_elems)
for elem in new_elems:
proof = mht.contains(elem)
vo.insert(elem)
self.assertTrue(MHTUtils.verify(mht.root.hval, elem, proof),
'Returned proof does not verify that inserted \
element is in the tree')
self.assertEqual(vo.root.hval, mht.root.hval)
def test_both_inserts(self):
for i in range(0, self.num_iters):
elems = [random.randint(0, 100000000) for i in range(0, 300)]
elems.sort()
mht = MHT.new(elems)
root = mht.root.hval
new_elems = [random.randint(elems[0], elems[-1])
for i in range(0, 50)]
vo = mht.range_query(min(new_elems), max(new_elems))
mht.batch_insert(new_elems)
for elem in new_elems:
vo.insert(elem)
self.assertEqual(vo.root.hval, mht.root.hval)
pace/ads/merkle/mht_utils.py
+44
View File
@@ -0,0 +1,44 @@
## **************
## Copyright 2014 MIT Lincoln Laboratory
## Project: PACE
## Authors: ZS
## Description: Merkle hash tree utility functions
## (that server & client can run)
## Modifications:
## Date Name Modification
## ---- ---- ------------
## 18 Jul 2014 ZS Original file
## **************
from hashlib import sha256
class MHTUtils(object):
@staticmethod
def hash(elem):
return sha256(bytes(elem)).digest()
@staticmethod
def merge_hashes(h1, h2):
return sha256(h1 + h2).digest()
@staticmethod
def verify(root_hval, elem, proof):
""" Arguments:
root_hval - the hash value stored at the root of the MHT
elem - the elem that is (allegedly) in the MHT
proof - the alleged proof that elem is in the MHT
Returns:
True if proof is a proof that elem is in the MHT with root_hval
False otherwise
"""
hval = MHTUtils.hash(elem)
for is_left, other_side in proof:
if is_left:
hval = MHTUtils.merge_hashes(hval, other_side)
else:
hval = MHTUtils.merge_hashes(other_side, hval)
return hval == root_hval
pace/ads/merkle/vo.py
+223
View File
@@ -0,0 +1,223 @@
## **************
## Copyright 2014 MIT Lincoln Laboratory
## Project: PACE
## Authors: ZS
## Description: Verification objects for MHTs
## Modifications:
## Date Name Modification
## ---- ---- ------------
## 18 Jul 2014 ZS Original file
## **************
from base64 import b64encode, b64decode
import bisect
from pace.ads.merkle.eq import EqMixin
from pace.ads.merkle.vo_node import VONode
from pace.ads.merkle.hash_node import HashNode
from pace.ads.merkle.mht_utils import MHTUtils
class VO(EqMixin):
""" Verification objects for MHTs. Structure is similar to a MHT,
but may contain hashes to replace subtrees that are not necessary
for the current verification.
"""
def serialize(self):
""" Create a string out of a VO
NB: assumes the hash function used is standard to avoid
having to serialize the hash class.
"""
return '%s<=>%s<=>%s' %(str(self.left), str(self.right),
self.root.serialize())
@staticmethod
def deserialize(s):
sleft, sright, sroot = s.split('<=>')
left, right = int(sleft), int(sright)
root, leaves = VONode.deserialize(sroot)
return VO(left, right, root, leaves)
def __init__(self, left, right, root, leaves):
""" Defines a verification object for a query on mht that
returns elements elems, with left and right boundaries
as given.
"""
self.left = left
self.right = right
self.root = root
self.leaves = leaves
@staticmethod
def new(left, elems, right, mht_root):
""" Defines a verification object for a query on mht that
returns elements elems, with left and right boundaries
as given.
"""
all_elems = [left] + elems + [right]
root, leaves = VO.build_node(mht_root, all_elems)
return VO(left, right, root, leaves)
@staticmethod
def build_node(mht_node, elems):
if mht_node.left and mht_node.right:
left, l_leaves = VO.build_node(mht_node.left, elems)
right, r_leaves = VO.build_node(mht_node.right, elems)
# Check if we can collapse this to just a hash value
if not (isinstance(left, VONode) or isinstance(right, VONode)):
return HashNode(mht_node.hval), []
new_node = VONode.node(mht_node.hval, left, right)
return new_node, l_leaves + r_leaves
else:
# It's a leaf, so see if it's in the set or not
if mht_node.elem in elems:
new_node = VONode.leaf(mht_node.elem, mht_node.hval)
return new_node, [new_node]
else:
return HashNode(mht_node.hval), []
def verify_insertion(self, old_root_hval, left, right, elem):
""" Verify that 'self' is a proof that 'elem' was correctly inserted
into an MHT between 'left' and 'right', whose old root value was
old_root_hval.
Arguments:
old_root_hval - the root hash of the MHT before the insertion
left, right - the elements on the left & right sides of the element
post-insertion
elem - the element that was inserter
Returns:
root_hval - the new root hash value of the MHT after the insertion
"""
# MHT.insert() needs to return a VO for the empty range between left &
# right and the new root value; that way, we can check the VO to make
# sure it's consistent with what we expect, then insert it into the VO
# to make sure we get back the correct hash value.
#
# There might even be a way to do both of these at the same time? But
# we should leave optimizations like that for later.
self.verify(left, right, old_root_hval)
return self.insert(elem)
def insert(self, elem):
""" Insert an element into a VO, returning the new root hash value.
"""
if elem <= self.left or elem >= self.right:
raise VerificationObjectException(
"Element given for insertion is not within the given range.")
just_elems = [leaf.elem for leaf in self.leaves]
i = bisect.bisect_left(just_elems, elem)
new_leaf = VONode.leaf(elem)
sibling_leaf = self.leaves[i-1]
x = VONode.insert(new_leaf, sibling_leaf)
self.leaves = self.leaves[:i] + [new_leaf] + self.leaves[i:]
return (x, self.root.hval)
def verify(self, left, right, root_hval):
""" Verify that 'self' (the verification object) does in fact
verify the range given. It's sent by the (untrusted) server,
so we must verify everything.
"""
leaves = self.leaves
if len(leaves) < 2:
raise VerificationObjectException(
'Leaves length must be at least 2 (actual: %d)' %len(leaves))
left_leaf = leaves[0]
right_leaf = leaves[-1]
leaves = leaves[1:-1]
## Check boundary conditions---minimum possible check on
## whether the results are valid
if left_leaf.elem >= left or right_leaf.elem <= right:
raise VerificationObjectException(
'Left and right boundary objects must be outside of range:\n\
Range: %s to %s\n\
Left object: %s, Right object: %s' %(str(left),
str(right),
str(left_leaf.elem),
str(right_leaf.elem)))
if not all(left <= leaf.elem <= right for leaf in leaves):
raise VerificationObjectException(
'All leaves must be within the left & right bounds')
if not all(leaves[i].elem <= leaves[i+1].elem
for i in range(0, len(leaves)-1)):
raise VerificationObjectException(
'Leaves must be sorted')
## Make sure everything hashes together correctly
if self.root.hval != root_hval:
raise VerificationObjectException(
'Root hash value does not match published value')
VO.verify_node(self.root, self.leaves, len(self.leaves))
@staticmethod
def verify_node(node, leaves, num_leaves):
if not isinstance(node, VONode):
# If leaves is neither empty nor full, there's a gap in the
# result set that means the server attempted to omit an element
if leaves and num_leaves != len(leaves):
raise VerificationObjectException(
'Incomplete VO detected---elements omitted in result set.')
return leaves
# need the extra check for non-none False objects
if node.elem is not None:
## It's a leaf
if node.hval != MHTUtils.hash(node.elem):
raise VerificationObjectException(
'Node hval must match element hval')
if node.elem != leaves[0].elem:
raise VerificationObjectException(
'Node element must be the first of the remaining leaves')
return leaves[1:]
elif not (VO.child_node_class(node.left) and
VO.child_node_class(node.right)):
raise VerificationObjectException(
'Nodes must have exactly zero or one children.')
else:
## It's an internal node
left_hval = node.left.hval
right_hval = node.right.hval
if not (MHTUtils.merge_hashes(left_hval, right_hval) ==
node.hval):
raise VerificationObjectException(
"Node hash value does not match its childrens' hvals")
remaining_leaves = VO.verify_node(node.left, leaves, num_leaves)
return VO.verify_node(node.right, remaining_leaves, num_leaves)
@staticmethod
def child_node_class(maybenode):
return isinstance(maybenode, VONode) or isinstance(maybenode, HashNode)
class VerificationObjectException(Exception):
def __init__(self, msg):
self.msg = msg
pace/ads/merkle/vo_node.py
+93
View File
@@ -0,0 +1,93 @@
## **************
## Copyright 2014 MIT Lincoln Laboratory
## Project: PACE
## Authors: ZS
## Description: Node class for VOs
## Modifications:
## Date Name Modification
## ---- ---- ------------
## 28 Jul 2014 ZS Original file
## **************
from base64 import b64encode, b64decode
from pace.ads.merkle.mht_node import MHTNode
from pace.ads.merkle.empty_node import EmptyNode
from pace.ads.merkle.hash_node import HashNode
class VONode(MHTNode):
def __init__(self, hval, left=None, right=None, elem=None):
if left is None:
left = EmptyNode()
if right is None:
right = EmptyNode()
super(VONode, self).__init__(hval, left, right, elem)
def set_parent(self, parent):
self.parent = parent
def serialize(node, depth=0):
""" Create a (reversible) string out of a node
NB: assumes the hash function used is standard to avoid
having to serialize the hash class.
"""
if not isinstance(node, MHTNode):
if node:
return b64encode(node)
else:
return str(node)
separator = ',' + str(depth) + ','
return '(%s)' %(separator.join([b64encode(node.hval),
node.left.serialize(depth+1),
node.right.serialize(depth+1),
str(node.elem)]))
@staticmethod
def deserialize(s, depth=0):
if s == 'None':
return (EmptyNode(), [])
elif s[0] != '(':
# It's a hash value
return (HashNode(b64decode(s)), [])
separator = ',' + str(depth) + ','
shval, sleft, sright, selem = s[1:-1].split(separator)
if shval == 'None':
hval = None
else:
hval = b64decode(shval)
if sleft == 'None':
left = EmptyNode()
else:
(left, left_leaves) = VONode.deserialize(sleft, depth + 1)
if sright == 'None':
right = EmptyNode()
else:
(right, right_leaves) = VONode.deserialize(sright, depth + 1)
if selem == 'None':
elem = None
else:
elem = int(selem)
node = VONode(hval, left, right, elem)
left.set_parent(node)
right.set_parent(node)
if elem is not None:
leaves = [node]
else:
leaves = left_leaves + right_leaves
return node, leaves
pace/ads/merkle/vo_test.py
+220
View File
@@ -0,0 +1,220 @@
## **************
## Copyright 2014 MIT Lincoln Laboratory
## Project: PACE
## Authors: ZS
## Description: Unit tests for verification objects
## Modifications:
## Date Name Modification
## ---- ---- ------------
## 21 Jul 2014 ZS Original file
## **************
import os
import sys
this_dir = os.path.dirname(os.path.abspath(__file__))
base_dir = os.path.join(this_dir, '../../..')
sys.path.append(base_dir)
from hashlib import sha256
from unittest import TestCase
import time
import random
from pace.ads.merkle.mht_utils import MHTUtils
from pace.ads.merkle.mht import MHT
from pace.ads.merkle.vo import VO, VerificationObjectException
from pace.ads.merkle.vo_node import VONode
from pace.ads.merkle.empty_node import EmptyNode
from pace.common.pacetest import PACETestCase
class VOTest(PACETestCase):
def setUp(self):
random.seed(int(time.time()))
# Setup for small, hand-made tests
self.elems = [0, 5, 6, 10]
self.mht = MHT.new(self.elems)
self.lower = 1
self.upper = 9
self.vo = self.mht.range_query(self.lower, self.upper)
def test_range(self):
""" Ensure that range queries work correctly
"""
for i in range(0, self.num_iters):
elems = [random.randint(0, 100000000) for i in range(0, 1000)]
elems.sort()
mht = MHT.new(elems)
lower = random.choice(elems[1:200])
upper = random.choice(elems[500:999])
vo = mht.range_query(lower, upper)
try:
vo.verify(lower, upper, mht.root.hval)
except VerificationObjectException as e:
self.assertTrue(False, 'Error: %s' %e.msg)
def test_empty_vo_fails(self):
elems = self.elems
mht = self.mht
lower = self.lower
upper = self.upper
vo = self.vo
# Make sure it won't verify an empty VO
bad_vo = VO(0, 10, EmptyNode(), [])
passed = False
try:
bad_vo.verify(lower, upper, mht.root.hval)
except VerificationObjectException as e:
passed = True
self.assertTrue(passed, 'Should not verify an empty VO')
def test_bogus_root_hash(self):
elems = self.elems
mht = self.mht
lower = self.lower
upper = self.upper
vo = self.vo
# Make sure it won't verify a bogus root hash
bad_root_hval = sha256(b'foobar').digest()
passed = False
try:
vo.verify(lower, upper, bad_root_hval)
except VerificationObjectException as e:
passed = True
self.assertTrue(passed, 'Should not verify a bogus root hash')
def test_omitted_element(self):
elems = self.elems
mht = self.mht
lower = self.lower
upper = self.upper
vo = self.vo
# Make sure it won't let us omit an element
leaf0 = VONode.leaf(0)
leaf5 = VONode.leaf(5)
leaf6 = VONode.leaf(6)
bad_leaf6 = leaf6.hval
leaf10 = VONode.leaf(10)
node05 = VONode.node(MHTUtils.merge_hashes(leaf0.hval, leaf5.hval),
leaf0, leaf5)
node610 = VONode.node(MHTUtils.merge_hashes(bad_leaf6, leaf10.hval),
bad_leaf6, leaf10)
bad_node610 = VONode.node(
MHTUtils.merge_hashes(bad_leaf6, leaf10.hval), bad_leaf6, leaf10)
root = VONode.node(
MHTUtils.merge_hashes(node05.hval, bad_node610.hval),
node05,
bad_node610)
bad_vo = VO(0, 10, root, [leaf0, leaf5, leaf10])
passed = False
assert root.hval == mht.root.hval
try:
bad_vo.verify(lower, upper, mht.root.hval)
except VerificationObjectException as e:
passed = True
self.assertTrue(passed, 'Should not verify an incomplete tree')
def test_too_long_range(self):
elems = self.elems
mht = self.mht
lower = self.lower
upper = self.upper
vo = self.vo
# Make sure it won't let us return too many elements
passed = False
try:
vo.verify(1, 5, mht.root.hval)
except VerificationObjectException as e:
passed = True
self.assertTrue(passed, 'Should not verify an overextended range')
def test_stringify_leaf(self):
for i in range(0, self.num_iters):
elem = random.randint(0, 1000)
l = VONode.leaf(elem)
s = l.serialize()
new_l, leaves = VONode.deserialize(s)
self.assertEqual(l, new_l,
'Taking a leaf to string and back must return an equivalent \
object')
def test_stringify_node(self):
for i in range(0, self.num_iters):
elem1 = random.randint(0, 1000)
elem2 = random.randint(0, 1000)
l1, l2 = VONode.leaf(elem1), VONode.leaf(elem2)
n = VONode.node(MHTUtils.merge_hashes(l1.hval, l2.hval), l1, l2)
s = n.serialize()
new_n, leaves = VONode.deserialize(s)
self.assertEqual(n, new_n,
'Taking a node to string and back must return an equivalent \
object')
def test_stringify(self):
for i in range(0, self.num_iters):
elems = [random.randint(0, 100000000) for i in range(0, 1000)]
elems.sort()
mht = MHT.new(elems)
lower = random.choice(elems[1:200])
upper = random.choice(elems[500:-1])
vo = mht.range_query(lower, upper)
assert isinstance(vo, VO)
serialized = vo.serialize()
new_vo = VO.deserialize(serialized)
self.assertEqual(vo, new_vo,
'Error: to_string & from_string should return an \
equivalent object')
def test_vo_insert(self):
for i in range(0, self.num_iters):
elems = [random.randint(0, 100000000) for i in range(0, 100)]
elems.sort()
mht = MHT.new(elems)
root = mht.root.hval
index = random.choice(range(1, 99))
left = elems[index-1]
right = elems[index]
elem = random.choice(range(left, right))
vo = mht.insert(elem)
mht.valid()
vo.verify(left+1, right-1, root)
x, new_root = vo.insert(elem)
vo.verify(left+1, right-1, new_root)
self.assertEqual(vo.root.hval, mht.root.hval)
pace/ads/skiplist/.authskiplist.py.swp
BIN
View File
Binary file not shown.
pace/ads/skiplist/.skiplist.py.swp
BIN
View File
Binary file not shown.
pace/ads/skiplist/README.md
+218
View File
@@ -0,0 +1,218 @@
# Authenticated Skip Lists
This is a Python Authenticated Data Structure (ADS) library for Accumulo,
specifically implementing authenticated skip lists. ADSs allow users to verify
the correctness of entire queries from an untrusted database. To compare,
digital signatures only allow verification of one element of the query at a
time, and do not provide a mechanism for determining whether elements were
omitted from a query.
This library includes three primary components:
- An implementation of skip lists, a random, balanced tree-like data structure
with expected `O(log(n))` insert and lookup operations.
- An implementation of authenticated skip lists, a Merkle tree-like data
structure that is implemented over a skip list rather than a binary tree.
- A prototype embedding of our authenticated skip list implementation into
Accumulo. The two goals behind this prototype were to determine a schema to
use to store an ADS computed over an Accumulo table, and to implement that
schema reusing as much of our existing authenticated skip list code as
possible.
In this document, we focus on how a user would install, run, configure, and use
the authenticated skip list embedding to verify queries from an untrusted
Accumulo server.
## Installation
### Dependencies
This project uses the [pyaccumulo](https://pypi.python.org/pypi/pyaccumulo)
library in order to interface with Accumulo. Our code has been written and
tested using pyaccumulo version 1.5.0.6.
## Use
In addition to the authenticated skip list code, there are several classes that
support our skip list definition. We now describe these support classes, their
roles, and how to use them, before going on to explain how to use our skip list
embedding.
### Elements
In `pace.ads.skiplist.elemclass`, we define an abstract interface,
`BaseElem`, for the type of elements stored by a skip list, along with several
useful implementations oft his interface. The definition of `BaseElem`,
including its abstract methods that implementing classes must define, is as
follows:
```python
class BaseElem:
__metaclass__ = ABCMeta
def __init__(self, key):
self.key = key
def __cmp__(self, other):
...
@abstractmethod
def serialize(self):
pass
@abstractmethod
def deserialize(s):
pass
```
This interface requires two capabilities: comparison of different elements of
the same class, and (de)serialization of elements within a class. By default, a
`__cmp__()` method and a `serialize()` method are implemented using the `key`
field's comparison and to-string methods, respectively, but these default
implementations can be overridden.
The `elemclass` package also contains several useful implementations of the
`BaseElem` interface. Two of them, `IntElem` and `StrElem`, are simple wrappers
around Python's string and integer types, allowing them to be stored in skip
lists. The other two, `AccumuloKey` and `AccumuloEntry`, are for constructive
skip lists over Accumulo cells.
### Coins
In `pace.ads.skiplist.coin`, we define an abstraction, `BaseCoin`, for
simulating coin flips, i.e. generating `True` and `False` values randomly or
pseudorandomly. This class's interface has one function, as follows:
```python
def flip(self, *args):
return random.randint(0,1) == 1
```
This uses Python's `random` module to return either `True` or `False`, each with
probability 0.5. A `BaseCoin` can be used as follows:
```python
>>> from pace.ads.skiplist.coin import BaseCoin
>>> coin = BaseCoin()
>>> flips = [coin.flip() for _ in xrange(100)]
>>> flips[:10]
[False, False, True, False, False, False, False, False, True, False]
```
In addition to this basic interface, we provide several extensions with
different capabilities, and in some cases, extra methods, for different use
cases:
- `RecordedCoin`: records the result of each flip into a list that can be
accessed with the `read()` method, which will then reset the record to the
empty list.
- `PrefixCoin`: takes as input to its constructor a prefix of coin flip results
that will be drawn from before switching to randomly generating coin flips as
in `BaseCoin`. The stored prefix can be extended by calling the `extend()`
method with the suffix to extend it with.
- `RecordedPrefixCoin`: combines the functionalities of `RecordedCoin` and
`PrefixCoin`.
- `SeededCoin`: generates its random numbers based on a given seed value,
keeping track of the resulting state to make sure it will always generate the
same sequence of values as another coin with the same seed.
- `HashCoin`: deterministically generates coin flips based on iterative
application of a hash function to the element being inserted into the skip
list. This adds an extra argument to the hash function: one now calls
`hashcoin.flip(hash)`, where `hash` is the hash value of the element being
inserted into the skip list.
A primary motivation behind these subclasses is the ability to generate the same
coin flips on a client and server. We support three ways to do this:
- The client uses `RecordedCoin` and sends the results to the server, which
appends them on to its `PrefixCoin`. This can also work if one or both parties
uses a `RecordedPrefixCoin`.
- Both parties use a `SeededCoin` and agree on the seed to use beforehand.
- Both parties use a `HashCoin`. This can be done with no additional
communication, since the flips are randomly generated from each element in the
skip list in isolation.
We recommend using `HashCoin`, the third of these options. In situations where
this option's determinism is undesirable, there is a tradeoff: `SeededCoin`
requires no communication, but the cost of storing and restoring the RNG's state
adds considerable overhead; conversely, recording and pulling from a prefix is a
relatively fast operation, but the communication cost of sending the recorded
values across a network connection may be prohibitive.
### Embedded Skip Lists
To create an authenticated skip list embedded in Accumulo, one calls the `new()`
method from the `EmbeddedSkipList` class, located in the
`pace.ads.skiplist.accumulo.embeddedskiplist` package:
```python
from pace.ads.skiplist.accumulo.embeddedskiplist import EmbeddedSkipList
ads = EmbeddedSkipList.new(elems, lbound, rbound, coin,
conn_info, table, elemclass)
```
This method's first four arguments are common to all of our skip list classes:
- `elems`: the list of elements to add to the skip list.
- `lbound`, `rbound`: the left and right boundaries of the skip list's range.
These should have the property that for all elements `e` in `elems`,
`lbound < e < rbound`, and that each element inserted into the list in the
future is also between `lbound` and `rbound`.
- `coin`: the coin class (as defined above) to use to generate the coin flips
for insertion into the skip list. We recommend using `HashCoin` for this, as
it automatically computes a skip list of the same structure for both server
and client without any extra input from the user.
The next two arguments are embedding-specific:
- `conn_info`: a named tuple object, called `ConnInfo`, containing the
information needed to connect to the Accumulo server in which the skip list is
being embedded. This tuple class is also defined in `embeddedskiplist.py`. The
fields of this named tuple are, in order, `hostname`, `port`, `username`, and
`password`.
- `table`: the name of the table in which to store the skip list. This field
should be different for each authenticated skip list stored in the Accumulo
server.
Finally, the last argument is the class of elements stored in the skip list.
In order to perform a range query over the skip list, one uses the `SkipListVO`
class, defined in `pace.ads.skiplist.skiplistvo`. This computes a
verification object for the given range client-side using the information
embedded in the server. In particular, the syntax for a range query over the
skip list `ads` works as follows:
```python
from pace.ads.skiplist.skiplistvo import SkipListVO
vo = SkipListVO.range_query(ads, lbound, rbound, coin)
```
The four arguments for `range_query()` are as follows:
- `ads`: the skip list over which the range query is being performed.
- `lbound`: the lower bound (inclusive) of the range query.
- `rbound`: the upper bound (inclusive) of the range query.
- `coin`: the coin class to use to generate the coin flips for insertion into
the VO. This is used primarily in the `insert()` method, described below, and
_must_ be synchronized with the coin used for `ads`. As above, we recommend
using `HashCoin` for this, as it simplifies synchronization.
The verification object returned by a range query also supports insertion
operations, in order to allow data owners to update their local root hash value
when inserting data. This means inserts should have three steps:
- Perform and verify a range query over the range of data to be inserted, from
the minimum element to the maximum element.
- Insert into the remote skip list.
- Insert into the range query object.
## Testing the code
We use nosetests to run unit tests over the skip list code. These can by run by
invoking `nosetests` from the shell in either the `pace/ads/skiplist/`
directory, which will run all of the unit tests, or from its `accumulo/`
subdirectory, which will run only the embedded skip list tests.
pace/ads/skiplist/__init__.py
View File
Whitespace-only changes.
pace/ads/skiplist/accumulo/.demo.py.swp
BIN
View File
Binary file not shown.
pace/ads/skiplist/accumulo/__init__.py
View File
Whitespace-only changes.
pace/ads/skiplist/accumulo/embeddednode.py
+369
View File
@@ -0,0 +1,369 @@
## **************
## Copyright 2015 MIT Lincoln Laboratory
## Project: PACE
## Authors: CS
## Description: Skiplist nodes embedded into Accumulo
## Modifications:
## Date Name Modification
## ---- ---- ------------
## 11 Feb 2015 CS Original file
## **************
import os
import sys
this_dir = os.path.dirname(os.path.abspath(__file__))
base_dir = os.path.join(this_dir, '../../../..')
sys.path.append(base_dir)
# This is the python library implementation of SHA256, which is faster than
# the PyCrypto version in PyCrypto.Hash.SHA256
from hashlib import sha256
from pyaccumulo import Range, Mutation
from collections import deque
import string
from pace.ads.skiplist.authskiplist import AuthSkipList
from pace.ads.skiplist.authnode import AuthNode
from pace.ads.skiplist.elemclass import AccumuloEntry
from pace.common.common_utils import get_single_entry
class EmbeddedNode(AuthNode):
""" An embedding of authenticated skiplist nodes into Accumulo,
attempting to duplicate as little of the complicated code
as possible.
Assumptions:
This class assumes that the user never wants to delete any of
the metadata about a node, just write or overwrite it. In particular,
this means writes of right, down, and parent nodes will always take
a node, not None.
"""
def __init__(self, sl, name):
""" Each node must refer to its parent skiplist, which contains
information about the server and table the embedding resides
in, and the name of the node, which is a unique ID used to
look it up in the relevant table in Accumulo.
Arguments:
sl - the overall embedded skiplist object
name - the unique identifier of this node
"""
self.name = name
self.sl = sl
@staticmethod
def _make_name(down, right, elem):
""" Generate a name for a node with the given children & element.
"""
if down:
# Node is an internal node; make sure its name is (probably) unique
dname = down.name
rname = right.name if right else ''
name = sha256(','.join([dname, rname, elem.serialize()])).digest()
else:
# Node is a leaf; make the hash depend only on the element.
name = sha256(elem.serialize()).digest()
return name
@classmethod
def newnode(cls, sl, down, right, elem, assign_label=False, left=None):
""" Create a new node.
Arguments:
sl - the skiplist this node is a part of.
down - the bottom neighbor of the new node (if any)
right - the right neighbor of the new node (if any)
elem - the element to be stored in the new node.
assign_label - whether to assign a hash label to this node
upon creation (alternative is to defer it to
be called manually by whoever is using this
code). Default: False (do not assign a label
yet)
left - the node to the left of this node, if any. Default: None
Returns:
A new SkipListNode with fields as given.
"""
name = cls._make_name(down, right, elem)
new_node = cls(sl, name)
new_node.elem = elem
if left is not None:
new_node.parent = left, False
if down is not None:
down.parent = (new_node, True)
new_node.down = down
if right is not None:
new_node.right = right
par = right.parent
if par is not None:
parent, from_up = par
if not from_up:
right.parent = (new_node, False)
else:
right.parent = (new_node, False)
if assign_label:
new_node.assign_label()
return new_node
def search(self, elem):
""" Search the SkipList for node containing an element that is
either equal to elem (if elem is in the list) or the closest
element in the list that is less than elem (if elem is not in the
list)
Since this is embedded in an Accumulo table, we can look up the
node in the table first by its hash value. If it is not present,
we then fail back to the older, slower method in the superclass.
Argument:
elem - the element (from elemclass.py) to be searched for
Returns:
visited - a collection of nodes visited, stored in reverse order
current - the closest node in the base list that is less than or
equal to elem
"""
# Look for the element; if it's found, return it; if not, search.
node = self.sl.from_elem(elem)
if node is not None:
# It was found! Return the iterable object for its 'visited'
# path and the node itself.
return node.path(), node
# It wasn't found! Time to search for it. This may arise when the
# actual element being searched for is not stored in Accumulo, in
# which case we need to find the greatest element less than the one
# being searched for.
current = self
right = current.right
visited = deque([])
while (isinstance(right, type(self)) and
right.elem <= elem):
visited.appendleft((current, False))
current = right
right = current.right
down = current.down
while down is not None:
visited.appendleft((current, True))
current = down
right = current.right
while (isinstance(right, type(self)) and
right.elem <= elem):
visited.appendleft((current, False))
current = right
right = current.right
down = current.down
return visited, current
def path(self):
""" Return a generator for all the nodes between this node
and the root.
"""
node = self
parent = node.parent
while parent is not None:
node, flag = parent
yield node, flag
parent = node.parent
@property
def tower(self):
_, from_above = self.parent
return not from_above
@tower.setter
def tower(self, value):
raise Exception('Cannot set embedded tower values.')
@property
def down(self):
entry = get_single_entry(self.sl.conn, self.sl.table, row=self.name,
cf='child', cq='down')
if entry is not None:
return EmbeddedNode(self.sl, entry.val)
else:
return None
@down.setter
def down(self, value):
""" Set the value of the node underneath this one.
Argument must be another node.
Arguments:
value - the EmbeddedNode object to the right of `self`
"""
assert isinstance(value, EmbeddedNode)
m = Mutation(self.name)
m.put(cf='child', cq='down', val=value.name)
self.sl.conn.write(self.sl.table, m)
@property
def right(self):
entry = get_single_entry(self.sl.conn, self.sl.table, row=self.name,
cf='child', cq='right')
if entry:
return EmbeddedNode(self.sl, entry.val)
else:
return None
@right.setter
def right(self, value):
""" Set the value of the node to the right of this one.
Argument must be another node.
Arguments:
value - the EmbeddedNode object to the right of `self`
"""
assert isinstance(value, EmbeddedNode)
m = Mutation(self.name)
m.put(cf='child', cq='right', val=value.name)
self.sl.conn.write(self.sl.table, m)
@property
def raw_elem(self):
""" Return just the serialization of the element.
"""
entry = get_single_entry(self.sl.conn, self.sl.table, row=self.name,
cf='other', cq='elem')
assert entry is not None
return entry.val
@raw_elem.setter
def raw_elem(self, value):
m = Mutation(self.name)
m.put(cf='other', cq='elem', val=value)
self.sl.conn.write(self.sl.table, m)
@property
def elem(self):
return self.sl.elemclass.deserialize(self.raw_elem)
@elem.setter
def elem(self, value):
self.raw_elem = value.serialize()
@property
def label(self):
entry = get_single_entry(self.sl.conn, self.sl.table, row=self.name,
cf='other', cq='label')
return entry.val
@label.setter
def label(self, value):
# labels are represented as unstructured strings, so we can
# just write them directly to the value field
m = Mutation(self.name)
m.put(cf='other', cq='label', val=value)
self.sl.conn.write(self.sl.table, m)
@property
def parent(self):
""" Return the parent node and a flag denoting whether it is the
node above this one ('True') or to the left ('False').
"""
parent = get_single_entry(self.sl.conn, self.sl.table, row=self.name,
cf='parent')
if parent is None:
return None
path, name = string.split(parent.val, ',', 1)
parent_node = EmbeddedNode(self.sl, name)
if path == 'from_left':
return parent_node, False
elif path == 'from_up':
return parent_node, True
else:
raise Exception('Invalid parent column qualifier: %s' %parent.cq)
@parent.setter
def parent(self, value):
""" Set the value of this node's parent node. Argument must be a
tuple of an EmbeddedNode and a boolean denoting whether the
parent is an upper (as opposed to left) neighbor.
NB: `parent` is only set in `newnode()`
"""
parnode, from_up = value
assert isinstance(parnode, EmbeddedNode)
if from_up:
strval = ','.join(['from_up', parnode.name])
else:
strval = ','.join(['from_left', parnode.name])
m = Mutation(self.name)
m.put(cf='parent', cq='', val=strval)
self.sl.conn.write(self.sl.table, m)
@property
def tower(self):
""" Return whether this node is a tower node. If the parent node in the
path is above this one, then it is a tower node; otherwise it is
a plateau node.
"""
parent = self.parent
if parent:
return self.parent[1]
else:
# Must be the root
return False
@tower.setter
def tower(self, value):
# tower is a derived property now---no need to store it explicitly
pass
def assign_label(self):
""" One possible source of optimization for this function would
be to figure out how to streamline/cache the lookups it
performs.
"""
node = self
right = node.right
if right is None:
node.label = str(0)
return
if node.down is None:
if right.tower:
node.label = AuthNode.chash(
AuthNode._hash(node.raw_elem),
AuthNode._hash(right.raw_elem))
else:
node.label = AuthNode.chash(
AuthNode._hash(node.raw_elem),
right.label)
else:
if right.tower:
node.label = node.down.label
else:
node.label = AuthNode.chash(node.down.label, right.label)
pace/ads/skiplist/accumulo/embeddedskiplist.py
+176
View File
@@ -0,0 +1,176 @@
## **************
## Copyright 2015 MIT Lincoln Laboratory
## Project: PACE
## Authors: CS
## Description: Authenticated skiplists embedded in Accumulo
## Modifications:
## Date Name Modification
## ---- ---- ------------
## 10 Feb 2015 CS Original file
## **************
import os
import sys
this_dir = os.path.dirname(os.path.abspath(__file__))
base_dir = os.path.join(this_dir, '../../../..')
sys.path.append(base_dir)
from hashlib import sha256
from pyaccumulo import Accumulo, Range
from collections import namedtuple
from pace.ads.skiplist.authskiplist import AuthSkipList
from pace.ads.skiplist.coin import BaseCoin
from pace.ads.skiplist.elemclass import IntElem
from pace.ads.skiplist.accumulo.embeddednode import EmbeddedNode
from pace.common.fakeconn import FakeConnection
ConnInfo = namedtuple('ConnInfo', ['hostname', 'port', 'username', 'password'])
class EmbeddedSkipList(AuthSkipList):
""" A class for an authenticated skip list embedded in an Accumulo
instance. Most of the work on the actual embedding is in the
node class, found in embeddednode.py
"""
nodeclass = EmbeddedNode
def from_elem(self, elem):
""" Check if an element is in the database. If so, return an
embedded node pointing to that element's leaf node in the
skiplist. If not, return None.
"""
key = sha256(elem.serialize()).digest()
candidates = self.conn.scan(self.table, Range(srow=key, erow=key))
try:
first = next(candidates)
except StopIteration:
# It's not there, so return None
return None
# Otherwise, this row is there, so return the node with the right name
return EmbeddedNode(self, key)
@classmethod
def new(cls, elems, lbound, rbound, coin=BaseCoin(),
conn_info=ConnInfo('localhost', 42424, 'root', 'secret'),
table='__ADS_metadata___',
elemclass=IntElem):
""" Create a new skiplist that stores all of its data inside an
Accumulo instance.
Arguments:
cls - the class implementing this class method
elems - the elements to create the skiplist over
lbound, rbound - the left and right boundary elements of the list
coin - the source of randomness to use
(see pace.ads.skiplist.coin)
conn_info - how to connect to the Accumulo instance being used
table - the name of the table to store the ADS in
elemclass - the class to use to store the elements in the skiplist
"""
sl = cls(None, lbound, rbound, coin)
if conn_info is not None:
# For connecting to a live Accumulo instance
host, port, user, password = conn_info
conn = Accumulo(host=conn_info.host,
port=conn_info.port,
user=conn_info.user,
password=conn_info.password)
else:
# For testing/debug
conn = FakeConnection()
sl.conn = conn
sl.table = table
sl.elemclass = elemclass
if not conn.table_exists(table):
conn.create_table(table)
right = cls.nodeclass.newnode(sl, None, None, rbound, True)
left = cls.nodeclass.newnode(sl, None, right, lbound, True)
sl.root = left
for elem in elems:
sl.insert(elem)
return sl
def do_query(self, visited, closest, elem):
""" Given the input to a search for an element and that element, return
a proof that either that element is in the skiplist or that it is
not (so a proof that the elements on either side of it are adjacent
in the skiplist.
Arguments:
visited - the nodes visited in the skiplist stored in reverse order,
paired with a flag indicating whether the next step was
to go right or down.
closest - the node of the base list containing either the element
being queried or the greatest element in the skiplist
less than the element being queried.
elem - the element (from elemclass.py) being queried
Returns:
ret_elems - The elements in the base list starting from 'closest'
and going to (and including) the next tower node in the
base list.
proof - A list of hash values that, when combined with ret_elems,
proves whether or not elem is in the skiplist.
"""
# NB: this (and several other functions in this class) has only been
# been changed to reduce the number of redundant accesses to
# node.right, node.down, etc.
proof = []
last = closest
found = (last.elem == elem)
ret_elems = []
## First, return the elements of each node to the right of the node
## where the search ended until a tower node is reached. This provides
## enough information to verify a query in either the positive or
## negative direction, as well as enough information for a range
## query to be able to search for the next tower node in the base
## list.
nxt = last
ret_elems.append(nxt.elem)
right = nxt.right
right2 = right.right
## Continue until we encounter a tower node or exhaust the base list
while (not right.tower) and right2:
ret_elems.append(right.elem)
nxt = right
right = right2
right2 = right.right
ret_elems.append(right.elem)
for v, flag in visited:
right = v.right
if not right.tower:
# its right neighbor is a plateau node, so add things
if right.name != last.name: # edited to use names, since they
# might not be pointer-eq
proof.append(right.label)
else:
down = v.down
if down is None:
proof.append(EmbeddedNode._hash(v.elem.serialize()))
else:
proof.append(down.label)
last = v
# Return the elements from the end node in the base list to the next
# tower node; this provides enough information for proof of membership
# and proof of lack of membership, plus some extra information for
# other functions that use do_query()
return ret_elems, proof
pace/ads/skiplist/accumulo/embeddedskiplist_test.py
+381
View File
@@ -0,0 +1,381 @@
## **************
## Copyright 2014 MIT Lincoln Laboratory
## Project: PACE
## Authors: ZS
## Description: Unit tests for authenticated skip lists
## Modifications:
## Date Name Modification
## ---- ---- ------------
## 21 Aug 2014 ZS Original file
## **************
import os
import sys
this_dir = os.path.dirname(os.path.abspath(__file__))
base_dir = os.path.join(this_dir, '../../../..')
sys.path.append(base_dir)
from unittest import TestCase
import time
import random
from pace.ads.skiplist.skiplistnode import InvalidSkipListException
from pace.ads.skiplist.accumulo.embeddednode import EmbeddedNode
from pace.ads.skiplist.accumulo.embeddedskiplist import EmbeddedSkipList
from pace.ads.skiplist.elemclass import IntElem
from pace.ads.skiplist.coin import HashCoin
from pace.ads.skiplist.authnode import AuthNode
from pace.ads.skiplist.authskiplist import AuthSkipList
from pace.common.pacetest import PACETestCase
from pace.common.fakeconn import FakeConnection
class EmbeddedSkipListTests(PACETestCase):
def setUp(self):
random.seed(int(time.time()))
# NB: tests are already slow; chose a small
# number to make them feasible
self.num_iters = 1
def test_valid(self):
""" Ensure that we only create valid skip lists
"""
for i in xrange(0, self.num_iters):
elems = map(IntElem, self.generate_elems())
sl = EmbeddedSkipList.new(
elems, IntElem(-1), IntElem(1000000001), conn_info=None)
try:
sl.valid()
except InvalidSkipListException as e:
self.assertTrue(False, 'Error: %s' %e.msg)
def test_membership(self):
""" Ensure that every element of the skip list is reflected as in it.
"""
for i in xrange(0, self.num_iters):
elems = map(IntElem, self.generate_elems())
sl = EmbeddedSkipList.new(elems, IntElem(-1), IntElem(1000000001), conn_info=None)
for elem in elems:
found, proof = sl.contains(elem)
self.assertTrue(found,
'skip list claims not to have element %d' %elem.key)
def test_verify_in(self):
""" Ensure that proofs of membership validate.
"""
for i in xrange(0, self.num_iters):
elems = map(IntElem, self.generate_elems(-100, 100, 50))
sl = EmbeddedSkipList.new(elems, IntElem(-101), IntElem(101), conn_info=None)
for elem in elems:
found, proof = sl.contains(elem)
self.assertEqual(EmbeddedSkipList.verify(proof), sl.root.label)
def test_verify_in_small(self):
""" Ensure that proofs of membership validate (smaller test).
"""
for i in xrange(0, self.num_iters):
elems = map(IntElem, self.generate_elems(0, 50, 10))
sl = EmbeddedSkipList.new(elems, IntElem(-1), IntElem(101), conn_info=None, coin=HashCoin())
for elem in elems:
found, proof = sl.contains(elem)
self.assertEqual(EmbeddedSkipList.verify(proof), sl.root.label)
def test_non_membership(self):
""" Ensure that non-elements of the skip list are not in it
"""
for i in xrange(0, 10 * self.num_iters):
elems = map(IntElem, self.generate_elems(0, 25, 5))
sl = EmbeddedSkipList.new(elems, IntElem(-201), IntElem(201), conn_info=None)
found, proof = sl.contains(IntElem(-150))
verification = EmbeddedSkipList.verify(proof)
self.assertEqual(verification, sl.root.label)
found, proof = sl.contains(IntElem(150))
self.assertEqual(EmbeddedSkipList.verify(proof), sl.root.label)
bad_elem = IntElem(random.randint(-100, 100))
while bad_elem in elems:
bad_elem = IntElem(random.randint(-100, 100))
found, proof = sl.contains(bad_elem)
self.assertEqual(EmbeddedSkipList.verify(proof), sl.root.label,
'proof of absence of %d fails to verify' %bad_elem.key)
def test_medium_range_query(self):
""" Check that range queries work
"""
lower_bound = -10000
upper_bound = 10000
num_elems = 100
for i in xrange(0, self.num_iters):
elems = map(IntElem,
self.generate_elems(
lower_bound, upper_bound, num_elems))
for x in elems:
if elems.count(x) > 1:
elems.remove(x)
sl = EmbeddedSkipList.new(elems, IntElem(lower_bound-1),
IntElem(upper_bound+1), conn_info=None)
for j in xrange(0, self.num_iters):
lower = IntElem(random.randint(lower_bound/5, upper_bound/100))
upper = IntElem(random.randint(lower.key+10, upper_bound/2))
proofs = sl._range_query(lower, upper)
received = EmbeddedSkipList._verify_range_query(
proofs, lower, upper, sl.root.label)
bottom = sl.root
while bottom.down:
bottom = bottom.down
expected = []
while bottom.right and bottom.elem <= upper:
assert bottom.elem != bottom.right.elem
if bottom.elem >= lower:
expected.append(bottom.elem)
bottom = bottom.right
self.assertTrue(all([lower <= x <= upper for x in expected]))
self.assertTrue(sorted(expected) == expected)
self.assertEqual(expected, received)
def test_smaller_range_query(self):
""" Check that range queries work
"""
lower_bound = -100
upper_bound = 100
num_elems = 5
for i in xrange(0, self.num_iters):
elems = map(IntElem,
self.generate_elems(
lower_bound, upper_bound, num_elems))
for x in elems:
if elems.count(x) > 1:
elems.remove(x)
sl = EmbeddedSkipList.new(elems, IntElem(lower_bound-1),
IntElem(upper_bound+1), conn_info=None)
for j in xrange(0, self.num_iters):
lower = IntElem(random.randint(lower_bound/5, upper_bound/100))
upper = IntElem(random.randint(lower.key+10, upper_bound/2))
proofs = sl._range_query(lower, upper)
received = EmbeddedSkipList._verify_range_query(
proofs, lower, upper, sl.root.label)
bottom = sl.root
while bottom.down:
bottom = bottom.down
expected = []
while bottom.right and bottom.elem <= upper:
assert bottom.elem != bottom.right.elem
if bottom.elem >= lower:
expected.append(bottom.elem)
bottom = bottom.right
self.assertTrue(all([lower <= x <= upper for x in expected]))
self.assertTrue(sorted(expected) == expected)
self.assertEqual(expected, received)
def test_verified_insert(self):
""" Test that verified insert actually allows the client to compute
the new root label.
"""
lower_bound = -1000
upper_bound = 1000
num_elems = 10
num_new_elems = 150
for i in xrange(0, self.num_iters):
elems = map(IntElem,
self.generate_elems(
lower_bound, upper_bound, num_elems))
sl = EmbeddedSkipList.new(elems, IntElem(lower_bound-1),
IntElem(upper_bound+1), conn_info=None)
old_label = sl.root.label
new_elems = map(IntElem,
self.generate_elems(
lower_bound, upper_bound, num_new_elems))
new_elems = [elem for elem in new_elems if elem not in elems]
for elem in new_elems:
ret_elems, proof, proof_diff = sl.insert_with_diff(elem)
self.assertEqual(
EmbeddedSkipList.verify(
[EmbeddedNode._hash(e.serialize())
for e in reversed(ret_elems)] + proof),
old_label)
new_proof = EmbeddedSkipList.update_query(
ret_elems, proof, proof_diff, elem)
x, qproof = sl.contains(elem)
self.assertTrue(x,
'Claims just-inserted element is not in list')
self.assertEqual(EmbeddedSkipList.verify(qproof), sl.root.label)
np = EmbeddedSkipList.verify(new_proof)
self.assertEqual(np, sl.root.label)
old_label = sl.root.label
def test_small_verified_insert(self):
""" Test that verified insert actually allows the client to compute
the new root label.
"""
lower_bound = -100
upper_bound = 100
num_elems = 5
num_new_elems = 10
for i in xrange(0, self.num_iters):
elems = map(IntElem,
self.generate_elems(
lower_bound, upper_bound, num_elems))
sl = EmbeddedSkipList.new(elems, IntElem(lower_bound-1),
IntElem(upper_bound+1), conn_info=None,
coin=HashCoin())
old_label = sl.root.label
new_elems = map(IntElem,
self.generate_elems(
lower_bound, upper_bound, num_new_elems))
new_elems = [elem for elem in new_elems if elem not in elems]
for elem in new_elems:
ret_elems, proof, proof_diff = sl.insert_with_diff(elem)
self.assertEqual(
EmbeddedSkipList.verify(
[EmbeddedNode._hash(e.serialize())
for e in reversed(ret_elems)] + proof),
old_label)
new_proof = EmbeddedSkipList.update_query(
ret_elems, proof, proof_diff, elem)
x, qproof = sl.contains(elem)
self.assertTrue(x,
'Claims just-inserted element is not in list')
self.assertEqual(EmbeddedSkipList.verify(qproof), sl.root.label)
np = EmbeddedSkipList.verify(new_proof)
self.assertEqual(np, sl.root.label)
old_label = sl.root.label
def test_comparison_to_auth(self):
""" Test embedded list and authenticated list side-by-side to see where
the former goes wrong. Mostly here for debugging.
"""
for i in xrange(0, self.num_iters):
elems = map(IntElem, self.generate_elems(0, 50, 10))
esl = EmbeddedSkipList.new(elems, IntElem(-1), IntElem(101), coin=HashCoin(), conn_info=None)
asl = AuthSkipList.new(elems, IntElem(-1), IntElem(101), coin=HashCoin())
for elem in elems:
evisited, eclosest = esl.root.search(elem)
avisited, aclosest = asl.root.search(elem)
new = []
for n, f in evisited:
new.append((n, f))
evisited = new
eelems, eproof = esl.do_query(evisited, eclosest, elem)
aelems, aproof = asl.do_query(avisited, aclosest, elem)
efound = eelems[0] == elem
afound = aelems[0] == elem
eproof = [AuthNode._hash(x.serialize())
for x in reversed(eelems)] + eproof
aproof = [AuthNode._hash(x.serialize())
for x in reversed(aelems)] + aproof
# Make sure the embedded one isn't cheating somehow
self.assertEqual(esl.root.label, asl.root.label)
self.assertEqual(AuthSkipList.verify(aproof),
asl.root.label)
self.assertEqual(EmbeddedSkipList.verify(eproof),
esl.root.label)
# Make sure the embedded one isn't cheating somehow
self.assertEqual(esl.root.label, asl.root.label)
def test_non_membership_comp(self):
""" Ensure that non-elements of the skip list are not in it,
comparing the results to the authenticated skip list.
"""
for i in xrange(0, 10 * self.num_iters):
elems = map(IntElem, self.generate_elems(0, 25, 5))
esl = EmbeddedSkipList.new(elems, IntElem(-201), IntElem(201), coin=HashCoin(), conn_info=None)
asl = AuthSkipList.new(elems, IntElem(-201), IntElem(201), coin=HashCoin())
bad_elem = IntElem(random.randint(-100, 100))
while bad_elem in elems:
bad_elem = IntElem(random.randint(-100, 100))
bad_elems = [IntElem(-150), IntElem(150), bad_elem]
for elem in bad_elems:
evisited, eclosest = esl.root.search(elem)
avisited, aclosest = asl.root.search(elem)
new = []
for n, f in evisited:
new.append((n, f))
evisited = new
eelems, eproof = esl.do_query(evisited, eclosest, elem)
aelems, aproof = asl.do_query(avisited, aclosest, elem)
efound = eelems[0] == elem
afound = aelems[0] == elem
eproof = [AuthNode._hash(x.serialize())
for x in reversed(eelems)] + eproof
aproof = [AuthNode._hash(x.serialize())
for x in reversed(aelems)] + aproof
self.assertEqual(efound, afound)
self.assertTrue(not efound)
self.assertEqual(AuthSkipList.verify(aproof),
asl.root.label)
self.assertEqual(EmbeddedSkipList.verify(eproof),
esl.root.label)
# Make sure the embedded one isn't cheating somehow
self.assertEqual(esl.root.label, asl.root.label)
pace/ads/skiplist/authnode.py
+84
View File
@@ -0,0 +1,84 @@
## **************
## Copyright 2014 MIT Lincoln Laboratory
## Project: PACE
## Authors: ZS
## Description: Authenticated skip list nodes
## Modifications:
## Date Name Modification
## ---- ---- ------------
## 21 Aug 2014 ZS Original file
## **************
# This is the python library implementation of SHA256, which is faster than
# the PyCrypto version in PyCrypto.Hash.SHA256
from hashlib import sha256
from pace.ads.skiplist.skiplist import SkipList
from pace.ads.skiplist.skiplistnode import SkipListNode
class AuthNode(SkipListNode):
@staticmethod
def _hash(x):
return sha256(x).digest()
@staticmethod
def chash(x, y):
""" A commutative hash function, defined in terms of the local _hash
function (in this case, sha256). Hashes the concatenation of the
minimum of its two arguments with the maximum of its two arguments.
"""
return sha256(min(x, y) + max(x, y)).digest()
@classmethod
def newnode(cls, sl, down, right, elem, assign_label=False, *args, **kwargs):
""" Create a new node. Can be overridden by subclasses.
Arguments:
sl - the skiplist this node is a part of.
down - the bottom neighbor of the new node
right - the right neighbor of the new node
elem - the element to be stored in the new node.
assign_label - whether to assign a hash label to this node
upon creation (alternative is to defer it to
be called manually by whoever is using this
code). Default: False (do not assign a label
yet)
*args, **kwargs - any more arguments that a subclass
might want to use
Returns:
A new SkipListNode with fields as given.
"""
new_node = cls(sl, down, right, elem)
new_node.tower = False
if assign_label:
new_node.assign_label()
if down:
down.tower = True
return new_node
def assign_label(self):
node = self
if node.right is None:
node.label = str(0)
return
if node.down is None:
if node.right.tower:
node.label = AuthNode.chash(
AuthNode._hash(node.elem.serialize()),
AuthNode._hash(node.right.elem.serialize()))
else:
node.label = AuthNode.chash(
AuthNode._hash(node.elem.serialize()),
node.right.label)
else:
if node.right.tower:
node.label = node.down.label
else:
node.label = AuthNode.chash(node.down.label, node.right.label)
pace/ads/skiplist/authskiplist.py
+557
View File
@@ -0,0 +1,557 @@
## **************
## Copyright 2014 MIT Lincoln Laboratory
## Project: PACE
## Authors: ZS
## Description: Authenticated skip list library
## (as in Goodrich & Tamassia 2001)
## Modifications:
## Date Name Modification
## ---- ---- ------------
## 20 Aug 2014 ZS Original file
## **************
import os
import sys
this_dir = os.path.dirname(os.path.abspath(__file__))
base_dir = os.path.join(this_dir, '../../..')
sys.path.append(base_dir)
from hashlib import sha256
from pace.ads.skiplist.skiplist import SkipList
from pace.ads.skiplist.authnode import AuthNode
class AuthSkipList(SkipList):
nodeclass = AuthNode
def insert(self, elem):
""" Insert elem into the base level of the current SkipList, then
determine how many levels to elevate it by, returning the new hash
value of the tree. Note that this assumes some other information in
order to prove that the insert was done correctly (such as having
done a range query on the proper range); without this information,
one should use insert_with_diff, instead.
Returns:
ret_elems - the elements in the query set
label - the updated root label
Override: also needs to update metadata of each node
"""
visited, left_neighbor = self.root.search(elem)
# Have to do this query now before the skiplist gets updated
# TODO: can maybe do the query inline instead of duplicating effort
ret_elems, proof = self.do_query(visited, left_neighbor, elem)
## First, insert the element into the list---note that this does NOT
## label the node; that must be done later
new_node = self.nodeclass.newnode(self, None, left_neighbor.right, elem, left=left_neighbor)
left_neighbor.right = new_node
current = new_node
stopped = False
lefts = 0
current_hash_value = sha256(elem.serialize()).digest()
new_elems = []
## Determine how many levels of the skiplist to insert the new
## element on. Each time we visited a node in the traversal, if we
## dropped down (instead of going right), that node is a potential
## new neighbor for a new node. We look at these bottom-up, and keep
## inserting nodes until the element is gone.
for v, flag in visited:
if not flag:
# We just went right at this step, so don't do anything
continue
# Otherwise, we dropped down, so we need to see if we insert a new
# level of the inserted element
if self.stop(current_hash_value):
stopped = True
break
current_hash_value = sha256(current_hash_value).digest()
## Defer labeling until later, in case we're inserting it at the
## current top level.
next_level = self.nodeclass.newnode(self, current, v.right, elem, left=v)
new_elems.append(next_level)
v.right = next_level
current = next_level
new_roots = []
if not stopped:
# If we got here we need to add another level of just -infty, +infty
old_rght = self.root.right.right
new_rght = self.nodeclass.newnode(
self, self.root.right.right, None, self.rbound, True)
new_root = self.nodeclass.newnode(
self, self.root, new_rght, self.lbound)
self.root = new_root
new_roots.append(new_root)
# At this point, we can safely assign labels to all the new nodes
# created up until now.
new_node.assign_label()
for n in new_elems:
n.assign_label()
while not self.stop(current_hash_value):
current_hash_value = sha256(current_hash_value).digest()
# Add higher levels
old_right = self.root.right
old_left = self.root
new_right = self.nodeclass.newnode(
self, old_right, None, old_right.elem, True)
new_left = self.nodeclass.newnode(
self, old_left, new_right, old_left.elem)
new_roots.append(new_left)
# It's safe to label this one because the new level has
# already been inserted
new_level = self.nodeclass.newnode(
self, current, old_right, elem, True, left=old_left)
old_left.right = new_level
self.root = new_left
current = new_level
else:
# Need to update the nodes visited up until now
new_node.assign_label()
for n in new_elems:
n.assign_label()
# Now we have to update all the visited nodes' labels
left_neighbor.assign_label()
modified = [v for v, _ in visited] + new_roots
for v in modified:
v.assign_label()
return ret_elems, self.root.label
def insert_with_diff(self, elem):
""" Insert elem into the base level of the current SkipList, then
determine how many levels to elevate it by, returning a proof
that the spot used to be empty and a proof that the element was
actually inserted so the client can update its root label.
Returns three values:
- ret_elems: The elements returned by a query for `elem`
in the old skip list.
- proof: A proof that `elem` was not in the old skip list
- proof_diff: A diff that the client can use to insert
`elem` into `proof` and receive a new root
hash value.
"""
visited, left_neighbor = self.root.search(elem)
# Have to do this query now before the skiplist gets updated
# TODO: can maybe do the query inline instead of duplicating effort
ret_elems, proof = self.do_query(visited, left_neighbor, elem)
## First, insert the element into the list---note that this does NOT
## label the node; that must be done later
new_node = self.nodeclass.newnode(self, None, left_neighbor.right, elem, left=left_neighbor)
left_neighbor.right = new_node
current = new_node
stopped = False
lefts = 0
current_hash_value = sha256(elem.serialize()).digest()
new_elems = []
proof_diff = []
pre_proof_diff = []
## Determine how many levels of the skiplist to insert the new
## element on. Each time we visited a node in the traversal, if we
## dropped down (instead of going right), that node is a potential
## new neighbor for a new node. We look at these bottom-up, and keep
## inserting nodes until the element is gone.
for v, flag in visited:
if not flag:
# We just went right at this step, so don't do anything, but
# remember that we have to accumulate one more level if we
# end up going up again
lefts = lefts + 1
# If we're still on the base level, need to signal that this
# element was added from the base level, for bookkeeping
# purposes.
if not v.down:
pre_proof_diff.append('MEET')
continue
# Otherwise, we dropped down, so we need to see if we insert a new
# level of the inserted element
if self.stop(current_hash_value):
stopped = True
break
current_hash_value = sha256(current_hash_value).digest()
if not proof_diff:
proof_diff = pre_proof_diff
proof_diff.append('UP')
lefts = 0 # UP subsumes PASS
## Defer labeling until later, in case we're inserting it at the
## current top level.
next_level = self.nodeclass.newnode(self, current, v.right, elem, left=v)
new_elems.append(next_level)
## If this level was "interesting", we need to add to the returned
## value that tells the client how to modify the query to get the
## new root value.
##
## An "interesting" level is defined as the following:
##
## - The first new level (after the base level) is always
## interesting.
## - Any level that has just gone up after going left some nonzero
## number of times is interesting.
## - Any level with a plateau node on the right (other than +infty)
## is interesting.
if lefts:
# The tower of new elements just passed a plateau node on the
# left, so we need to accumulate all the labels from that chain
# of plateau nodes
proof_diff.append(str(lefts))
lefts = 0
vright = v.right
if (not vright.tower) and vright.right:
# The tower of new elements just met a plateau node on the right
proof_diff.append('MEET')
v.right = next_level
current = next_level
new_roots = []
if not stopped:
# If we got here we need to add another level of just -infty, +infty
old_rght = self.root.right.right
new_rght = self.nodeclass.newnode(
self, old_rght, None, self.rbound, True)
new_root = self.nodeclass.newnode(
self, self.root, new_rght, self.lbound)
self.root = new_root
new_roots.append(new_root)
# At this point, we can safely assign labels to all the new nodes
# created up until now.
new_node.assign_label()
for n in new_elems:
n.assign_label()
while not self.stop(current_hash_value):
current_hash_value = sha256(current_hash_value).digest()
# Add higher levels
old_right = self.root.right
old_left = self.root
#TODO: can cache min/max elems for fewer lookups
new_right = self.nodeclass.newnode(
self, old_right, None, old_right.elem, True)
new_left = self.nodeclass.newnode(
self, old_left, new_right, old_left.elem)
new_roots.append(new_left)
# It's safe to label this one because the new level has
# already been inserted
new_level = self.nodeclass.newnode(
self, current, old_right, elem, True, left=old_left)
old_left.right = new_level
self.root = new_left
current = new_level
else:
# Need to update the nodes visited up until now
new_node.assign_label()
for n in new_elems:
n.assign_label()
# Now we have to update all the visited nodes' labels
left_neighbor.assign_label()
modified = [v for v, _ in visited] + new_roots
for v in modified:
v.assign_label()
return ret_elems, proof, proof_diff
@staticmethod
def update_query(base_elems, old_proof, proof_diff, elem):
""" Updates a query of an element from an old skiplist without
that element in it to a query of that element in the skiplist
resulting from inserting it into the old one. Used to update
the root label after an insert.
"""
# No matter what, we're going to insert elem into the proof, so do it
# first
elem_loc = 1 # will always be immediately after the left neighbor
base_elems.insert(elem_loc, elem)
i = 0
current = None
# Recall that a proof is just a path from a base level node to the root
# of the skip list. This partitions the path into three parts:
#
# - Everything from the base of the new path to the top of the new tower
# created by adding elem
# - The label of the node below the node to the left of the new plateau
# node, if such a node exists (i.e. if at least one new non-base node
# was added). This is calculated from elements in the old proof.
# - Everything further up in the proof than the new tower reached, whose
# order is unchanged.
#
# The element i keeps track of the difference between these locations:
# everything to the left of i (not including i) is in the first group,
# everything to the right of i (including i) is in the third group, and
# the variable 'current' is the second group.
for action in proof_diff:
if action == 'MEET':
# If we meet a plateau node on the right, we need to make sure
# it occurs in the proof before the values we're accumulating
# (since they need to get hashed into the overall value only
# after the new tower stops & starts going left)
i = i + 1
elif action == 'UP':
# This action should only occur once per insert, if the inserted
# element's tower ends up going above the base list.
# First, we accumulate all the elements in the base list to
# the left of the new element (and including the new element
# itself) that occur in the proof.
current = AuthNode.chash(AuthNode._hash(base_elems[0].serialize()),
AuthNode._hash(elem.serialize()))
current = reduce(AuthNode.chash, old_proof[:i], current)
# Next, we update the proof we're modifying. The first element
# in it now needs to be the label of the bottom node in the
# new tower.
old_proof = old_proof[i:]
i = 1
# To compute this label, hash the elements from the inserted
# element to the next base-level tower node, then accumulate
# them together in order with the commutative hash.
proof_order_hashes = [AuthNode._hash(e.serialize())
for e in reversed(base_elems[elem_loc:])]
new_elem_node_label = reduce(AuthNode.chash, proof_order_hashes)
old_proof = [new_elem_node_label] + old_proof
# base_elems has been folded into old_proof, so we can set it
# to the empty list.
base_elems = []
else: # passed a plateau node
# Accumulate the next 'acc_levels' elements of the old proof
# into the current value.
acc_levels = int(action)
for z in range(acc_levels):
passed = old_proof.pop(i)
current = AuthNode.chash(passed, current)
if current is not None:
old_proof.insert(i, current)
# Return the single list to feed in to verify()
return ([AuthNode._hash(e.serialize()) for e in reversed(base_elems)] +
old_proof)
def contains(self, elem):
""" Override of super method. Now returns a proof that elem either is
or is not in the skip list.
NB: I have no idea what this algorithm is trying to do, why
it's right, or how to verify it, I'm just transcribing it
from the paper's pseudocode into Python.
"""
visited, closest = self.root.search(elem)
elems, proof = self.do_query(visited, closest, elem)
return elems[0] == elem, [AuthNode._hash(x.serialize())
for x in reversed(elems)] + proof
def do_query(self, visited, closest, elem):
""" Given the input to a search for an element and that element, return
a proof that either that element is in the skiplist or that it is
not (so a proof that the elements on either side of it are adjacent
in the skiplist.
Arguments:
visited - the nodes visited in the skiplist stored in reverse order,
paired with a flag indicating whether the next step was
to go right or down.
closest - the node of the base list containing either the element
being queried or the greatest element in the skiplist
less than the element being queried.
elem - the element being queried
Returns:
ret_elems - The elements in the base list starting from 'closest'
and going to (and including) the next tower node in the
base list.
proof - A list of hash values that, when combined with ret_elems,
proves whether or not elem is in the skiplist.
"""
proof = []
last = closest
found = (last.elem == elem)
ret_elems = []
## First, return the elements of each node to the right of the node
## where the search ended until a tower node is reached. This provides
## enough information to verify a query in either the positive or
## negative direction, as well as enough information for a range
## query to be able to search for the next tower node in the base
## list.
nxt = last
ret_elems.append(nxt.elem)
while (not nxt.right.tower) and nxt.right.right:
ret_elems.append(nxt.right.elem)
nxt = nxt.right
ret_elems.append(nxt.right.elem)
for v, flag in visited:
if not v.right.tower:
# its right neighbor is a plateau node, so add things
if v.right is not last:
proof.append(v.right.label)
elif v.down is None:
proof.append(AuthNode._hash(v.elem.serialize()))
else:
proof.append(v.down.label)
last = v
# Return the elements from the end node in the base list to the next
# tower node; this provides enough information for proof of membership
# and proof of lack of membership, plus some extra information for
# other functions that use do_query()
return ret_elems, proof
def _range_query(self, lower, upper):
""" Perform a range query on the skip list, returning all elements
between lower and upper (inclusive).
Deprecated code, useful mostly for benchmarking/regression tests
"""
## General algorithm overview:
## - First, search for 'lower' in the list. This will return either the
## "leaf" node that contains 'lower' (if it is in the list) or the
## first node with an element less than 'lower' (if it is not). Either
## way, this will provide the evidence that the first element of the
## result is in fact the first element in the list that is in the
## range.
##
## - Keeping track of the current element, move right along the list
## until finding an element greater than or equal to 'upper'. Each
## time a node is reached where the last node searched for does not
## depend on its element's value, perform a new search for that
## element. That will guarantee that the client can verify that the
## returned range is both sound and complete.
##
## - Return to the client the sequence of verification sequences
## generated by this process.
##
## TODO: We can almost certainly optimize this process, perhaps by
## only returning the paths to a least common ancestor (like in MHTs),
## or even more cleverly, tracing the previous visited sequence up until
## we find the place where the next search would branch off from it,
## then return a sequence (with possible subsequences) based on these
## traversals.
visited, left_bound = self.root.search(lower)
elems, proof = self.do_query(visited, left_bound, lower)
proofs = [(elems, proof)]
elem = elems[-1]
while elem < upper:
visited, left_bound = self.root.search(elem)
elems, proof = self.do_query(visited, left_bound, elem)
proofs.append((elems, proof))
elem = elems[-1]
return proofs
@staticmethod
def verify(proof):
# TODO: does this actually verify anything?
return reduce(AuthNode.chash, proof)
@staticmethod
def _verify_range_query(proofs, lower, upper, root_label):
""" Verifies a range query done with (the deprecated function)
_range_query(). Also deprecated.
"""
labels = [AuthSkipList.verify([AuthNode._hash(x.serialize())
for x in reversed(elems)] + proof)
for elems, proof in proofs]
if not labels:
raise InvalidVerificationObjectException(
'Empty list of proofs returned')
all_elems = proofs[0][0]
for elems, proof in proofs[1:]:
all_elems.extend(elems[1:])
## Check to make sure the original list of elements is sorted
if not (sorted(all_elems) == all_elems):
raise InvalidVerificationObjectException(
'Returned list of elements not sorted')
## There might be more than one upper boundary that got included, so
## make sure to get rid of all of them from the returned list
while all_elems[-1] > upper:
all_elems.pop()
## The lower boundary may or may not have been needed, so make sure to
## include it just in case
if all_elems[0] < lower:
all_elems = all_elems[1:]
if not all([label == root_label for label in labels]):
raise InvalidVerificationObjectException(
'Invalid path exists in VO')
if not all([lower <= elem <= upper for elem in all_elems]):
raise InvalidVerificationObjectException(
'Elements returned not in requested range')
return all_elems
class InvalidVerificationObjectException(Exception):
def __init__(self, msg):
self.msg = msg
pace/ads/skiplist/authskiplist_test.py
+228
View File
@@ -0,0 +1,228 @@
## **************
## Copyright 2014 MIT Lincoln Laboratory
## Project: PACE
## Authors: ZS
## Description: Unit tests for authenticated skip lists
## Modifications:
## Date Name Modification
## ---- ---- ------------
## 21 Aug 2014 ZS Original file
## **************
import os
import sys
this_dir = os.path.dirname(os.path.abspath(__file__))
base_dir = os.path.join(this_dir, '../../..')
sys.path.append(base_dir)
from unittest import TestCase
import time
import random
from pace.ads.skiplist.skiplistnode import InvalidSkipListException
from pace.ads.skiplist.authnode import AuthNode
from pace.ads.skiplist.authskiplist import AuthSkipList
from pace.ads.skiplist.elemclass import IntElem
from pace.common.pacetest import PACETestCase
def gen_elems(lower, upper, num):
return [IntElem(random.randint(lower, upper)) for i in range(num)]
class AuthSkipListTests(PACETestCase):
def setUp(self):
random.seed(int(time.time()))
def test_valid(self):
""" Ensure that we only create valid skip lists
"""
for i in range(0, self.num_iters):
elems = gen_elems(0, 1000000000, self.size)
sl = AuthSkipList.new(elems, IntElem(-1), IntElem(1000000001))
try:
sl.valid()
except InvalidSkipListException as e:
self.assertTrue(False, 'Error: %s' %e.msg)
def test_membership(self):
""" Ensure that every element of the skip list is reflected as in it.
"""
for i in range(0, self.num_iters):
elems = gen_elems(0, 100000000, self.size)
sl = AuthSkipList.new(elems, IntElem(-1), IntElem(1000000001))
for elem in elems:
found, proof = sl.contains(elem)
self.assertTrue(found,
'skip list claims not to have element %d' %elem.key)
def test_verify_in(self):
""" Ensure that proofs of membership validate.
"""
for i in range(0, self.num_iters):
elems = gen_elems(-100, 100, 50)
sl = AuthSkipList.new(elems, IntElem(-101), IntElem(101))
for elem in elems:
found, proof = sl.contains(elem)
self.assertEqual(AuthSkipList.verify(proof), sl.root.label)
def test_non_membership(self):
""" Ensure that non-elements of the skip list are not in it
"""
for i in range(0, 10 * self.num_iters):
elems = gen_elems(0, 25, 5)
sl = AuthSkipList.new(elems, IntElem(-201), IntElem(201))
found, proof = sl.contains(IntElem(-150))
verification = AuthSkipList.verify(proof)
self.assertEqual(verification, sl.root.label)
found, proof = sl.contains(IntElem(150))
self.assertEqual(AuthSkipList.verify(proof), sl.root.label)
bad_elem = IntElem(random.randint(-100, 100))
while bad_elem in elems:
bad_elem = IntElem(random.randint(-100, 100))
found, proof = sl.contains(bad_elem)
self.assertEqual(AuthSkipList.verify(proof), sl.root.label,
'proof of absence of %d fails to verify' %bad_elem.key)
def test_range_query(self):
""" Check that range queries work
"""
lower_bound = -100000
upper_bound = 100000
num_elems = 1000
for i in range(0, self.num_iters):
elems = gen_elems(lower_bound, upper_bound, num_elems)
for x in elems:
if elems.count(x) > 1:
elems.remove(x)
sl = AuthSkipList.new(elems, IntElem(lower_bound-1),
IntElem(upper_bound+1))
for j in range(0, self.num_iters):
lower = IntElem(random.randint(lower_bound/5, upper_bound/100))
upper = IntElem(random.randint(lower.key+1000, upper_bound/5))
proofs = sl._range_query(lower, upper)
received = AuthSkipList._verify_range_query(
proofs, lower, upper, sl.root.label)
bottom = sl.root
while bottom.down:
bottom = bottom.down
expected = []
while bottom.right and bottom.elem <= upper:
if bottom.elem >= lower:
expected.append(bottom.elem)
bottom = bottom.right
self.assertTrue(all([lower <= x <= upper for x in expected]))
self.assertTrue(sorted(expected) == expected)
print 'Range query from %d to %d' %(lower.key, upper.key)
print "Expected length: %d" %len(expected)
print "Received length: %d" %len(received)
self.assertEqual(expected, received)
def test_smaller_range_query(self):
""" Check that range queries work
"""
lower_bound = -100
upper_bound = 100
num_elems = 5
for i in range(0, self.num_iters):
elems = gen_elems(lower_bound, upper_bound, num_elems)
for x in elems:
if elems.count(x) > 1:
elems.remove(x)
sl = AuthSkipList.new(elems, IntElem(lower_bound-1),
IntElem(upper_bound+1))
for j in range(0, self.num_iters):
lower = IntElem(random.randint(lower_bound/5, upper_bound/100))
upper = IntElem(random.randint(lower.key+10, upper_bound/2))
proofs = sl._range_query(lower, upper)
received = AuthSkipList._verify_range_query(
proofs, lower, upper, sl.root.label)
bottom = sl.root
while bottom.down:
bottom = bottom.down
expected = []
while bottom.right and bottom.elem <= upper:
assert bottom.elem != bottom.right.elem
if bottom.elem >= lower:
expected.append(bottom.elem)
bottom = bottom.right
self.assertTrue(all([lower <= x <= upper for x in expected]))
self.assertTrue(sorted(expected) == expected)
self.assertEqual(expected, received)
def test_verified_insert(self):
""" Test that verified insert actually allows the client to compute
the new root label.
"""
lower_bound = -1000
upper_bound = 1000
num_elems = 10
num_new_elems = 150
for i in range(0, self.num_iters):
elems = gen_elems(lower_bound, upper_bound, num_elems)
elems = set(elems)
sl = AuthSkipList.new(elems, IntElem(lower_bound-1),
IntElem(upper_bound+1))
old_label = sl.root.label
new_elems = gen_elems(lower_bound, upper_bound, num_new_elems)
new_elems = [elem for elem in new_elems if elem not in elems]
for elem in new_elems:
print 'inserting elem %d' %elem.key
print 'into list %s' %str(sl.to_list_of_lists())
ret_elems, proof, proof_diff = sl.insert_with_diff(elem)
print 'result list: %s' %str(sl.to_list_of_lists())
self.assertEqual(
AuthSkipList.verify(
[AuthNode._hash(e.serialize())
for e in reversed(ret_elems)] + proof),
old_label)
new_proof = AuthSkipList.update_query(
ret_elems, proof, proof_diff, elem)
x, qproof = sl.contains(elem)
self.assertTrue(x,
'Claims just-inserted element is not in list')
self.assertEqual(AuthSkipList.verify(qproof), sl.root.label)
np = AuthSkipList.verify(new_proof)
print 'Root label: %s' %str(sl.root.label)
print 'Recv label: %s' %str(np)
self.assertEqual(np, sl.root.label)
old_label = sl.root.label
pace/ads/skiplist/coin.py
+116
View File
@@ -0,0 +1,116 @@
## **************
## Copyright 2014 MIT Lincoln Laboratory
## Project: PACE
## Authors: CS
## Description: Coin toss classes with various extra functionalities
## Modifications:
## Date Name Modification
## ---- ---- ------------
## 19 Aug 2014 CS Original file
## **************
import random
from collections import deque
class BaseCoin(object):
""" Provides basic coin-toss functionality: uses the random module to
'flip' a 'coin', returning either True or False with equal probability.
NB: I believe the entropy used by the random module is low-quality, so
this should only be used for non-security-required operations, such
as determining the structure of a random skip list.
"""
def flip(self, *args):
return random.randint(0,1) == 1
class RecordedPrefixCoin(BaseCoin):
def __init__(self, prefix):
self.record = []
self.prefix = deque(prefix)
def flip(self, *args):
if self.prefix:
bit = self.prefix.popleft()
self.record.append(bit)
return bit
else:
bit = super(RecordedPrefixCoin, self).flip()
self.record.append(toss)
return bit
# Actually, we just need the client to record, and the server to be able to
# access a prefix, so we can split up these functionalities
class RecordedCoin(BaseCoin):
""" Acts like a BaseCoin, but also records the outputs it produced, to be
used later if needed.
"""
def __init__(self):
self.record = []
def flip(self, *args):
bit = super(RecordedCoin, self).flip()
self.record.append(bit)
return bit
def read(self):
""" Read the record of the flips so far, then reset it to be empty.
"""
tmp = self.record
self.record = []
return tmp
class PrefixCoin(BaseCoin):
""" Acts like a base coin, but first uses the provided list of boolean
values, switching to the random module once it runs out of values from
the list to use.
"""
def __init__(self, prefix):
self.prefix = deque(prefix)
def flip(self, *args):
if self.prefix:
bit = self.prefix.popleft()
return bit
else:
bit = super(PrefixCoin, self).flip()
return bit
def extend(self, suffix):
""" Extend 'self' with the elements in 'suffix', to be used after
the ones already contained in the prefix.
"""
self.prefix.extend(suffix)
# Also useful for benchmarking---coins that start from a specific seed
class SeededCoin(BaseCoin):
""" Acts like a base coin, but first seeds the random module with a given
seed. Each call to flip() then restores the previous random state,
computes the next bit, and stores the resulting state.
"""
def __init__(self, seed):
random.seed(seed)
self.state = random.getstate()
def flip(self, *args):
random.setstate(self.state)
bit = super(SeededCoin, self).flip()
self.state = random.getstate()
return bit
def reseed(self, seed):
random.seed(seed)
self.state = random.getstate()
class HashCoin(BaseCoin):
""" Iteratively computes the hash of each base element to compute the coin
flip of each node. Inherently stateful & dependent on how skiplists
are structured.
The value returned is False if the least-significant bit of the hash
value is 0, and True if the least-significant bit is 1.
"""
def flip(self, elem_hash, *args):
return (ord(elem_hash[-1]) % 2) == 1
pace/ads/skiplist/coin_test.py
+178
View File
@@ -0,0 +1,178 @@
## **************
## Copyright 2014 MIT Lincoln Laboratory
## Project: PACE
## Authors: CS
## Description: Unit tests for coin library
## Modifications:
## Date Name Modification
## ---- ---- ------------
## 08 Aug 2014 CS Original file
## **************
import os
import sys
this_dir = os.path.dirname(os.path.abspath(__file__))
base_dir = os.path.join(this_dir, '../../..')
sys.path.append(base_dir)
from unittest import TestCase
import time
import random
from pace.ads.skiplist.coin import BaseCoin, PrefixCoin, SeededCoin, RecordedPrefixCoin, RecordedCoin, HashCoin
from pace.ads.skiplist.authskiplist import AuthSkipList
from pace.ads.skiplist.elemclass import IntElem
from pace.common.pacetest import PACETestCase
class SkipListTests(PACETestCase):
def setUp(self):
self.sizes = [10, 100, 1000, 10000]
random.seed(time.time())
def hash_coin_test(self):
for n in self.sizes:
for iters in range(self.num_iters):
elems = [str(random.randint(-10000000, 10000000))
for i in range(1000)]
hc = HashCoin()
hash_elems1 = [hc.flip(e) for e in elems]
hash_elems2 = [hc.flip(e) for e in elems]
self.assertEqual(hash_elems1, hash_elems2)
def prefix_coin_test(self):
for n in self.sizes:
for iters in range(self.num_iters):
bc = BaseCoin()
elems = [bc.flip() for i in range(n)]
pc = PrefixCoin(elems)
prefix_elems = [pc.flip() for i in range(n)]
self.assertEqual(elems, prefix_elems)
def seeded_coin_test(self):
for n in self.sizes:
for iters in range(self.num_iters):
rand = random.randint(0,1000)
random.seed(rand)
bc = BaseCoin()
elems = [bc.flip() for i in range(n)]
sc = SeededCoin(rand)
seeded_elems = [sc.flip() for i in range(n)]
self.assertEqual(elems, seeded_elems)
def record_coin_test(self):
for n in self.sizes:
for iters in range(self.num_iters):
rand = random.randint(0,1000)
random.seed(rand)
bc = BaseCoin()
elems = [bc.flip() for i in range(n)]
random.seed(rand)
rc = RecordedCoin()
more_elems = [rc.flip() for i in range(n)]
rec_elems = rc.record
self.assertEqual(elems, rec_elems)
self.assertEqual(elems, more_elems)
def record_prefix_coin_test(self):
for n in self.sizes:
for iters in range(self.num_iters):
rand = random.randint(0,1000)
random.seed(rand)
bc = BaseCoin()
elems = [bc.flip() for i in range(n)]
random.seed(rand)
rpc = RecordedPrefixCoin(elems)
more_elems = [rpc.flip() for i in range(n)]
rec_elems = rpc.record
self.assertEqual(elems, rec_elems)
self.assertEqual(elems, more_elems)
def base_coin_test(self):
for n in self.sizes:
for iters in range(self.num_iters):
rand = random.randint(0,1000)
random.seed(rand)
elems = [random.randint(0,1) == 1 for i in range(n)]
random.seed(rand)
bc = BaseCoin()
flipped_elems = [bc.flip() for i in range(n)]
self.assertEqual(elems, flipped_elems)
def skiplist_test(self):
""" Make sure the different coins can be used to guarantee the same
skiplist.
"""
for n in self.sizes[:-1]:
for iters in range(self.num_iters):
max_elem = n * 100
min_elem = -1 * max_elem
lower = IntElem(min_elem-1)
upper = IntElem(max_elem+1)
elems = [IntElem(random.randint(min_elem, max_elem))
for i in range(n)]
rand = random.randint(0,1000)
random.seed(rand)
prefix = [random.randint(0,1) == 1 for i in range(10 * n)]
random.seed(rand)
sl_base = AuthSkipList.new(
elems, lower, upper, BaseCoin())
random.seed(rand)
sl_recd = AuthSkipList.new(
elems, lower, upper, RecordedCoin())
sl_pref = AuthSkipList.new(
elems, lower, upper, PrefixCoin(prefix))
sl_rpre = AuthSkipList.new(
elems, lower, upper, RecordedPrefixCoin(prefix))
sl_seed = AuthSkipList.new(
elems, lower, upper, SeededCoin(rand))
self.assertEqual(sl_seed.to_list_of_lists(),
sl_rpre.to_list_of_lists())
self.assertEqual(sl_rpre.to_list_of_lists(),
sl_pref.to_list_of_lists())
self.assertEqual(sl_pref.to_list_of_lists(),
sl_recd.to_list_of_lists())
self.assertEqual(sl_recd.to_list_of_lists(),
sl_base.to_list_of_lists())
self.assertEqual(sl_seed.root.label,
sl_rpre.root.label)
self.assertEqual(sl_rpre.root.label,
sl_pref.root.label)
self.assertEqual(sl_pref.root.label,
sl_recd.root.label)
self.assertEqual(sl_recd.root.label,
sl_base.root.label)
pace/ads/skiplist/elemclass.py
+146
View File
@@ -0,0 +1,146 @@
## **************
## Copyright 2014 MIT Lincoln Laboratory
## Project: PACE
## Authors: CS
## Description: Interface for skiplist elements & some specific implemetations
## Modifications:
## Date Name Modification
## ---- ---- ------------
## 03 Oct 2014 CS Original file
## **************
from abc import ABCMeta, abstractmethod
class BaseElem(object):
""" Abstract superclass for wrapping base Python types into elements
suitable for insertion into SkipLists. Must be abstract since it
can't generically define the deserialize method.
"""
__metaclass__ = ABCMeta
def __init__(self, key):
""" This class is a wrapper around the type of keys being stored.
"""
self.key = key
def __str__(self):
return str(self.key)
def __cmp__(self, other):
""" Default method for comparing two objects. Assumes
the key's __cmp__ method is the desired method for
ordering.
"""
if self.key < other.key:
return -1
elif self.key == other.key:
return 0
else:
assert self.key > other.key
return 1
@abstractmethod
def serialize(self):
""" Turn the element into a string to transportation over
a network. Needs to be convertable back into an equivalent
object with deserialize()
"""
@abstractmethod
def deserialize(s):
""" Turn a string into an element of the implementation class.
Resulting element needs to serialize to the same original
string.
"""
pass
class IntElem(BaseElem):
def serialize(self):
return str(self.key)
@staticmethod
def deserialize(s):
return IntElem(int(s))
class StrElem(BaseElem):
# Elements are strings, so serialize and deserialize operations are
# the identity.
def serialize(self):
return self.key
@staticmethod
def deserialize(s):
return StrElem(s)
class AccumuloKey(object):
""" Class to use for Accumulo keys to fit Accumulo entries
into a subclass of BaseElem.
"""
def __init__(self, row, cf, cq, cv, ts):
self.row = row
self.cf = cf
self.cq = cq
self.cv = cv
self.ts = ts
def __cmp__(self, other):
if self.row == other.row:
if self.cf == other.cf:
if self.cq == other.cq:
if self.cv == other.cv:
if self.ts == other.ts:
return 0
else:
# timestamps are ordered in the other direction
return -1 if self.ts > other.ts else 1
else:
return -1 if self.cv < other.cv else 1
else:
return -1 if self.cq < other.cq else 1
else:
return -1 if self.cf < other.cf else 1
else:
return -1 if self.row < other.row else 1
def serialize(self):
return ','.join([self.row, self.cf, self.cq, self.cv, self.ts])
class AccumuloEntry(BaseElem):
keyclass = AccumuloKey
def __init__(self, key, val):
self.key = key
self.val = val
def serialize(self):
return ','.join([self.key.serialize(), self.val])
@classmethod
def deserialize(cls, s):
i = 0
j = s.find(',', i)
row = s[i:j]
i = j+1
j = s.find(',', i)
cf = s[i:j]
i = j+1
j = s.find(',', i)
cq = s[i:j]
i = j+1
j = s.find(',', i)
cv = s[i:j]
i = j+1
j = s.find(',', i)
ts = s[i:j]
val = s[j+1:]
return AccumuloEntry(cls.keyclass(row, cf, cq, cv, ts), val)
pace/ads/skiplist/elemclass_test.py
+225
View File
@@ -0,0 +1,225 @@
## **************
## Copyright 2014 MIT Lincoln Laboratory
## Project: PACE
## Authors: CS
## Description: Tests for skiplist element classes
## Modifications:
## Date Name Modification
## ---- ---- ------------
## 04 Oct 2014 CS Original file
## **************
import os
import sys
this_dir = os.path.dirname(os.path.abspath(__file__))
base_dir = os.path.join(this_dir, '../../..')
sys.path.append(base_dir)
from unittest import TestCase
import time
import random
from pace.ads.skiplist.elemclass import IntElem, StrElem, AccumuloKey, AccumuloEntry
from pace.common.pacetest import PACETestCase
def _fix_tup_sign(tup):
a, b, c, d, e = tup
return (a, b, c, d, -e)
class ElemClassTests(PACETestCase):
def setUp(self):
random.seed(int(time.time()))
def test_ord(self):
for i in range(self.num_iters):
epairs = [(random.randint(-100000, 100000),
random.randint(-100000, 100000))
for _ in range(self.size)]
epairs.append((0,0))
epairs.append((10000,10000))
epairs.append((-10000,-10000))
for x, y in epairs:
self.assertEqual(x < y, IntElem(x) < IntElem(y))
self.assertEqual(str(x) < str(y),
StrElem(str(x)) < StrElem(str(y)))
self.assertEqual(x > y, IntElem(x) > IntElem(y))
self.assertEqual(str(x) > str(y),
StrElem(str(x)) > StrElem(str(y)))
self.assertEqual(x >= y, IntElem(x) >= IntElem(y))
self.assertEqual(str(x) >= str(y),
StrElem(str(x)) >= StrElem(str(y)))
self.assertEqual(x <= y, IntElem(x) <= IntElem(y))
self.assertEqual(str(x) <= str(y),
StrElem(str(x)) <= StrElem(str(y)))
self.assertEqual(x == y, IntElem(x) == IntElem(y))
self.assertEqual(str(x) == str(y),
StrElem(str(x)) == StrElem(str(y)))
self.assertEqual(x != y, IntElem(x) != IntElem(y))
self.assertEqual(str(x) != str(y),
StrElem(str(x)) != StrElem(str(y)))
def test_serialize(self):
for i in range(self.num_iters):
elems = [random.randint(-100000, 100000)
for _ in range(self.size)]
for elem in elems:
ie = IntElem(elem)
se = StrElem(str(elem))
self.assertEqual(ie, IntElem.deserialize(ie.serialize()))
self.assertEqual(se, StrElem.deserialize(se.serialize()))
self.assertEqual(ie.serialize(),
IntElem.deserialize(
ie.serialize()).serialize())
self.assertEqual(se.serialize(),
StrElem.deserialize(se.serialize()).serialize())
def test_accumulo_serialize(self):
for i in range(self.num_iters):
for row, cf, cq, cv, ts, val in [[str(random.randint(-10000, 10000))
for i in range(6)]
for _ in range(self.size)]:
key = AccumuloKey(row, cf, cq, cv, ts)
entry = AccumuloEntry(key, val)
self.assertEqual(entry, AccumuloEntry.deserialize(entry.serialize()))
self.assertEqual(entry.val,
AccumuloEntry.deserialize(
entry.serialize()).val)
self.assertEqual(entry.serialize(),
AccumuloEntry.deserialize(
entry.serialize()).serialize())
def test_accumulo_ord(self):
for i in range(self.num_iters):
for tup1, tup2 in [([random.randint(-10000, 10000) * (-1 if i == 4 else 0)
for i in range(5)],
[random.randint(-10000, 10000) * (-1 if i == 4 else 0)
for i in range(5)])
for _ in range(self.size)]:
tup1 = tuple(tup1)
tup2 = tuple(tup2)
tk1 = _fix_tup_sign(tup1)
tk2 = _fix_tup_sign(tup2)
print 'tup1:', tup1
print 'tup2:', tup2
self.assertEqual(tup1 < tup2,
AccumuloKey(*tk1) < AccumuloKey(*tk2))
self.assertEqual(tup1 <= tup2,
AccumuloKey(*tk1) <= AccumuloKey(*tk2))
self.assertEqual(tup1 >= tup2,
AccumuloKey(*tk1) >= AccumuloKey(*tk2))
self.assertEqual(tup1 > tup2,
AccumuloKey(*tk1) > AccumuloKey(*tk2))
self.assertEqual(tup1 == tup2,
AccumuloKey(*tk1) == AccumuloKey(*tk2))
self.assertEqual(tup1 != tup2,
AccumuloKey(*tk1) != AccumuloKey(*tk2))
for tup1, tup2 in [([random.randint(-10000, 10000) * (0 if i < 1 else 1)
for i in range(5)],
[random.randint(-10000, 10000) * (0 if i < 1 else 1)
for i in range(5)])
for _ in range(self.size)]:
tup1 = tuple(tup1)
tup2 = tuple(tup2)
tk1 = _fix_tup_sign(tup1)
tk2 = _fix_tup_sign(tup2)
self.assertEqual(tup1 < tup2,
AccumuloKey(*tk1) < AccumuloKey(*tk2))
self.assertEqual(tup1 <= tup2,
AccumuloKey(*tk1) <= AccumuloKey(*tk2))
self.assertEqual(tup1 >= tup2,
AccumuloKey(*tk1) >= AccumuloKey(*tk2))
self.assertEqual(tup1 > tup2,
AccumuloKey(*tk1) > AccumuloKey(*tk2))
self.assertEqual(tup1 == tup2,
AccumuloKey(*tk1) == AccumuloKey(*tk2))
self.assertEqual(tup1 != tup2,
AccumuloKey(*tk1) != AccumuloKey(*tk2))
for tup1, tup2 in [([random.randint(-10000, 10000) * (0 if i < 2 else 1)
for i in range(5)],
[random.randint(-10000, 10000) * (0 if i < 2 else 1)
for i in range(5)])
for _ in range(self.size)]:
tup1 = tuple(tup1)
tup2 = tuple(tup2)
tk1 = _fix_tup_sign(tup1)
tk2 = _fix_tup_sign(tup2)
self.assertEqual(tup1 < tup2,
AccumuloKey(*tk1) < AccumuloKey(*tk2))
self.assertEqual(tup1 <= tup2,
AccumuloKey(*tk1) <= AccumuloKey(*tk2))
self.assertEqual(tup1 >= tup2,
AccumuloKey(*tk1) >= AccumuloKey(*tk2))
self.assertEqual(tup1 > tup2,
AccumuloKey(*tk1) > AccumuloKey(*tk2))
self.assertEqual(tup1 == tup2,
AccumuloKey(*tk1) == AccumuloKey(*tk2))
self.assertEqual(tup1 != tup2,
AccumuloKey(*tk1) != AccumuloKey(*tk2))
for tup1, tup2 in [([random.randint(-10000, 10000) * (0 if i < 3 else 1)
for i in range(5)],
[random.randint(-10000, 10000) * (0 if i < 3 else 1)
for i in range(5)])
for _ in range(self.size)]:
tup1 = tuple(tup1)
tup2 = tuple(tup2)
tk1 = _fix_tup_sign(tup1)
tk2 = _fix_tup_sign(tup2)
self.assertEqual(tup1 < tup2,
AccumuloKey(*tk1) < AccumuloKey(*tk2))
self.assertEqual(tup1 <= tup2,
AccumuloKey(*tk1) <= AccumuloKey(*tk2))
self.assertEqual(tup1 >= tup2,
AccumuloKey(*tk1) >= AccumuloKey(*tk2))
self.assertEqual(tup1 > tup2,
AccumuloKey(*tk1) > AccumuloKey(*tk2))
self.assertEqual(tup1 == tup2,
AccumuloKey(*tk1) == AccumuloKey(*tk2))
self.assertEqual(tup1 != tup2,
AccumuloKey(*tk1) != AccumuloKey(*tk2))
for tup1, tup2 in [([random.randint(-10000, 10000) * (0 if i < 4 else 1)
for i in range(5)],
[random.randint(-10000, 10000) * (0 if i < 4 else 1)
for i in range(5)])
for _ in range(self.size)]:
tup1 = tuple(tup1)
tup2 = tuple(tup2)
tk1 = _fix_tup_sign(tup1)
tk2 = _fix_tup_sign(tup2)
print 'tup1:', tup1
print 'tup2:', tup2
self.assertEqual(tup1 < tup2,
AccumuloKey(*tk1) < AccumuloKey(*tk2))
self.assertEqual(tup1 <= tup2,
AccumuloKey(*tk1) <= AccumuloKey(*tk2))
self.assertEqual(tup1 >= tup2,
AccumuloKey(*tk1) >= AccumuloKey(*tk2))
self.assertEqual(tup1 > tup2,
AccumuloKey(*tk1) > AccumuloKey(*tk2))
self.assertEqual(tup1 == tup2,
AccumuloKey(*tk1) == AccumuloKey(*tk2))
self.assertEqual(tup1 != tup2,
AccumuloKey(*tk1) != AccumuloKey(*tk2))
pace/ads/skiplist/skiplist.py
+162
View File
@@ -0,0 +1,162 @@
## **************
## Copyright 2014 MIT Lincoln Laboratory
## Project: PACE
## Authors: ZS
## Description: Skip list library
## Modifications:
## Date Name Modification
## ---- ---- ------------
## 19 Aug 2014 ZS Original file
## **************
import random
from hashlib import sha256
from pace.ads.skiplist.skiplistnode import SkipListNode, InvalidSkipListException
from pace.ads.skiplist.coin import BaseCoin
class SkipList(object):
# the class to be used for nodes
nodeclass = SkipListNode
def __init__(self, root, lbound, rbound, coin):
self.root = root
self.lbound = lbound
self.rbound = rbound
self.coin = coin
@classmethod
def new(cls, elems, lbound, rbound, coin=BaseCoin()):
""" Build a new SkipList
Arguments:
elems - the list of elements to put into the SkipList
lbound - the leftmost element in the SkipList (acts like -infty)
rbound - the rightmost element in the Skiplist (acts like +infty)
"""
sl = cls(None, lbound, rbound, coin)
right = cls.nodeclass.newnode(sl, None, None, rbound, True)
left = cls.nodeclass.newnode(sl, None, right, lbound, True)
sl.root = left
for elem in elems:
sl.insert(elem)
return sl
def insert(self, elem):
""" Insert elem into the base level of the current SkipList, then
determine how many levels to elevate it by
"""
new_node, left_neighbor, visited = self.root.insert(elem)
current = new_node
current_hash_value = sha256(str(elem)).digest()
for v, flag in visited:
if not flag:
continue
if self.stop(current_hash_value):
return
current_hash_value = sha256(current_hash_value).digest()
next_level = self.nodeclass.newnode(self, current, v.right, elem)
v.right = next_level
current = next_level
# If we got here, we need to add another level of just -infty, +infty
new_rght = self.nodeclass.newnode(self, self.root.right.right, None, self.rbound)
new_root = self.nodeclass.newnode(self, self.root, new_rght, self.lbound)
self.root = new_root
while not self.stop(current_hash_value):
current_hash_value = sha256(current_hash_value).digest()
# Add higher levels
old_right = self.root.right
old_left = self.root
new_right = self.nodeclass.newnode(self, old_right, None, old_right.elem)
new_left = self.nodeclass.newnode(self, old_left, new_right, old_left.elem)
new_level = self.nodeclass.newnode(self, current, old_right, elem)
old_left.right = new_level
self.root = new_left
current = new_level
def contains(self, elem):
""" Returns True if elem is in the skip list, and False otherwise
"""
visited, last = self.root.search(elem)
return last.elem == elem
def stop(self, elem):
""" The 'coin flip' for a randomized SkipList to determine how
high to elevate each element once it's inserted
"""
return self.coin.flip(elem)
def to_list_of_lists(self):
""" Returns the list containing each level in the skiplist, from top
to bottom.
The first element of the result will just contain the left and
right bounds ("-infty" and "+infty"), and the last element will
contain those bounds with every element of the list between them.
"""
current = self.root
lists = []
while current:
row = current
rlist = []
while row:
rlist.append(row.elem)
row = row.right
lists.append(rlist)
current = current.down
return lists
def valid(self):
## Guarantee that the top level has exactly two elements
if self.root is None:
raise InvalidSkipListException(
'Top level must have exactly two elements (found: zero)')
if self.root.right is None:
raise InvalidSkipListException(
'Top level must have exactly two elements (found: one)')
if self.root.right.right is not None:
raise InvalidSkipListException(
'Top level must have exactly two elements (found: more than 2)')
## Guarantee each level has strictly fewer elements than the one after
prev_elems = 2
current_left = self.root.down
while current_left:
current = current_left
current_elems = 0
while current:
current_elems = current_elems + 1
current = current.right
if not (current_elems >= prev_elems):
raise InvalidSkipListException(
'Each level must have at least as many nodes as the previous level')
prev_elems = current_elems
current_left = current_left.down
## Guarantee various per-node correctness properties
self.root.valid()
pace/ads/skiplist/skiplist_server.py
+346
View File
@@ -0,0 +1,346 @@
## **************
## Copyright 2014 MIT Lincoln Laboratory
## Project: PACE
## Authors: CS
## Description: Skiplist server/data publisher code
## Modifications:
## Date Name Modification
## ---- ---- ------------
## 10 Oct 2014 CS Original file (copied from ../merkle/mht_server.py)
## **************
import socket
import logging
from abc import abstractmethod, ABCMeta
from SocketServer import StreamRequestHandler, TCPServer, ThreadingMixIn
from threading import Thread
from base64 import b64encode, b64decode
from pace.ads.skiplist.authskiplist import AuthSkipList
from pace.ads.skiplist.skiplistvo import SkipListVO
from pace.ads.skiplist.coin import SeededCoin, PrefixCoin, HashCoin
def _recv(sfile, loc=''):
return sfile.readline().strip()
def _recvi(sfile, i, loc=''):
return [_recv(sfile, loc) for x in range(0, i)]
def _send(wfile, msg, loc=''):
total_msg = msg + '\n'
wfile.write(total_msg)
wfile.flush()
def _sendi(wfile, msgs, loc=''):
for msg in msgs:
_send(wfile, msg, loc)
def _recv_iterate(rfile, name):
while True:
x = _recv(rfile)
if x == name:
return
else:
yield x
class SkipListServerException(Exception):
def __init__(self, msg):
self.msg = msg
class SLHandler(StreamRequestHandler):
def handle(self):
# Initialize the skiplist
# Can have a persistent skiplist once batch insertion has been
# implemented.
coin_type = _recv(self.rfile, 'handler')
if coin_type == 'SEED':
raw_seed = _recv(self.rfile, 'handler')
try:
seed = int(raw_seed)
except ValueError:
raise SkipListServerException(
'Non-number %s passed in for seed value' %raw_seed)
coin = SeededCoin(seed)
elif coin_type == 'PREFIX':
prefix = [p == 'True' for p in _recv_iterate(self.rfile, 'ELEMS')]
coin = PrefixCoin(prefix)
elif coin_type == 'HASH':
coin = HashCoin()
else:
raise SkipListServerException('Invalid coin type %s' %coin_type)
elems = [self.server.elemClass.deserialize(elem)
for elem in _recv_iterate(self.rfile, 'END_ELEMS')]
# Protocol: send the lower bound, then the upper bound, then the entire
# list of elements to insert at first
sl = AuthSkipList.new(elems[2:], elems[0], elems[1], coin)
while True:
task = self.rfile.readline().strip()
if task == 'QUERY':
raw_elem = _recv(self.rfile, 'handler')
elem = self.server.elemClass.deserialize(raw_elem)
proof = sl.contains(elem)
sproof = SLHandler.serialize_query_result(proof)
_send(self.wfile, sproof, 'handler')
elif task == 'RANGE_QUERY':
raw_lower, raw_upper = _recvi(self.rfile, 2, 'handler')
lower, upper = (self.server.elemClass.deserialize(raw_lower),
self.server.elemClass.deserialize(raw_upper))
vo = SkipListVO.range_query(sl, lower, upper)
_send(self.wfile, vo.serialize(), 'handler')
elif task == 'INSERT':
for elem in _recv_iterate(self.rfile, 'END_INSERT'):
sl.insert(self.server.elemClass.deserialize(elem))
elif task == 'REPREFIX':
prefix = [int(y)
for y in _recv_iterate(self.rfile, 'END_REPREFIX')]
sl.coin.extend(prefix)
elif task == 'RESEED':
seed = int(_recv(self.rfile))
sl.coin.reseed(seed)
elif task == 'PING':
_send(self.wfile, 'PONG')
elif task == 'QUIT':
return
else:
raise Exception('Unknown task %s' %task)
@staticmethod
def serialize_query_result(qr):
""" Serialization protocol for query result lists.
The result of a query is the result of joining the proof list
together with the ';' delimiter, then pairing that with the boolean
telling whether or not the element was found with the ',' delimiter
"""
found, proof = qr
inner = ';'.join([b64encode(hval) for hval in proof])
return ','.join([str(found), inner])
class SocketSLServer(ThreadingMixIn, TCPServer):
""" A skiplist server that serves the skiplist over a socket, freeing
the client from having to store anything more than a
handle on the server to send & receive information to & from.
(this will become more interesting once it starts to serve the same
skiplist to multiple clients)
"""
def __init__(self, elemClass, *args, **kwargs):
self.elemClass = elemClass
TCPServer.__init__(self, *args, **kwargs)
class AbstractSocketSLClient(object):
__metaclass__ = ABCMeta
def __init__(self, sock):
self.sock = sock
self.rfile = sock.makefile('r')
self.wfile = sock.makefile('w')
def __enter__(self):
return self
def __exit__(self, type, value, tb):
_send(self.wfile, 'QUIT')
self.rfile.close()
self.wfile.close()
self.sock.close()
@staticmethod
def deserialize_query_result(qr):
found, proof = qr.split(',')
return found == 'True', [b64decode(s) for s in proof.split(';')]
@abstractmethod
def new(elems, min_elem, max_elem, seed, elemClass, host='localhost', port=9999):
""" Creates a connection to a server and initializes it. Assumes
that the server's SkipList has not already been initialized.
"""
pass
@abstractmethod
def initialize(self, elems):
""" Initialize the server being connected to. Should be called exactly
once per server.
Arguments may vary depending on the child class.
"""
pass
@classmethod
def connect(cls, host='localhost', port=9999):
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect((host, port))
return cls(sock)
def query(self, elem):
""" Query to see whether an element is in the skiplist.
Arguments:
self - the skiplist object
elem - the object to query
Returns:
None if elem was not contained in the skiplist, or a proof
that it was contained in the tree.
"""
_send(self.wfile, 'QUERY')
_send(self.wfile, elem.serialize(), 'client')
proof = _recv(self.rfile, 'client')
return AbstractSocketSLClient.deserialize_query_result(proof)
def range_query(self, lower, upper):
""" Query a range of elements beteween lower and upper (inclusive)
Arguments:
self - the skiplist object
lower - the (inclusive) lower bound of the range
upper - the (inclusive) upper bound of the range
Returns:
vo - a verification object (as in vo.py) for the given range
"""
_send(self.wfile, 'RANGE_QUERY')
_sendi(self.wfile, [lower.serialize(), upper.serialize()])
vo = _recv(self.rfile, 'client')
return SkipListVO.deserialize(vo, self.elemClass)
def ping(self):
""" Ping a server. Mostly for testing/benchmarking purposes; assumes
it can't respond until it's finished with its previous task.
"""
_send(self.wfile, 'PING')
_recv(self.rfile)
class SeededSocketSLClient(AbstractSocketSLClient):
@staticmethod
def new(elems, min_elem, max_elem, seed, elemClass, host='localhost', port=9999):
""" Creates a connection to a server and initializes it. Assumes
that the server's SkipList has not already been initialized.
"""
s = SeededSocketSLClient.connect(host, port)
s.initialize(seed, [min_elem, max_elem] + elems)
s.elemClass = elemClass
return s
def initialize(self, seed, elems):
_send(self.wfile, 'SEED', 'client')
_send(self.wfile, str(seed), 'client')
for elem in elems:
_send(self.wfile, elem.serialize(), 'client')
_send(self.wfile, 'END_ELEMS', 'client')
def batch_insert(self, expected_initial_root, elems, least, greatest, seed):
vo = self.range_query(least, greatest)
vo.verify(least, greatest, expected_initial_root)
_send(self.wfile, 'RESEED')
_send(self.wfile, str(seed))
_send(self.wfile, 'INSERT')
vo.coin = SeededCoin(seed)
for elem in elems:
_send(self.wfile, elem.serialize())
vo.insert(elem)
_send(self.wfile, 'END_INSERT')
return vo.root.label
class PrefixSocketSLClient(AbstractSocketSLClient):
@staticmethod
def new(elems, min_elem, max_elem, prefix, elemClass, host='localhost', port=9999):
""" Creates a connection to a server and initializes it. Assumes
that the server's SkipList has not already been initialized.
"""
s = PrefixSocketSLClient.connect(host, port)
s.initialize(prefix, [min_elem, max_elem] + elems)
s.elemClass = elemClass
return s
def initialize(self, prefix, elems):
_send(self.wfile, 'PREFIX', 'client')
for p in prefix:
_send(self.wfile, str(p), 'client')
_send(self.wfile, 'ELEMS', 'client')
for elem in elems:
_send(self.wfile, elem.serialize(), 'client')
_send(self.wfile, 'END_ELEMS', 'client')
# TODO: might be a way to make sure we only have to go through elems once
def batch_insert(self, expected_initial_root, elems, least, greatest):
vo = self.range_query(least, greatest)
vo.verify(least, greatest, expected_initial_root)
vo.coin = RecordedCoin()
for elem in elems:
vo.insert(elem)
_send(self.wfile, 'REPREFIX')
for p in vo.coin.read():
_send(self.wfile, str(p))
_send(self.wfile, 'END_REPREFIX')
_send(self.wfile, 'INSERT')
vo.coin = PrefixCoin(seed)
for elem in elems:
_send(self.wfile, elem.serialize())
_send(self.wfile, 'END_INSERT')
return vo.root.label
class HashSocketSLClient(AbstractSocketSLClient):
@staticmethod
def new(elems, min_elem, max_elem, elemClass, host='localhost', port=9999):
""" Creates a connection to a server and initializes it. Assumes
that the server's SkipList has not already been initialized.
"""
s = HashSocketSLClient.connect(host, port)
s.initialize([min_elem, max_elem] + elems)
s.elemClass = elemClass
return s
def initialize(self, elems):
_send(self.wfile, 'HASH', 'client')
for elem in elems:
_send(self.wfile, elem.serialize(), 'client')
_send(self.wfile, 'END_ELEMS', 'client')
# TODO: might be a way to make sure we only have to go through elems once
def batch_insert(self, expected_initial_root, elems, least, greatest):
vo = self.range_query(least, greatest)
vo.coin = HashCoin()
vo.verify(least, greatest, expected_initial_root)
_send(self.wfile, 'INSERT')
for elem in elems:
_send(self.wfile, elem.serialize())
vo.insert(elem)
_send(self.wfile, 'END_INSERT')
return vo.root.label
pace/ads/skiplist/skiplist_server_test.py
+379
View File
@@ -0,0 +1,379 @@
## **************
## Copyright 2014 MIT Lincoln Laboratory
## Project: PACE
## Authors: CS
## Description: Unit tests for SkipList servers
## Modifications:
## Date Name Modification
## ---- ---- ------------
## 10 Oct 2014 CS Original file (copied from ../merkle/mht_server_test.py
## **************
import os
import sys
this_dir = os.path.dirname(os.path.abspath(__file__))
base_dir = os.path.join(this_dir, '../../..')
sys.path.append(base_dir)
from unittest import TestCase
import random
import socket
import time
from threading import Thread
from pace.ads.skiplist.authskiplist import AuthSkipList
from pace.ads.skiplist.skiplist_server import SocketSLServer, SLHandler, SeededSocketSLClient, PrefixSocketSLClient, HashSocketSLClient
from pace.ads.skiplist.skiplistvo import SkipListVO, VerificationObjectException
from pace.ads.skiplist.coin import SeededCoin, RecordedCoin, HashCoin
from pace.ads.skiplist.elemclass import IntElem
from pace.common.pacetest import PACETestCase
class SLServerTest(PACETestCase):
@staticmethod
def run_server(server):
server.serve_forever()
def setUp(self):
# Need the extra randomness to make sure it doesn't generate the
# same seed on different tests.
random.seed(int(time.time()) + random.randint(0, 1000))
self.host = 'localhost'
self.port = random.randint(9000, 9999)
# Try 10 times to get a valid port; if not, will fail. If this
# fails, check to make sure ports 9000-9999 are not in use.
for _ in xrange(10):
try:
self.server = SocketSLServer(IntElem, (self.host, self.port),
SLHandler)
break
except socket.error:
self.port = random.randint(9000, 9999)
self.thread = Thread(group=None, target=SLServerTest.run_server,
name=None, args=[self.server], kwargs={})
self.thread.daemon = True
self.thread.start()
print 'Starting test',
self.start_time = time.time()
def tearDown(self):
self.end_time = time.time()
print 'Test finished. Time taken: %s' %str(self.end_time - self.start_time)
self.server.shutdown()
self.thread.join()
def test_remote_prefix_query(self):
for i in range(0, self.num_iters):
elems = map(IntElem, self.generate_elems())
elems.sort()
coin = RecordedCoin()
sl = AuthSkipList.new(elems, IntElem(-1), IntElem(1000000001), coin)
with PrefixSocketSLClient.new(
elems, IntElem(-1), IntElem(1000000001), coin.record,
IntElem, self.host, self.port) as client:
for j in range(0, self.num_iters):
elem = random.choice(elems)
good_result = sl.contains(elem)
test_result = client.query(elem)
self.assertEqual(good_result, test_result)
def test_remote_seed_query(self):
for i in range(0, self.num_iters):
elems = map(IntElem, self.generate_elems())
elems.sort()
seed = random.randint(0, 100000)
sl = AuthSkipList.new(elems, IntElem(-1), IntElem(1000000001), SeededCoin(seed))
with SeededSocketSLClient.new(
elems, IntElem(-1), IntElem(1000000001), seed, IntElem, self.host, self.port) as client:
for j in range(0, self.num_iters):
elem = random.choice(elems)
good_result = sl.contains(elem)
test_result = client.query(elem)
self.assertEqual(good_result, test_result)
def test_remote_hash_query(self):
for i in range(0, self.num_iters):
elems = map(IntElem, self.generate_elems())
elems.sort()
sl = AuthSkipList.new(elems, IntElem(-1), IntElem(1000000001), HashCoin())
with HashSocketSLClient.new(
elems, IntElem(-1), IntElem(1000000001), IntElem, self.host, self.port) as client:
for j in range(0, self.num_iters):
elem = random.choice(elems)
good_result = sl.contains(elem)
test_result = client.query(elem)
self.assertEqual(good_result, test_result)
def test_remote_range_query(self):
total_time = 0
for i in range(0, self.num_iters):
elems = map(IntElem, self.generate_elems())
elems.sort()
seed = random.randint(0, 100000)
sl = AuthSkipList.new(elems, IntElem(-1), IntElem(1000000001), SeededCoin(seed))
start = time.time()
with SeededSocketSLClient.new(
elems, IntElem(-1), IntElem(1000000001), seed, IntElem, self.host, self.port) as client:
## Get valid lower & upper bounds with enough room between
## them (about a third of the list) for a substantial range
## query
lower = random.choice(elems[1:self.size/3])
upper = random.choice(elems[2*self.size/3:-1])
self.assertEqual(SkipListVO.range_query(sl, lower, upper),
client.range_query(lower, upper))
end = time.time()
total_time = total_time + (end - start)
print 'Average time taken: %s' %str(total_time / self.num_iters)
def test_remote_range_query_2(self):
for i in range(0, self.num_iters):
orig_seed = random.randint(0, 10000000)
elems = map(IntElem, self.generate_elems())
elems.sort()
sl = AuthSkipList.new(elems, IntElem(elems[0].key-1),
IntElem(elems[-1].key+1),
SeededCoin(orig_seed))
with SeededSocketSLClient.new(
elems, IntElem(elems[0].key-1), IntElem(elems[-1].key+1),
orig_seed, IntElem, self.host, self.port) as client:
lower = random.choice(elems[1:self.size/3])
upper = random.choice(elems[2*self.size/3:-1])
localvo = SkipListVO.range_query(sl, lower, upper)
othervo = client.range_query(lower, upper)
try:
ret_elems = localvo.verify(lower, upper, sl.root.label)
except VerificationObjectException as e:
self.assertTrue(False, 'Error: %s' %e.msg)
self.assertEqual(ret_elems, [x for x in elems
if lower <= x <= upper])
try:
ret_elems = othervo.verify(lower, upper, sl.root.label)
except VerificationObjectException as e:
self.assertTrue(False, 'Error: %s' %e.msg)
self.assertEqual(ret_elems, [x for x in elems
if lower <= x <= upper])
self.assertEqual(localvo, othervo)
def test_remote_batch_insert(self):
for i in range(0, self.num_iters):
orig_seed = random.randint(0, 10000000)
elems = map(IntElem, self.generate_elems())
elems.sort()
sl = AuthSkipList.new(elems, IntElem(elems[0].key-1),
IntElem(elems[-1].key+1),
SeededCoin(orig_seed))
old_root = sl.root.label
self.assertEqual(len(sl.to_list_of_lists()[-1])-2, self.size)
## Generate a small number (in this case, n/10) of random elements
## between the minimum and maximum elements (not including the
## boundary elements) to insert.
new_elems = map(IntElem,
self.generate_elems(
elems[1].key, elems[-2].key, self.size/10))
min_ne = min(new_elems)
max_ne = max(new_elems)
vo = SkipListVO.range_query(sl, min_ne, max_ne)
try:
ret_elems = vo.verify(min_ne, max_ne, sl.root.label)
except VerificationObjectException as e:
self.assertTrue(False, 'Error: %s' %e.msg)
self.assertEqual(ret_elems, [x for x in elems
if min_ne <= x <= max_ne])
seed = random.randint(0, 1000000)
vo.coin = SeededCoin(seed)
sl.coin = SeededCoin(seed)
self.assertEqual(vo.root.label, sl.root.label)
with SeededSocketSLClient.new(
elems, IntElem(elems[0].key-1), IntElem(elems[-1].key+1),
orig_seed, IntElem, self.host, self.port) as client:
lower = random.choice(elems[1:self.size/3])
upper = random.choice(elems[2*self.size/3:-1])
self.assertEqual(SkipListVO.range_query(sl, lower, upper),
client.range_query(lower, upper))
self.assertEqual(SkipListVO.range_query(sl, min_ne, max_ne),
client.range_query(min_ne, max_ne))
for elem in new_elems:
vo.insert(elem)
for elem in new_elems:
sl.insert(elem)
self.assertEqual(sl.root.label, vo.root.label)
try:
new_root = client.batch_insert(old_root, new_elems,
min_ne, max_ne, seed)
except VerificationObjectException as e:
self.assertTrue(False, 'Error: %s' %e.msg)
self.assertEqual(new_root, vo.root.label)
self.assertEqual(sl.root.label, new_root)
self.assertEqual(len(sl.to_list_of_lists()[-1])-2,
self.size + (self.size/10))
for elem in new_elems:
proof = client.query(elem)
self.assertEqual(proof, sl.contains(elem))
self.assertEqual(AuthSkipList.verify(proof[1]), sl.root.label)
def test_remote_batch_insert_helper(self):
for i in range(0, self.num_iters):
orig_seed = random.randint(0, 10000000)
elems = map(IntElem, self.generate_elems())
elems.sort()
sl = AuthSkipList.new(elems, IntElem(elems[0].key-1),
IntElem(elems[-1].key+1),
SeededCoin(orig_seed))
new_elems = [IntElem(random.randint(elems[1].key, elems[-2].key))
for i in range(0, 2)]
min_ne = min(new_elems)
max_ne = max(new_elems)
while min_ne == max_ne:
new_elems = [IntElem(random.randint(elems[1].key, elems[-2].key))
for i in range(0, 2)]
min_ne = min(new_elems)
max_ne = max(new_elems)
vo = SkipListVO.range_query(sl, min_ne, max_ne)
try:
ret_elems = vo.verify(min_ne, max_ne, sl.root.label)
except VerificationObjectException as e:
self.assertTrue(False, 'Error: %s' %e.msg)
seed = random.randint(0, 1000000)
vo.coin = SeededCoin(seed)
sl.coin = SeededCoin(seed)
self.assertEqual(vo.root.label, sl.root.label)
orig_label = sl.root.label
for elem in new_elems:
vo.insert(elem)
sl.insert(elem)
self.assertEqual(sl.root.label, vo.root.label, 'failed on index %d' %new_elems.index(elem))
self.assertEqual(sl.root.label, vo.root.label)
with SeededSocketSLClient.new(
elems, IntElem(elems[0].key-1), IntElem(elems[-1].key+1),
orig_seed, IntElem, self.host, self.port) as client:
try:
new_root = client.batch_insert(orig_label, new_elems,
min_ne, max_ne, seed)
except VerificationObjectException as e:
self.assertTrue(False, 'Error: %s' %e.msg)
self.assertEqual(new_root, vo.root.label)
self.assertEqual(sl.root.label, new_root)
self.assertEqual(len(sl.to_list_of_lists()[-1])-2,
len(elems) + len(new_elems))
for elem in new_elems:
proof = client.query(elem)
self.assertEqual(proof, sl.contains(elem))
self.assertEqual(AuthSkipList.verify(proof[1]), sl.root.label)
def test_remote_batch_insert_hash_coin(self):
for i in range(0, self.num_iters):
elems = map(IntElem, self.generate_elems())
elems.sort()
sl = AuthSkipList.new(elems, IntElem(elems[0].key-1),
IntElem(elems[-1].key+1), HashCoin())
old_root = sl.root.label
self.assertEqual(len(sl.to_list_of_lists()[-1])-2, self.size)
## Generate a small number (in this case, n/10) of random elements
## between the minimum and maximum elements (not including the
## boundary elements) to insert.
new_elems = [IntElem(random.randint(elems[1].key, elems[-2].key))
for i in range(0, self.size/10)]
new_elems = map(IntElem,
self.generate_elems(
elems[1].key, elems[-2].key, self.size/10))
min_ne = min(new_elems)
max_ne = max(new_elems)
vo = SkipListVO.range_query(sl, min_ne, max_ne, HashCoin())
try:
ret_elems = vo.verify(min_ne, max_ne, sl.root.label)
except VerificationObjectException as e:
self.assertTrue(False, 'Error: %s' %e.msg)
self.assertEqual(ret_elems, [x for x in elems
if min_ne <= x <= max_ne])
self.assertEqual(vo.root.label, sl.root.label)
with HashSocketSLClient.new(
elems, IntElem(elems[0].key-1), IntElem(elems[-1].key+1),
IntElem, self.host, self.port) as client:
lower = random.choice(elems[1:self.size/3])
upper = random.choice(elems[2*self.size/3:-1])
self.assertEqual(SkipListVO.range_query(sl, lower, upper),
client.range_query(lower, upper))
self.assertEqual(SkipListVO.range_query(sl, min_ne, max_ne),
client.range_query(min_ne, max_ne))
for elem in new_elems:
vo.insert(elem)
for elem in new_elems:
sl.insert(elem)
self.assertEqual(sl.root.label, vo.root.label)
try:
new_root = client.batch_insert(old_root, new_elems,
min_ne, max_ne)
except VerificationObjectException as e:
self.assertTrue(False, 'Error: %s' %e.msg)
self.assertEqual(new_root, vo.root.label)
self.assertEqual(sl.root.label, new_root)
self.assertEqual(len(sl.to_list_of_lists()[-1])-2,
self.size + (self.size/10))
for elem in new_elems:
proof = client.query(elem)
self.assertEqual(proof, sl.contains(elem))
self.assertEqual(AuthSkipList.verify(proof[1]), sl.root.label)
pace/ads/skiplist/skiplist_test.py
+75
View File
@@ -0,0 +1,75 @@
## **************
## Copyright 2014 MIT Lincoln Laboratory
## Project: PACE
## Authors: ZS
## Description: Unit tests for skip list library
## Modifications:
## Date Name Modification
## ---- ---- ------------
## 19 Aug 2014 ZS Original file
## **************
import os
import sys
this_dir = os.path.dirname(os.path.abspath(__file__))
base_dir = os.path.join(this_dir, '../../..')
sys.path.append(base_dir)
from unittest import TestCase
import time
import random
from pace.ads.skiplist.skiplist import SkipList
from pace.common.pacetest import PACETestCase
class SkipListTests(PACETestCase):
def setUp(self):
random.seed(int(time.time()))
def test_valid(self):
""" Ensure that we only create valid skip lists
"""
for i in range(0, self.num_iters):
elems = [random.randint(0, 1000000000)
for i in range(0, self.size)]
sl = SkipList.new(elems, -1, 1000000001)
try:
sl.valid()
except InvalidSkipListException as e:
self.assertTrue(False, 'Error: %s' %e.msg)
def test_membership(self):
""" Ensure that every element of the skip list is reflected as in it.
"""
for i in range(0, self.num_iters):
elems = [random.randint(0, 100000000) for i in range(0, self.size)]
sl = SkipList.new(elems, -1, 1000000001)
for elem in elems:
self.assertTrue(sl.contains(elem),
'skip list claims not to have element %d' %elem)
def test_non_membership(self):
""" Ensure that non-elements of the skip list are not in it
"""
for i in range(0, self.num_iters):
elems = [random.randint(0, 100000000) for i in range(0, self.size)]
sl = SkipList.new(elems, -1, 1000000001)
self.assertEqual(sl.contains(-2), False,
"skip list thinks it contains -2 (it doesn't)")
self.assertEqual(sl.contains(100000002), False,
"skip list thinks it contains 100000002 (it doesn't)")
bad_elem = random.randint(0, 100000000)
while bad_elem in elems:
bad_elem = random.randint(0, 100000000)
self.assertEqual(sl.contains(bad_elem), False,
"skip list thinks it contains %d (it doesn't)" %bad_elem)
pace/ads/skiplist/skiplistnode.py
+111
View File
@@ -0,0 +1,111 @@
## **************
## Copyright 2014 MIT Lincoln Laboratory
## Project: PACE
## Authors: ZS
## Description: Skip list node library
## Modifications:
## Date Name Modification
## ---- ---- ------------
## 19 Aug 2014 ZS Original file
## **************
from collections import deque
class SkipListNode(object):
def __init__(self, sl, down, right, elem):
self.sl = sl
self.down = down
self.right = right
self.elem = elem
@staticmethod
def newnode(sl, down, right, elem, *args, **kwargs):
""" Create a new node. Can be overridden by subclasses.
Arguments:
sl - the skiplist this node is a part of.
down - the bottom neighbor of the new node
right - the right neighbor of the new node
elem - the element to be stored in the new node.
*args, **kwargs - any more arguments that a subclass
might want to use
Returns:
A new SkipListNode with fields as given.
"""
new_node = SkipListNode(sl, down, right, elem)
return new_node
def search(self, elem):
""" Search the SkipList for node containing an element that is
either equal to elem (if elem is in the list) or the closest
element in the list that is less than elem (if elem is not in the
list)
Argument:
elem - the element to be searched for
Returns:
visited - a collection of nodes visited, stored in reverse order
current - the closest node in the base list that is less than or
equal to elem
"""
current = self
visited = deque([])
while (isinstance(current.right, type(self)) and
current.right.elem <= elem):
visited.appendleft((current, False))
current = current.right
while current.down:
visited.appendleft((current, True))
current = current.down
while (isinstance(current.right, type(self)) and
current.right.elem <= elem):
visited.appendleft((current, False))
current = current.right
return visited, current
def insert(self, elem):
""" Insert an element into the base list.
NB: this does NOT perform the 'coin flips' to determine which
other lists the element being inserted will go in; it only
adds it to the base lists. Elevation is the job of
SkipList.insert.
"""
visited, left_neighbor = self.search(elem)
new_node = self.newnode(self.sl, None, left_neighbor.right, elem)
left_neighbor.right = new_node
return new_node, left_neighbor, visited
def valid(self, recur=True):
current = self
while current:
crow = current
while crow:
# Nodes must appear in sorted order
if crow.right:
if not (crow.elem <= crow.right.elem):
raise InvalidSkipListException(
'Nodes must appear in sorted order')
# The node beneath a node must have the same element
if crow.down:
if not (crow.elem == crow.down.elem):
raise InvalidSkipListException(
'The node beneath a node must have the same element')
crow = crow.right
current = current.down
class InvalidSkipListException(Exception):
def __init__(self, msg):
self.msg = msg
pace/ads/skiplist/skiplistvo.py
+415
View File
@@ -0,0 +1,415 @@
## **************
## Copyright 2014 MIT Lincoln Laboratory
## Project: PACE
## Authors: ZS
## Description: Verification objects for skiplists
## Modifications:
## Date Name Modification
## ---- ---- ------------
## 4 Sep 2014 ZS Copied from python/merkle/ to use with skiplists
## 18 Jul 2014 ZS Original file (python/merkle/vo.py)
## **************
from hashlib import sha256
from base64 import b64encode, b64decode
import bisect
from pace.ads.skiplist.authskiplist import AuthSkipList
from pace.ads.skiplist.vonode import VONode, VerificationObjectException
from pace.ads.skiplist.coin import BaseCoin
class SkipListVO(AuthSkipList):
""" Verification objects for skiplists.
Note that while this is built on skiplists as a base class, a
skip list VO is _not_ necessarily a valid skip list. There are
two major distinctions for the current, unoptimized case:
1) The VO has a tree-like structure. In particular, if the right
neighbor of a node in a skiplist is irrelevant to the root
label contribution of the current path, its corresponding node
in the VO will have an empty entry for its right neighbor.
2) If a branch has no elements in the query's range on it, it will
get collapsed to its label value, which will be inserted at the
appropriate point in the VO in lieu of the branch itself.
"""
nodeclass = VONode
def __eq__(self, other):
if self.lbound != other.lbound:
return False
if self.rbound != other.rbound:
return False
if not (self.root == other.root):
return False
return True
def __neq__(self, other):
return not self.__eq__(other)
@classmethod
def new(cls, auth_skiplist, lbound, rbound, coin=BaseCoin()):
""" Create a new VO from an AuthSkipList for a given range.
The new VO has a tree-like structure. Notably, it does NOT have
the usual grid-like structure of a skiplist; that is, there is
not a horizontal connection between each element of each row.
"""
vo = cls(None, lbound, rbound, coin)
vo_root = vo.build_node(auth_skiplist.root)
vo.root = vo_root
return vo
@classmethod
def range_query(cls, auth_skiplist, lbound, rbound, coin=BaseCoin()):
""" Do a range query on a skiplist with a given range. Same as new()
"""
return cls.new(auth_skiplist, lbound, rbound, coin)
def serialize(self):
return ';'.join(
[self.lbound.serialize(), self.rbound.serialize(),
self.root.serialize()])
@staticmethod
def deserialize(node, elemClass):
lbound, rbound, tree = node.split(';')
vo = SkipListVO(None,
elemClass.deserialize(lbound),
elemClass.deserialize(rbound),
None)
vo.root = VONode.deserialize(vo, tree, elemClass)
return vo
def insert(self, elem):
""" Insert the element 'elem' into 'self'. The result of this should
be the result of inserting 'elem' into the skiplist that generated
the VO, then performing the same range query on it (assuming both
coins produce the same flips). 'elem' must be within the VO's
range.
"""
visited, left_neighbor, index = self.root.search(elem)
## First, insert the element into the list---note that this does NOT
## label the node; that must be done later
## NB: if the node doesn't move up, 'elem' just gets inserted into the
## list of elements in left_neighbor. if it does go up, it gets
## the elements after a certain point in its own leaf list, and
## left_neighbor replaces those elements with 'elem'
elem_leaf = None
current = elem_leaf
stopped = False
current_hash_value = sha256(elem.serialize()).digest()
new_elems = []
## Determine how many levels of the skiplist to insert the new
## element on. Each time we visited a node in the traversal, if we
## dropped down (instead of going right), that node is a potential
## new neighbor for a new node. We look at these bottom-up, and keep
## inserting nodes until the element is gone.
for v, flag in visited:
if not flag:
# We just went right at this step, so don't do anything.
continue
# Otherwise, we dropped down, so we need to see if we insert a new
# level of the inserted element
if self.stop(current_hash_value):
stopped = True
break
current_hash_value = sha256(current_hash_value).digest()
if current is None:
elem_leaf = self.nodeclass.newnode(self, None, None,
[elem] + left_neighbor.elem[index:])
assert all([y >= elem for y in elem_leaf.elem])
del(left_neighbor.elem[index:])
left_neighbor.elem.append(elem)
left_neighbor.assign_label()
next_level = self.nodeclass.newnode(self, elem_leaf, v.right, elem)
new_elems.append(next_level)
current = next_level
left_neighbor.right = None
v.right = next_level
left_neighbor = v
else:
## Defer labeling until later, in case we're inserting it at the
## current top level.
next_level = self.nodeclass.newnode(self, current, v.right, elem)
new_elems.append(next_level)
left_neighbor.right = None
left_neighbor = v
v.right = next_level
current = next_level
new_roots = []
if not stopped:
# If we got here we need to add another level of just -infty, +infty
self.root.right.right = None
left_neighbor = self.root
new_rght = '0'
new_root = self.nodeclass.newnode(
self, self.root, new_rght, self.root.elem)
self.root = new_root
new_roots.append(new_root)
# At this point, we can safely assign labels to all the new nodes
# created up until now.
elem_leaf.assign_label()
for n in new_elems:
n.assign_label()
while not self.stop(current_hash_value):
current_hash_value = sha256(current_hash_value).digest()
# Add higher levels
old_right = self.root.right
old_left = self.root
if isinstance(old_right, VONode):
new_right = self.nodeclass.newnode(
self, None, None, old_right.elem, True)
else:
new_right = old_right
old_right = None
new_left = self.nodeclass.newnode(
self, old_left, new_right, old_left.elem)
new_roots.append(new_left)
# It's safe to label this one because the new level has
# already been inserted
new_level = self.nodeclass.newnode(self, current, None, elem, True)
old_left.right = new_level
self.root = new_left
left_neighbor.right = None
left_neighbor = old_left
current = new_level
else:
if current is None:
assert not new_elems
left_neighbor.elem.insert(index, elem)
left_neighbor.assign_label()
else:
# Need to update the nodes visited up until now
elem_leaf.assign_label()
for n in new_elems:
n.assign_label()
# Now we have to update all the visited nodes' labels
modified = [v for v, _ in visited] + new_roots
for v, f in visited:
v.assign_label()
for v in new_roots:
v.assign_label()
return
def build_node(self, skiplist_node):
""" Builds a VO node for a skip list within a range.
Returns:
One value, depending on what it found:
If the node is on a path from something needed in the range query
to the root, return the VONode of that node.
If not, return the hash value of that node.
"""
# two cases: base level & not-base level
#
# base level: if nothing between here & the next tower node (not
# including the next tower node) is in range, return
# nothing; otherwise return those elements
#
# not-base level: two more cases
# right-hand neighbor is a tower node: just make a new node & figure
# out what to put under it
# right-hand neighbor is a plateau node: two *more* cases
# if it's not in range: put its label there
# if it is in range: recur!
#
# TODO: how do we list the elements returned, exactly? was thinking a
# list of lists, but actually that might not be necessary if we
# just always check for being out-of-range and keep track of the
# last element we saw in the last 'branch' of the VO during
# verification
# Potential answer: don't list them at all!
# TODO: can probably "unroll" at least some of this recursion into loops
lower = self.lbound
upper = self.rbound
if skiplist_node.down is None:
# we're in the base level
# need to find out if we're including this branch by going to the
# rightmost non-tower element & seeing if it's >= the
# lower bound of the range.
# If it is, we include each element of the base level here.
#
# For now, we achieve this inclusion by returning a list of all
# the base elements in this branch, up to and including the next
# tower node.
current = skiplist_node
elems = [current.elem]
# current.right should exist---if it fails because
# of that, it's a bug & should be investigated!
assert current.right
while not current.right.tower:
current = current.right
elems.append(current.elem)
if current.right.elem >= lower:
elems.append(current.right.elem)
leaf = VONode.newnode(self, None, None, elems)
return leaf
else:
return skiplist_node.label
else:
# We're not in the base level. Need to check if the element to
# the right is a plateau node. If it's not, we just make a node here
# and continue.
if skiplist_node.right.tower:
# TODO: this is the recursion we can probably eliminate
lower_node = self.build_node(skiplist_node.down)
if isinstance(lower_node, VONode):
current_node = VONode.newnode(self, lower_node, None,
skiplist_node.elem)
return current_node
else:
return lower_node
else:
# Othwerwise, the thing on the right is a plateau node, so we
# need to take it into account when constructing this branch
# of the path.
lower_node = self.build_node(skiplist_node.down)
if skiplist_node.right.elem <= upper:
right_node = self.build_node(skiplist_node.right)
else:
right_node = skiplist_node.right.label
if not (isinstance(lower_node, VONode) or
isinstance(right_node, VONode)):
return VONode.chash(lower_node, right_node)
else:
current_node = VONode.newnode(self, lower_node,
right_node,
skiplist_node.elem)
return current_node
def verify(self, lbound, rbound, root_label):
""" Verifies that the range query is in fact a valid range query over
the expected bounds and on a skiplist with the given root label.
"""
if lbound != self.lbound or rbound != self.rbound:
raise VerificationObjectException(
'Malformed VO: unexpected bound(s)')
elems = self.root.verify(self.lbound, self.rbound)
if self.root.label != root_label:
raise VerificationObjectException(
'Malformed VO: root hash does not match expected root hash\nreceived: %s\nexpected: %s' %(self.root.label, root_label))
# If the root labels matched, then the elements in the list must all
# be the right elements in the hash tree, in order, so make sure they
# include the necessary boundaries. Note that this cannot possibly
# succeed if there are fewer than 2 elements returned.
leftmost = bisect.bisect_left(elems, lbound)
if not (leftmost > 0):
raise VerificationObjectException(
'Malformed VO: left boundary for range query not included in VO')
rightmost = bisect.bisect_right(elems, rbound)
if not (rightmost < len(elems)):
raise VerificationObjectException(
'Malformed VO: right boundary for range query not included in VO')
for i in range(len(elems)-1):
if elems[i] <= self.lbound and elems[i+1] > self.lbound:
if elems[i] == self.lbound:
leftmost = i
else:
leftmost = i + 1
for j in range(i+1, len(elems)):
if elems[j-1] < self.rbound:
if elems[j] == self.rbound:
# If it gets here, it found both a left and a right
# bound
return elems[leftmost:j+1]
elif elems[j] > self.rbound:
return elems[leftmost:j]
# ...and if it gets here, it didn't
raise VerificationObjectException(
'Malformed VO: boundaries for range query not included in VO')
def to_list_of_lists(self):
""" Attempts to turn the current object into a list of lists, similar
to SkipList.to_list_of_lists(). Due to the structure of skiplist
VOs, this is mostly only useful for rudimentary debugging.
"""
current = self.root
rows = []
while isinstance(current, VONode):
row = []
crow = current
while isinstance(crow, VONode):
row.append(crow.elem)
crow = crow.right
if crow:
row.append(crow)
rows.append(row)
current = current.down
if current:
rows.append(current)
return rows
pace/ads/skiplist/skiplistvo_test.py
+492
View File
@@ -0,0 +1,492 @@
## **************
## Copyright 2014 MIT Lincoln Laboratory
## Project: PACE
## Authors: ZS
## Description: Unit tests for verification objects
## Modifications:
## Date Name Modification
## ---- ---- ------------
## 21 Jul 2014 ZS Original file
## **************
import os
import sys
this_dir = os.path.dirname(os.path.abspath(__file__))
base_dir = os.path.join(this_dir, '../../..')
sys.path.append(base_dir)
from unittest import TestCase
from hashlib import sha256
import time
import random
import bisect
from pace.ads.skiplist.authskiplist import AuthSkipList
from pace.ads.skiplist.skiplistvo import SkipListVO
from pace.ads.skiplist.vonode import VONode, VerificationObjectException
from pace.ads.skiplist.coin import RecordedCoin, PrefixCoin, SeededCoin
from pace.ads.skiplist.elemclass import IntElem
from pace.common.pacetest import PACETestCase
class SkipListVOTest(PACETestCase):
def setUp(self):
random.seed(int(time.time()))
# Setup for small, hand-made tests
self.elems = map(IntElem, [0, 5, 6, 10])
self.sl = AuthSkipList.new(self.elems,
IntElem(self.elems[0].key-1),
IntElem(self.elems[-1].key+1))
self.lower = IntElem(1)
self.upper = IntElem(9)
self.vo = SkipListVO.range_query(self.sl, self.lower, self.upper)
def test_range(self):
""" Ensure that range queries work correctly
"""
for i in range(0, self.num_iters):
elems = []
# hacked up to avoid repeated elements in elems
i = 0
while i < 5:
x = IntElem(random.randint(-50, 50))
if x not in elems:
i = i + 1
elems.append(x)
sl = AuthSkipList.new(elems, IntElem(-101), IntElem(101))
lower = random.choice(sorted(elems)[0:2])
upper = random.choice(sorted(elems)[3:5])
vo = SkipListVO.range_query(sl, lower, upper)
try:
elems = vo.verify(lower, upper, sl.root.label)
except VerificationObjectException as e:
self.assertTrue(False, 'Error: %s' %e.msg)
self.assertEqual(elems,
[e for e in sl.to_list_of_lists()[-1]
if lower <= e <= upper])
def test_big_range_query(self):
""" Check that range queries work
"""
lower_bound = -100000
upper_bound = 100000
num_elems = 1000
for i in range(0, self.num_iters):
elems = [random.randint(lower_bound, upper_bound)
for i in range(num_elems)]
elems = map(IntElem, set(elems))
sl = AuthSkipList.new(elems, IntElem(lower_bound-1), IntElem(upper_bound+1))
for j in range(0, self.num_iters):
lower = IntElem(random.randint(lower_bound/5, upper_bound/100))
upper = IntElem(random.randint(lower.key + 1000,
upper_bound/5))
vo = SkipListVO.range_query(sl, lower, upper)
try:
elems = vo.verify(lower, upper, sl.root.label)
except VerificationObjectException as e:
self.assertTrue(False, 'Error: %s' %e.msg)
expected = [e for e in sl.to_list_of_lists()[-1]
if lower <= e <= upper]
if len(elems) != len(expected):
print 'num returned elems:', len(elems)
print 'num expected elems:', len(expected)
for x, y in zip(elems, expected):
print x,
print '---',
print y
self.assertEqual(elems, expected)
def test_empty_vo_fails(self):
elems = self.elems
sl = self.sl
lower = self.lower
upper = self.upper
vo = self.vo
# Make sure it won't verify an empty VO
bad_vo = SkipListVO(IntElem(0),
IntElem(10),
VONode(None, sl.root.down.label, sl.root.right.label, -1),
None)
passed = False
try:
bad_vo.verify(lower, upper, sl.root.label)
except VerificationObjectException as e:
passed = True
self.assertTrue(passed, 'Should not verify an empty VO')
def test_bogus_root_hash(self):
elems = self.elems
sl = self.sl
lower = self.lower
upper = self.upper
vo = self.vo
# Make sure it won't verify a bogus root hash
bad_root_hval = sha256(b'foobar').digest()
passed = False
try:
vo.verify(lower, upper, bad_root_hval)
except VerificationObjectException as e:
passed = True
self.assertTrue(passed, 'Should not verify a bogus root hash')
def test_too_long_range(self):
elems = self.elems
sl = self.sl
lower = self.lower
upper = self.upper
vo = self.vo
# Make sure it won't let us return too many elements
passed = False
try:
vo.verify(IntElem(1), IntElem(5), sl.root.label)
except VerificationObjectException as e:
passed = True
self.assertTrue(passed, 'Should not verify an overextended range')
def test_stringify(self):
for i in range(0, self.num_iters):
elems = [IntElem(random.randint(0, 100000000))
for i in range(0, 1000)]
elems.sort()
sl = AuthSkipList.new(elems, IntElem(-1), IntElem(100000001))
lower = random.choice(elems[1:200])
upper = random.choice(elems[500:-1])
vo = SkipListVO.range_query(sl, lower, upper)
self.assertTrue(isinstance(vo, SkipListVO))
serialized = vo.serialize()
new_vo = SkipListVO.deserialize(serialized, IntElem)
print 'vo:'
for l in vo.to_list_of_lists():
print l
print
print new_vo.root
print 'new_vo:'
for l in new_vo.to_list_of_lists():
print l
self.assertEqual(vo, new_vo)
def test_vo_insert(self):
seed = str(time.time())
random.seed(seed)
for i in range(0, self.num_iters):
elems = [IntElem(random.randint(0, 100000000))
for i in range(0, self.size)]
num_elems = self.size
elems.sort()
for i in range(len(elems)-1):
if elems[i] == elems[i+1]:
elems[i+1] = elems[i+1] + 1
orig_elems = elems
min_elem = IntElem(-1)
max_elem = IntElem(100000001)
sl = AuthSkipList.new(elems, min_elem, max_elem, RecordedCoin())
left = IntElem(random.randint(5, 6000))
right = IntElem(random.randint(left.key+100, 9000))
vo = SkipListVO.range_query(
sl, left, right, PrefixCoin([]))
try:
elems = vo.verify(left, right, sl.root.label)
except VerificationObjectException as e:
self.assertTrue(False, 'Error: %s' %e.msg)
self.assertEqual(elems, [e for e in orig_elems if left<=e<=right])
elem = IntElem(random.randint(left.key+1, right.key-1))
while elem in orig_elems:
elem = IntElem(random.randint(left.key+1, right.key-1))
bisect.insort(orig_elems, elem)
sl.coin.read()
sl.insert(elem)
vo.coin.extend(sl.coin.read())
vo.insert(elem)
try:
elems = vo.verify(left, right, sl.root.label)
except VerificationObjectException as e:
self.assertTrue(False, 'Error: %s' %e.msg)
self.assertEqual(sl.root.label, vo.root.label)
self.assertEqual(elems, [e for e in orig_elems if left<=e<=right])
def test_vo_insert_many_seeded(self):
seed = str(time.time())
random.seed(seed)
for i in range(0, self.num_iters):
seed = random.randint(0, 1000000)
elems = [IntElem(random.randint(0, 100000000))
for i in range(0, self.size)]
num_elems = self.size
elems.sort()
for i in range(len(elems)-1):
if elems[i] == elems[i+1]:
elems[i+1] = elems[i+1] + 1
orig_elems = elems
min_elem = IntElem(-1)
max_elem = IntElem(100000001)
sl = AuthSkipList.new(elems, min_elem, max_elem)
left = IntElem(random.randint(5, 6000000))
right = IntElem(random.randint(left.key+100, 9000000))
vo = SkipListVO.range_query(sl, left, right)
try:
elems = vo.verify(left, right, sl.root.label)
except VerificationObjectException as e:
self.assertTrue(False, 'Error: %s' %e.msg)
self.assertEqual(elems, [e for e in orig_elems if left<=e<=right])
new_elems = [IntElem(random.randint(left.key+1, right.key-1))
for i in range(20)]
sl.coin = SeededCoin(seed)
vo.coin = SeededCoin(seed)
for elem in new_elems:
sl.insert(elem)
for elem in new_elems:
vo.insert(elem)
try:
elems = vo.verify(left, right, sl.root.label)
except VerificationObjectException as e:
self.assertTrue(False, 'Error: %s' %e.msg)
self.assertEqual(sl.root.label, vo.root.label)
def test_vo_insert_many(self):
seed = str(time.time())
random.seed(seed)
for i in range(0, self.num_iters):
elems = [IntElem(random.randint(0, 100000000))
for i in range(0, self.size)]
num_elems = self.size
elems.sort()
for i in range(len(elems)-1):
if elems[i] == elems[i+1]:
elems[i+1] = elems[i+1] + 1
orig_elems = elems
min_elem = IntElem(-1)
max_elem = IntElem(100000001)
sl = AuthSkipList.new(elems, min_elem, max_elem, RecordedCoin())
left = IntElem(random.randint(5, 6000000))
right = IntElem(random.randint(left.key+100, 9000000))
vo = SkipListVO.range_query(
sl, left, right, PrefixCoin([]))
try:
elems = vo.verify(left, right, sl.root.label)
except VerificationObjectException as e:
self.assertTrue(False, 'Error: %s' %e.msg)
self.assertEqual(elems, [e for e in orig_elems if left<=e<=right])
new_elems = [IntElem(random.randint(left.key+1, right.key-1))
for i in range(20)]
sl.coin.read()
for elem in new_elems:
sl.insert(elem)
vo.coin.extend(sl.coin.read())
for elem in new_elems:
vo.insert(elem)
try:
elems = vo.verify(left, right, sl.root.label)
except VerificationObjectException as e:
self.assertTrue(False, 'Error: %s' %e.msg)
self.assertEqual(sl.root.label, vo.root.label)
def test_vo_insert_commutes(self):
seed = str(time.time())
random.seed(seed)
for i in range(0, self.num_iters):
elems = [IntElem(random.randint(0, 100000000))
for i in range(0, self.size)]
num_elems = self.size
elems.sort()
for i in range(len(elems)-1):
if elems[i] == elems[i+1]:
elems[i+1] = elems[i+1] + 1
orig_elems = elems
min_elem = IntElem(-1)
max_elem = IntElem(100000001)
sl = AuthSkipList.new(elems, min_elem, max_elem, RecordedCoin())
left = IntElem(random.randint(5, 6000))
right = IntElem(random.randint(left.key+100, 9000))
vo = SkipListVO.range_query(
sl, left, right, PrefixCoin([]))
try:
elems = vo.verify(left, right, sl.root.label)
except VerificationObjectException as e:
self.assertTrue(False, 'Error: %s' %e.msg)
elem = IntElem(random.randint(left.key+1, right.key-1))
while elem in orig_elems:
elem = IntElem(random.randint(left.key+1, right.key-1))
bisect.insort(orig_elems, elem)
sl.coin.read()
sl.insert(elem)
new_vo = SkipListVO.range_query(sl, left, right, PrefixCoin([]))
vo.coin.extend(sl.coin.read())
vo.insert(elem)
try:
elems = vo.verify(left, right, sl.root.label)
except VerificationObjectException as e:
self.assertTrue(False, 'Error: %s' %e.msg)
self.assertEqual(sl.root.label, vo.root.label)
self.assertEqual(elems, [e for e in orig_elems if left<=e<=right])
try:
elems = new_vo.verify(left, right, sl.root.label)
except VerificationObjectException as e:
self.assertTrue(False, 'Error: %s' %e.msg)
self.assertEqual(sl.root.label, vo.root.label)
self.assertEqual(elems, [e for e in orig_elems if left<=e<=right])
self.assertEqual(vo, new_vo)
def test_different_single_insert(self):
for i in range(0, self.num_iters):
print
print i
orig_seed = random.randint(0, 10000000)
elems = [IntElem(random.randint(0, 100)) for i in range(0, 5)]
elems.sort()
sl = AuthSkipList.new(elems, IntElem(elems[0].key-1),
IntElem(elems[-1].key+1),
SeededCoin(orig_seed))
new_elem = IntElem(random.randint(elems[1].key, elems[-2].key))
min_ne = elems[1]
max_ne = elems[-2]
vo = SkipListVO.range_query(sl, min_ne, max_ne)
try:
ret_elems = vo.verify(min_ne, max_ne, sl.root.label)
except VerificationObjectException as e:
self.assertTrue(False, 'Error: %s' %e.msg)
seed = random.randint(0, 1000000)
vo.coin = SeededCoin(seed)
sl.coin = SeededCoin(seed)
self.assertEqual(vo.root.label, sl.root.label)
vo.insert(new_elem)
sl.insert(new_elem)
self.assertEqual(sl.root.label, vo.root.label)
def test_multiple_inserts(self):
for i in range(0, self.num_iters):
orig_seed = random.randint(0, 10000000)
elems = [IntElem(random.randint(0, 100000000))
for i in range(0, 100)]
elems.sort()
sl = AuthSkipList.new(elems, IntElem(elems[0].key-1),
IntElem(elems[-1].key+1),
SeededCoin(orig_seed))
new_elems = [IntElem(random.randint(elems[1].key, elems[-2].key))
for i in range(0, 10)]
min_ne = min(new_elems)
max_ne = max(new_elems)
vo = SkipListVO.range_query(sl, min_ne, max_ne)
try:
ret_elems = vo.verify(min_ne, max_ne, sl.root.label)
except VerificationObjectException as e:
self.assertTrue(False, 'Error: %s' %e.msg)
seed = random.randint(0, 1000000)
vo.coin = SeededCoin(seed)
sl.coin = SeededCoin(seed)
self.assertEqual(vo.root.label, sl.root.label)
for elem in new_elems:
vo.insert(elem)
sl.insert(elem)
self.assertEqual(sl.root.label, vo.root.label, 'failed on index %d' %new_elems.index(elem))
self.assertEqual(sl.root.label, vo.root.label)
pace/ads/skiplist/vonode.py
+345
View File
@@ -0,0 +1,345 @@
## **************
## Copyright 2014 MIT Lincoln Laboratory
## Project: PACE
## Authors: ZS
## Description: Node class for VOs
## Modifications:
## Date Name Modification
## ---- ---- ------------
## 28 Jul 2014 ZS Original file
## **************
from base64 import b64encode, b64decode
from collections import deque
import bisect
from pace.ads.skiplist.authnode import AuthNode
class VONode(AuthNode):
def __eq__(self, other):
if not isinstance(other, VONode):
return False
if self.right != other.right:
return False
if self.down != other.down:
return False
if self.elem != other.elem:
return False
return True
def __ne__(self, other):
return not self.__eq__(other)
def search(self, elem):
""" Search the SkipList for node containing an element that is
either equal to elem (if elem is in the list) or the closest
element in the list that is less than elem (if elem is not in the
list)
Argument:
elem - the element to be searched for
Returns:
visited - a collection of nodes visited, stored in reverse order
current - the closest node in the base list that is less than or
equal to elem
"""
current = self
visited = deque([])
while (isinstance(current.right, type(self)) and
current.right.elem <= elem):
visited.appendleft((current, False))
current = current.right
while current.down:
visited.appendleft((current, True))
current = current.down
while (isinstance(current.right, type(self)) and
current.right.elem <= elem):
visited.appendleft((current, False))
current = current.right
return visited, current, bisect.bisect(current.elem, elem)
def serialize(self):
""" Return the current VO node as a string that can be sent over the
network and reconstructed on the other side.
Should obey the property that VONode.deserialize(n.serialize) == n
and (VONode.deserialize(s)).serialize == s (for all n and s)
"""
if self.down is None:
s_down = '{}'
elif not isinstance(self.down, VONode):
s_down = b64encode(self.down)
else:
s_down = self.down.serialize()
if self.right is None:
s_right = '{}'
elif not isinstance(self.right, VONode):
s_right = b64encode(self.right)
else:
s_right = self.right.serialize()
if isinstance(self.elem, list):
ser_elem = '[%s]' %('~'.join([elem.serialize() for elem in self.elem]))
else:
ser_elem = self.elem.serialize()
serialization = '(%s)' %(','.join([str(self.tower),
s_down,
s_right,
ser_elem]))
return serialization
@staticmethod
def deserialize(vo, node, elemClass):
""" Return the current VO node as a string that can be sent over the
network and reconstructed on the other side.
Should obey the property that
VONode.deserialize(n.sl, n.serialize) == n
and
(VONode.deserialize(sl, s)).serialize == s (for all n and s)
"""
here, _ = VONode.deserialize_helper(vo, node, 0, elemClass)
return here
@staticmethod
def deserialize_helper(vo, node, k, elemClass):
# TODO: remove the recursion (probably by making dummy nodes &
# backpatching them in with a loop?)
if node[k:k+2] == '{}':
return None, k+2
elif node[k] != '(':
# Must be a base64 encoding, so look for the ',' that delimits it
i = node.find(',', k)
return b64decode(node[k:i]), i
# Otherwise, it's a full node
i = k+1
if node[i:i+4] == 'True':
tower = True
i = i + 4
elif node[i:i+5] == 'False':
tower = False
i = i + 5
else:
raise VerificationObjectException(
'first element of serialized node must be True or False')
if node[i] != ',':
raise VerificationObjectException(
'commas must separate serialized node elements (1 & 2)')
i = i + 1
down, i = VONode.deserialize_helper(vo, node, i, elemClass)
if node[i] != ',':
raise VerificationObjectException(
'commas must separate serialized node elements (2 & 3)\nfound "%s" instead' %node[i])
i = i + 1
right, i = VONode.deserialize_helper(vo, node, i, elemClass)
if node[i] != ',':
raise VerificationObjectException(
'commas must separate serialized node elements (3 & 4)\nfound "%s" instead' %node[i])
i = i + 1
if node[i] == '[':
# it's a leaf list
i = i + 1
j = node.find('])', i)
elem = [elemClass.deserialize(x) for x in node[i:j].split('~')]
i = j+2
else:
j = node.find(')', i)
elem = elemClass.deserialize(node[i:j])
i = j+1
here = VONode(vo, down, right, elem)
here.tower = tower
return here, i
@staticmethod
def newnode(vo, down, right, elem, assign_label=False):
if isinstance(down, str) and isinstance(right, str):
return AuthNode.chash(down, right)
elif isinstance(down, str) and right is None:
return down
elif isinstance(right, str) and down is None:
return right
elif down is None and right is None:
return super(VONode, VONode).newnode(
vo, None, None, elem, assign_label)
if not isinstance(down, VONode):
node = super(VONode, VONode).newnode(
vo, None, right, elem, assign_label)
node.down = down
else:
node = super(VONode, VONode).newnode(
vo, down, right, elem, assign_label)
down.tower = True
return node
def verify(self, lower, upper):
""" Verify that a node (including its subtree) is correct, recursively
assigning labels along the way
"""
# TODO: recursion won't work for nontrivial cases in python;
# rewrite iteratively at some point
node = self
if node.down is None:
if node.right is not None:
raise VerificationObjectException(
'Malformed VO: base VO nodes have no children')
if len(node.elem) < 2:
raise VerificationObjectException(
'Malformed VO: base element list must have at least 2 elements')
for i in range(len(node.elem)-1):
if node.elem[i] > node.elem[i+1]:
raise VerificationObjectException(
'Malformed VO: elements in base must be in order')
if node.elem[-1] < lower:
raise VerificationObjectException(
'Malformed VO: element out of range (less than minimum)')
# Don't want to check if right end is greater than maximum---might
# have returned a base list whose first element is in the range,
# but whose next element is the right boundary element
if node.elem[0] > upper:
raise VerificationObjectException(
'Malformed VO: element out of range (greater than maximum)')
node.label = reduce(AuthNode.chash,
[AuthNode._hash(x.serialize())
for x in reversed(node.elem)])
return node.elem
# If it's not a base element (and therefore has a lower neighbor)
else:
if node.right is None:
if isinstance(node.down, VONode):
if node.down.down and node.elem != node.down.elem:
raise VerificationObjectException(
'Malformed VO: element of lower node must be same as parent')
elif (node.down.down is None and
node.elem != node.down.elem[0]):
raise VerificationObjectException(
'Malformed VO: first element of leaf list must be same as parent')
down_elems = node.down.verify(lower, upper)
node.label = node.down.label
return down_elems
else:
raise VerificationObjectException(
'Malformed VO: Branch not collapsed when it should have been (right neighbor None)')
else:
if not (isinstance(node.down, VONode) or
isinstance(node.right, VONode)):
raise VerificationObjectException(
'Malformed VO: Branch not collapsed when it should have been (neither neighbor VOnode)')
down_elems = []
right_elems = []
down_label = node.down
right_label = node.right
if isinstance(node.down, VONode):
if node.down.down and node.elem != node.down.elem:
raise VerificationObjectException(
'Malformed VO: element of lower node must be same as parent')
elif (node.down.down is None and
node.elem != node.down.elem[0]):
raise VerificationObjectException(
'Malformed VO: element of lower node must be same as parent')
down_elems = node.down.verify(lower, upper)
down_label = node.down.label
elif lower <= node.elem <= upper:
# This branch definitely should have been returned!
# This check keeps a malicious server from omitting
# branches in the middle of a range.
raise VerificationObjectException(
'Malformed VO: branch omitted that should not have been')
if isinstance(node.right, VONode):
if node.right.elem < node.elem:
raise VerificationObjectException(
'Malformed VO: elements on right must be >= elements on left')
right_elems = node.right.verify(lower, upper)
right_label = node.right.label
# Cut off the overlap, if it exists
if down_elems:
down_elems = down_elems[:-1]
node.label = AuthNode.chash(down_label, right_label)
return down_elems + right_elems
def assign_label(self):
""" VOs have a different structure than skiplists, so their label
assignment algorithm is completely different. If a node has a
right neighbor, then that node must have been a plateau node; if
not, then its right neighbor must have been a tower node.
"""
if self.down is None:
self.label = reduce(AuthNode.chash,
[AuthNode._hash(x.serialize())
for x in reversed(self.elem)])
return
if isinstance(self.down, VONode):
down_hash = self.down.label
else:
down_hash = self.down
if self.right is None:
# Just assign the bottom hash, since the right neighbor was a
# tower node
self.label = down_hash
return
if isinstance(self.right, VONode):
right_hash = self.right.label
else:
right_hash = self.right
self.label = AuthNode.chash(down_hash, right_hash)
return
class VerificationObjectException(Exception):
def __init__(self, msg):
self.msg = msg
pace/common/__init__.py
View File
Whitespace-only changes.
pace/common/common_utils.py
+202
View File
@@ -0,0 +1,202 @@
## **************
## Copyright 2015 MIT Lincoln Laboratory
## Project: PACE
## Authors: ATLH, SS
## Description: Common utility functions for testing the code
## Modifications:
## Date Name Modification
## ---- ---- ------------
## 20 Jan 2015 ATLH Original file - based on sign_utils
## 15 Dec 2015 SS Added functionality of data generation
## **************
""" Utility functions for testing PACE software with rows written to
Accumulo.
Any randomized test should use the same seed so that they are equivalent.
"""
import os
import sys
this_dir = os.path.dirname(os.path.dirname(__file__))
base_dir = os.path.join(this_dir, '../..')
sys.path.append(base_dir)
import random
import copy
import time, datetime
import string
from pyaccumulo import Range
class Timer(object):
"""
Timer that works with python's 'with' syntax
measures the time taken in seconds and milliseconds
"""
#modified from http://www.huyng.com/posts/python-performance-analysis/
def __enter__(self):
self.start = time.clock()
return self
def __exit__(self, *args):
self.end = time.clock()
self.secs = self.end - self.start
self.msecs = self.secs * 1000 # millisecs
def generate_data(filename,
seed,
vis=False,
default_vis="default",
vis_in_value=False,
num_entries=50000,
num_rows=1000,
vis_length=9,
row_length=9):
""" Generate a tab-delimited text file with five columns:
row, col fam, col qual, col vis, value
keyword arguments:
filename - name of file to which the data will be written
seed - a string to be used as the seed for the random number generator
visibility - if True, random column visibilities are added. If False,
the default visibility string is added. (default False)
default_vis - the default visibility string to use if 'visibiilty' is
set to False. (default "default")
vis_in_value - if True, a random "column visibility" is generated, but
this is appended to the value rather than column visibility. If
True, the param vis is treated as False.
num_entries - Number of entries written.
num_rows - Number of rows in the data written.
vis_length - The length (in chars) of the random col visibility added
(true length will be this + 2)
row_length - Number of chars in each row ID. Numbers are zero padded to
reach this length.
"""
# seed RNG
random.seed(seed)
rows = map(lambda x: str(x).zfill(row_length), \
random.sample(xrange(10 ** row_length), num_rows))
unused_rows = copy.deepcopy(rows)
def generate_entry():
""" Helper function for generate_data that returns a single entry
which will form a single line of the data file
"""
def generate_field(length=row_length):
""" Generates a single field (row, col fam, or col qual) randomly
"""
return ''.join(random.choice(string.ascii_uppercase + string.digits) for _ in range(length))
def generate_row():
""" Generates a single rowID from the list of rows. On successive
calls, ensures each rowID is used at least once, but makes no
other guarantees about how many times each rowID is used
"""
return unused_rows.pop() if unused_rows else random.choice(rows)
def generate_colvis():
""" Generates a single column visibility (either the default or a
random value)
"""
return default_vis if (vis_in_value or not vis) \
else generate_field(vis_length).zfill(vis_length)
def generate_value():
""" Generates a random value, accounting for vis_in_value """
val = generate_field()
return val + generate_field(vis_length).zfill(vis_length) if vis_in_value else val
return generate_row() + "\t" + \
generate_field() + "\t" + \
generate_field() + "\t" + \
generate_colvis() + "\t" + \
generate_value() + "\n"
# Open file and generate data
with open(filename, "w") as f:
for i in range(num_entries):
f.write(generate_entry())
def sanitize(string):
""" Turns '-' into '_' for accumulo table names
"""
return string.replace('-', '_')
def get_single_entry(conn, table, row=None, cf=None, cq=None):
""" Get the single entry with the given key fields (row, column
family, and column qualifier) from an Accumulo database.
Returns:
None if no entries were found
The found entry if exactly one was found
Raises an assertion error if more than one entry was found
"""
if cf is None:
assert cq is None
cols = None
elif cq is None:
cols = [[cf]]
else:
cols = [[cf, cq]]
candidates = conn.scan(table, Range(srow=row, erow=row), cols=cols)
try:
first = next(candidates)
except StopIteration:
# No candidates; return None
return None
try:
more = next(candidates)
assert False # only supposed to have one result
except StopIteration:
return first
def entry_exists(conn, table, row, cf=None, cq=None):
""" Check to see if at least one entry with the given information
exists in the given Accumulo table. Takes at least a row,
and possibly a column family & qualifier as well.
Arguments:
conn : Accumulo - a connection to the Accumulo instance to check
table : string - the name of the table to check
row : string - the name of the row to look for within the given table
cf : optional string - the column family to look for, if desired
cq : optional string - the column qualifier to look for, if desired
Returns:
bool - whether there is at least one entry in the given connection with
the given fields
Assumptions:
- The given table exists on the connection
- If cf is not specified/None, then neither is cq
"""
if cf is None:
assert cq is None
cols = None
elif cq is None:
cols = [[cf]]
else:
cols = [[cf, cq]]
candidates = conn.scan(table, Range(srow=row, erow=row), cols=cols)
try:
first = next(candidates)
except StopIteration:
# Nothing found
return False
else:
# Something found
return True
pace/common/fakeconn.py
+186
View File
@@ -0,0 +1,186 @@
## **************
## Copyright 2014 MIT Lincoln Laboratory
## Project: PACE
## Authors: CS
## Description: Code to simulate an Accumulo connection, for testing
## Modifications:
## Date Name Modification
## ---- ---- ------------
## 25 Nov 2014 CS Original file
## 7 Jan 2015 ATLH Changed FakeEntry to Cell
## **************
from collections import namedtuple
Cell = namedtuple("Cell", "row cf cq cv ts val")
class FakeWriter(object):
def __init__(self, table, conn):
self.table = table
self.conn = conn
self.mutations = []
def close(self):
for m in self.mutations:
self.conn.write(self.table, m)
self.mutations = []
def add_mutation(self, m):
self.mutations.append(m)
class FakeConnection(object):
""" Something that acts like an Accumulo connection, for testing.
"""
def __init__(self):
self.db = {}
def write(self, table, mut):
try:
tdict = self.db[table]
except KeyError:
raise Exception('Table %s not in database' %table)
row = mut.row
try:
rdict = tdict[row]
except KeyError:
rdict = {}
tdict[row] = rdict
for u in mut.updates:
col = u.colFamily, u.colQualifier, u.colVisibility, u.timestamp
if u.deleteCell:
del rdict[col]
else:
rdict[col] = u.value
def table_exists(self, table):
return table in self.db
def create_table(self, table):
if table not in self.db:
self.db[table] = {}
else:
raise Exception('Table %s already in database' %table)
#TODO: table deletion
def aggregate_db(self):
out = {}
for table_name, table_contents in self.db.iteritems():
tmp = [(row, rdb.items())
for row, rdb in table_contents.iteritems()]
for row, relems in tmp:
contents = [Cell(row, cf, cq, cv, ts, val)
for (cf, cq, cv, ts), val in relems]
out[table_name] = contents
return out
def create_batch_writer(self, table):
return FakeWriter(table, self)
@staticmethod
def in_range(row, cf, cq, cv, scanrange):
if scanrange.srow is not None and scanrange.erow is not None:
if not scanrange.srow <= str(row) <= scanrange.erow:
return False
if scanrange.scf is not None and scanrange.ecf is not None:
if not scanrange.scf <= str(cf) <= scanrange.ecf:
return False
if scanrange.scq is not None and scanrange.ecq is not None:
if not scanrange.scq <= str(cq) <= scanrange.ecq:
return False
if scanrange.scv is not None and scanrange.ecv is not None:
if not scanrange.scv <= str(cv) <= scanrange.ecv:
return False
return True
@staticmethod
def _matches_cols(cf, cq, cols):
""" See if the given column family & qualifier are in the
given set of columns. Written to match pyaccumulo
functionality as closely as possible.
"""
if not cols:
# Nothing specified (either None or []), so just return True
return True
for col in cols:
assert len(col) > 0
if col[0] == cf:
if len(col) > 1:
if col[1] == cq:
return True
else:
continue
else:
return True
# No matches found
return False
def scan(self, table, scanrange=None, cols=None):
""" Scan a table, either in its entirety or over a given range.
Arguments:
table : string - the name of the table to scan
scanrange : pyaccumulo.Range - the range of rows to scan.
It is important to note that, while the pyaccumulo Range
object can contain start & end column fields, as well, it
seems to ignore those, so we also ignore them.
cols : optional [[string]] - a list of column family/qualifier
lists. This does not seem to be formally documented in
pyaccumulo, but seems to work as follows:
- If it is None or [], return all entries in `table`
matching `range`
- If it is nonempty, treat each element as a specification
of which column to return. Each element should be
a nonempty list, where the first element is treated
as the column family and the second element, if any,
is treated as the column qualifier, further refining
its paired column family.
For example, the `cols` value [['foo', 'bar']] would
match a cell with the column family 'foo' and column
qualifier 'bar', but not one with column family 'foo'
and column qualifier 'baz', nor would it match one
with column family 'qux' and column qualifier 'bar'.
The `cols` value [['foo', 'bar'], ['baz']], however,
would match a cell with column family 'foo' and
column qualifier 'bar', as well as a cell with column
family 'baz' and any column qualifier.
"""
if scanrange is not None:
for row, rdb in self.db[table].iteritems():
for (cf, cq, cv, ts), val in rdb.iteritems():
if FakeConnection.in_range(row, cf, cq, cv, scanrange):
if self._matches_cols(cf, cq, cols):
yield Cell(row, cf, cq, cv, ts, val)
else:
for row, rdb in self.db[table].iteritems():
for (cf, cq, cv, ts), val in rdb.iteritems():
if self._matches_cols(cf, cq, cols):
yield Cell(row, cf, cq, cv, ts, val)
def _scan(self, table, scanrange=None):
""" Helper function for scan() that returns a generator
NB: deprecated, since scan() is supposed to return a generator
after all
"""
return self.scan(table, scanrange)
def list_tables(self):
return self.db.keys()
pace/common/pacetest.py
+36
View File
@@ -0,0 +1,36 @@
## **************
## Copyright 2015 MIT Lincoln Laboratory
## Project: PACE
## Authors: CS
## Description: test case class for PACE project tests
## Modifications:
## Date Name Modification
## ---- ---- ------------
## 13 Jan 2015 CS Original file
## **************
import random
from unittest import TestCase
DEFAULT_NUM_ITERS = 5
DEFAULT_SIZE = 100
class PACETestCase(TestCase):
""" Subclass of TestCase with some convenient constants built-in.
"""
def __init__(self, *args):
super(PACETestCase, self).__init__(*args)
self.num_iters = DEFAULT_NUM_ITERS
self.size = DEFAULT_SIZE
def generate_elems(self, min_elem=0, max_elem=1000000000, size=None):
""" Generate up to size unique integers between
min_elem and max_elem.
"""
if size is None:
size = self.size
ints = (random.randint(min_elem, max_elem)
for _ in range(size))
return set(ints)
pace/encryption/.demo_commands.txt.swp
BIN
View File
Binary file not shown.
pace/encryption/.enc_mutation_test.py.swp
BIN
View File
Binary file not shown.
pace/encryption/.encrypt_test.py.swp
BIN
View File
Binary file not shown.
pace/encryption/.encryption_pki.py.swp
BIN
View File
Binary file not shown.
pace/encryption/AES_encrypt.py
+461
View File
@@ -0,0 +1,461 @@
## **************
## Copyright 2015 MIT Lincoln Laboratory
## Project: PACE
## Authors: ATLH
## Description: Class for AES encryption modules
## Modifications:
## Date Name Modification
## ---- ---- ------------
## 12 Mar 2015 ATLH Copied from abstract_encrypt.py
## **************
import os
import sys
this_dir = os.path.dirname(os.path.dirname(__file__))
base_dir = os.path.join(this_dir, '../..')
sys.path.append(base_dir)
import struct, fractions
from abc import ABCMeta, abstractmethod
from Crypto.Cipher import AES
from Crypto.Util import Counter, number
from Crypto import Random
from pyaccumulo import Cell
from pace.encryption.enc_mutation import EncMutation, EncCell, EncRange
from pace.encryption.vars import CELL_MUT_MAPPING
from pace.encryption.abstract_encrypt import AbstractEncrypt
from pace.encryption.encryption_exceptions import EncryptionException, DecryptionException
class Pycrypto_AES_Base(AbstractEncrypt):
'''
Wrapper for pycrypto's AES implementation.
Keys must be 16, 24, or 32 bytes long.
'''
@staticmethod
def _encrypt(plaintext, key, iv=None):
""" Encrypt the plaintext with the key. For modes of operation that
require an initialization vector (IV), the IV is generated randomly
inside the method; the IV argument should NOT be supplied except
for testing purposes.
Arguments:
plaintext (byte string) - the plaintext to encrypt
key (byte string) - the AES key
iv (optional byte string) - the initialization vector
Returns the encrypted data as a byte string. For modes of operation
that use an IV, the IV is prepended to the ciphertext.
"""
raise NotImplementedError(
'_encrypt is not implemented')
@staticmethod
def _decrypt(ciphertext, key):
""" Decrypt the ciphertext using the key.
Arguments:
ciphertext (byte string) - the ciphertext to decrypt, with the
IV prepended for modes of operation that use an IV.
key (byte string) - the AES key
Returns the decrypted data as a byte string.
"""
raise NotImplementedError(
'_decrypt is not implemented')
@staticmethod
def _pad(s):
'''
Returns the string padded so that its length is a multiple of 16 bytes.
Appends 10* at the bit level. Following ISO/IEC 9797-1
- padding mode 2
'''
pad_len = AES.block_size - (len(s) % AES.block_size) - 1
padding = chr(0x80)+'\0'*pad_len
return s + padding
@staticmethod
def _strip_pad(s):
'''
Strips the padding from the string
'''
return s.rstrip('\0')[:-1]
@staticmethod
def _is_multiple_16(s):
"""
Ensures string's length is a multple of 16
"""
if not (len(s) % 16) == 0:
raise DecryptionException("Ciphertext was not a multiple of 16 in length")
@staticmethod
def _has_iv_material(s):
"""
Make sure enough material for IV in ciphertext
"""
if len(s) < AES.block_size:
raise DecryptionException("Ciphertext did not contain enough material for an IV")
@staticmethod
def _get_encryption_key(key_container):
'''
Arguments:
key_container : a (keytor) object that contains a key_id
and the key_object handle to look up and obtain the
keys
Returns: (key,version_number) tuple where the key is the most recent encryption
key (byte string) and the version_number is the most current version of the key (int)
Raises PKILookupError if a key for the key_id is not contained in the keyobject.
'''
return key_container.key_object.get_current_key(key_container.key_id)
@staticmethod
def _get_decryption_key(key_container, version):
'''
Arguments:
key_container: a (keytor) object that contains a key_id
and the key_object handle to look up and obtain the
keys
version: (int) version of the decryption key to obtain
Returns: key is the decryption key for that version
Raises PKILookupError if a key for the key_id is not contained in the keyobject.
'''
return key_container.key_object.get_key(key_container.key_id, int(version))
@classmethod
def encrypt_mutation(cls, mutation, key_container, cell_sections):
(key, version) = cls._get_encryption_key(key_container)
ptexts = EncMutation.concatenate_cell_section_values(mutation, cell_sections)
#add version number delineated by 'ver' - version is int so this is fine
ctexts = [cls._encrypt(ptext, key)+'ver'+str(version) for ptext in ptexts]
return ctexts
@classmethod
def decrypt_mutation(cls, mutation, dec_mutation, key_container, cell_location, cell_sections):
ptexts = []
ctexts = mutation[cell_location]
for ctext in ctexts:
#grab the version, delineated by last instance of 'ver'
try:
(ctext, version) = ctext.rsplit('ver',1)
except ValueError:
raise DecryptionException('Ciphertext is not properly formatted: it '+\
'does not contain version information')
key = cls._get_decryption_key(key_container, version)
ptexts.append(cls._decrypt(ctext, key))
split_values = EncMutation.split_values(ptexts)
for sec, values in zip(cell_sections, split_values):
dec_mutation[sec] = list(values)
@classmethod
def encrypt_row(cls, row, key_container):
(key, version) = cls._get_encryption_key(key_container)
#add version number delineated by 'ver - version is int so this is fine
return cls._encrypt(row,key)+'ver'+str(version)
@classmethod
def encrypt_cols(cls, cols, key_container, cell_sections):
(key, version) = cls._get_encryption_key(key_container)
#add version number delineated by 'ver - version is int so this is fine
c_text = EncRange.get_value_by_cell_string(cols, cell_sections)
return cls._encrypt(c_text,key)+'ver'+str(version)
@classmethod
def encrypt_cell(cls, cell_dict, key_container, cell_sections):
(key, version) = cls._get_encryption_key(key_container)
ptext = EncCell.get_value_by_cell_string(cell_dict,cell_sections)
#add version number delineated by 'ver' - version is int so this is fine
return cls._encrypt(ptext, key)+'ver'+str(version)
@classmethod
def decrypt_cell(cls, cell_dict, dec_cell, key_container, cell_location, cell_sections):
#grab the version, delineated by last instance of 'ver'
try:
(ctext,version) = cell_dict[CELL_MUT_MAPPING[cell_location]].rsplit('ver',1)
except ValueError:
raise DecryptionException('Ciphertext is not properly formatted: it '+\
'does not contain version information')
key = cls._get_decryption_key(key_container,version)
ptext = cls._decrypt(ctext, key)
split_value = EncCell.split_value_by_cell_string(ptext)
for (sec, value) in zip(cell_sections, split_value):
dec_cell[CELL_MUT_MAPPING[sec]] = value
@classmethod
def encrypt(cls, string, key):
return cls._encrypt(string, key)
@classmethod
def decrypt(cls, string, key):
return cls._decrypt(string, key)
class Pycrypto_AES_CFB(Pycrypto_AES_Base):
'''
Cipher Feedback Mode with 8-bit segments (pycrypto's default segment length)
Wrapper for pycrypto's AES implementation.
Keys must be 16, 24, or 32 bytes long.
IV is needed, must be random for each new
encryption but can be made public
'''
name = 'Pycrypto_AES_CFB'
@staticmethod
def _encrypt(plaintext, key, iv=None):
#Deal with the case when field is empty
if plaintext is None:
plaintext = ''
if iv is not None and len(iv) != AES.block_size:
raise EncryptionException('IV size must equal cipher block size')
if iv is None:
iv = Random.new().read(AES.block_size)
cipher = AES.new(key, AES.MODE_CFB, iv)
return iv + cipher.encrypt(str(plaintext))
@staticmethod
def _decrypt(ciphertext, key):
#error handling
Pycrypto_AES_Base._has_iv_material(ciphertext)
iv = ciphertext[:AES.block_size]
cipher = AES.new(key, AES.MODE_CFB, iv)
return cipher.decrypt(str(ciphertext[AES.block_size:]))
class Pycrypto_AES_CBC(Pycrypto_AES_Base):
'''
Cipher Block Chaining
Wrapper for pycrypto's AES implementation.
Keys must be 16, 24, or 32 bytes long.
IV is needed, must be random and can be
made public. The plaintext must be a
multiple of 16 bytes long.
'''
name = 'Pycrypto_AES_CBC'
@staticmethod
def _encrypt(plaintext, key, iv=None):
#Deal with the case when field is empty
if plaintext is None:
plaintext = ''
if iv is not None and len(iv) != AES.block_size:
raise EncryptionException('IV size must equal cipher block size')
if iv is None:
iv = Random.new().read(AES.block_size)
cipher = AES.new(key, AES.MODE_CBC, iv)
return iv + cipher.encrypt(Pycrypto_AES_Base._pad(plaintext))
@staticmethod
def _decrypt(ciphertext, key):
#error handling
Pycrypto_AES_Base._has_iv_material(ciphertext)
Pycrypto_AES_Base._is_multiple_16(ciphertext)
iv = ciphertext[:AES.block_size]
cipher = AES.new(key, AES.MODE_CBC, iv)
return Pycrypto_AES_Base._strip_pad(cipher.decrypt(str(ciphertext[AES.block_size:])))
class Pycrypto_AES_OFB(Pycrypto_AES_Base):
'''
Output Feedback
Wrapper for pycrypto's AES implementation.
Keys must be 16, 24, or 32 bytes long.
IV is needed, must be random for each new
encryption but can be made public.
'''
name = 'Pycrypto_AES_OFB'
@staticmethod
def _encrypt(plaintext, key, iv=None):
#Deal with the case when field is empty
if plaintext is None:
plaintext = ''
if iv is not None and len(iv) != AES.block_size:
raise EncryptionException('IV size must equal cipher block size')
if iv is None:
iv = Random.new().read(AES.block_size)
cipher = AES.new(key, AES.MODE_OFB, iv)
return iv + cipher.encrypt(plaintext)
@staticmethod
def _decrypt(ciphertext, key):
#error handling
Pycrypto_AES_Base._has_iv_material(ciphertext)
iv = ciphertext[:AES.block_size]
cipher = AES.new(key, AES.MODE_OFB, iv)
return cipher.decrypt(str(ciphertext[AES.block_size:]))
class Pycrypto_AES_CTR(Pycrypto_AES_Base):
'''
Counter
Wrapper for pycrypto's AES implementation.
Keys must be 16, 24, or 32 bytes long.
Counter value is needed, this should not
be reused with the same key. Counter can
be public.
NIST SP 800-38A recommends generating the initial counter by setting the
leading 64 bits to be a message nonce and the remaining 64 bits to be a
block counter (starting at 1). We generate the message nonce as a 64-bit
random value but do not keep state to check that nonces are unique across
messages for the same key.
'''
name = 'Pycrypto_AES_CTR'
@staticmethod
def _encrypt(plaintext, key, init_ctr=None):
"""Optional initial counter argument, to be used only for testing
purposes, must be the AES block length (16 bytes).
"""
#Deal with the case when field is empty
if plaintext is None:
plaintext = ''
if init_ctr is not None:
if len(init_ctr) != AES.block_size:
raise EncryptionException('Initial counter must be ' +
str(AES.block_size) + ' bytes')
else:
#Convert counter bytes to an integer (an unsigned long long),
#in two steps because unpack takes 8-byte (not 16-byte) input
int1 = struct.unpack('>Q', init_ctr[:AES.block_size//2])[0]
int2 = struct.unpack('>Q', init_ctr[AES.block_size//2:])[0]
init_ctr_int = int1 * 2**(AES.block_size*4) + int2
if init_ctr is None:
#Generate 64-bit nonce randomly
nonce = Random.new().read(AES.block_size//2)
#Set remaining 64 bits to be a block counter starting at 1
init_ctr = struct.pack('15s', nonce) + '\x01'
#Compute integer version of initial counter by scaling nonce by
#64 bits and adding 1
init_ctr_int = struct.unpack('>Q', nonce)[0]*2**(AES.block_size*4)+1
ctr = Counter.new(AES.block_size*8, initial_value = init_ctr_int)
cipher = AES.new(key, AES.MODE_CTR, counter = ctr)
return init_ctr + cipher.encrypt(plaintext)
@staticmethod
def _decrypt(ciphertext, key):
#error handling
Pycrypto_AES_Base._has_iv_material(ciphertext)
try:
nonce = struct.unpack('>Q', ciphertext[:AES.block_size//2])[0]
block_ctr = struct.unpack(
'>Q',
ciphertext[AES.block_size//2 : AES.block_size])[0]
init_ctr = nonce * 2**(AES.block_size*4) + block_ctr
except ValueError:
raise DecryptionException("Value for the start counter value is not an integer.")
ctr = Counter.new(AES.block_size*8, initial_value = init_ctr)
cipher = AES.new(key, AES.MODE_CTR, counter = ctr)
return cipher.decrypt(str(ciphertext[AES.block_size:]))
class Pycrypto_AES_GCM(Pycrypto_AES_Base):
'''
Galois Counter Mode
Wrapper for pycrypto's AES implementation.
Keys must be 16, 24, or 32 bytes long.
Nonce is needed, should not be used again
with the same key but does not need to be
random.
Authenticated mode of encryption, generates
a digest and raises a ValueError if the
ciphertext and digest to do not verify.
'''
name = 'Pycrypto_AES_GCM'
@staticmethod
def _encrypt(plaintext, key, iv=None):
#Deal with the case when field is empty
if plaintext is None:
plaintext = ''
if iv is not None and len(nonce) != AES.block_size:
raise EncryptionException('IV size must equal cipher block size')
if iv is None:
iv = Random.new().read(AES.block_size)
cipher = AES.new(key, AES.MODE_GCM, nonce = iv)
(cipher_text, digest) = cipher.encrypt_and_digest(plaintext)
return iv + cipher_text + digest
@staticmethod
def _decrypt(ciphertext, key):
#error handling
Pycrypto_AES_Base._has_iv_material(ciphertext)
nonce = ciphertext[:AES.block_size]
digest = ciphertext[-AES.block_size:]
cipher = AES.new(key, AES.MODE_GCM, nonce = nonce)
cipher_text = str(ciphertext[AES.block_size:-AES.block_size])
return cipher.decrypt_and_verify(cipher_text, digest)
class Pycrypto_AES_SIV(Pycrypto_AES_Base):
'''
Synthetic IV
Wrapper for pycrypto's AES SIV implementation.
Deterministic mode of encryption, so no nonce
used.
Keys must be either 32 bytes or 64 bytes long.
The message does not need to be padded to a
multiple of 16 bytes.
Authenticated mode of encryption, generates
a digest and raises a ValueError if the
ciphertext and digest do not verify.
'''
name = "Pycrypto_AES_SIV"
@staticmethod
def _encrypt(plaintext, key):
if plaintext is None:
plaintext = ''
cipher = AES.new(key, AES.MODE_SIV)
cipher_text, digest = cipher.encrypt_and_digest(plaintext)
return cipher_text + digest
@staticmethod
def _decrypt(ciphertext, key):
#error handling
Pycrypto_AES_Base._has_iv_material(ciphertext)
digest = ciphertext[-AES.block_size:]
ciphertext = str(ciphertext[:-AES.block_size])
cipher = AES.new(key, AES.MODE_SIV)
return cipher.decrypt_and_verify(ciphertext, digest)
pace/encryption/AES_encrypt_test.py
+190
View File
@@ -0,0 +1,190 @@
## **************
## Copyright 2016 MIT Lincoln Laboratory
## Project: PACE
## Authors: ES
## Description: Test vectors for PACE pycrypto implementation of AES modes of
## operation: CBC, CFB, OFB, and CTR. Test vectors are from NIST SP 800-38A.
## CBC tests do not test the padding scheme; test plaintexts are whole blocks.
## Test vectors for GCM and SIV are not included. For GCM, the test vectors
## available in NIST's ``The Galois/Counter Mode of Operation'' document are
## not applicable because their IVs are not 128 bits, while pycrypto
## requires IVs to be 128 bits. For SIV, the test vectors available in RFC 5297
## include ``associated data'' input, which is not supported by pycrypto's
## implementation.
## Modifications:
## Date Name Modification
## ---- ---- ------------
## 25 Jan 2016 ES Original file
## **************
import os
import sys
this_dir = os.path.dirname(os.path.dirname(__file__))
base_dir = os.path.join(this_dir, '../..')
sys.path.append(base_dir)
from binascii import unhexlify
from pace.common.pacetest import PACETestCase
from pace.encryption.AES_encrypt import Pycrypto_AES_CFB, Pycrypto_AES_CBC, \
Pycrypto_AES_OFB, Pycrypto_AES_CTR
KEY_128 = unhexlify('2b7e151628aed2a6abf7158809cf4f3c')
KEY_192 = unhexlify('8e73b0f7da0e6452c810f32b809079e562f8ead2522c6b7b')
KEY_256 = unhexlify('603deb1015ca71be2b73aef0857d7781'+
'1f352c073b6108d72d9810a30914dff4')
PLAINTEXT = unhexlify('6bc1bee22e409f96e93d7e117393172a' +
'ae2d8a571e03ac9c9eb76fac45af8e51' +
'30c81c46a35ce411e5fbc1191a0a52ef' +
'f69f2445df4f9b17ad2b417be66c3710')
IV = unhexlify('000102030405060708090a0b0c0d0e0f')
INIT_COUNTER = unhexlify('f0f1f2f3f4f5f6f7f8f9fafbfcfdfeff')
class AESTests(PACETestCase):
def test_CBC_AES128(self):
expected_ct = unhexlify('7649abac8119b246cee98e9b12e9197d' +
'5086cb9b507219ee95db113a917678b2' +
'73bed6b8e3c1743b7116e69e22229516' +
'3ff1caa1681fac09120eca307586e1a7')
actual_ct = Pycrypto_AES_CBC._encrypt(PLAINTEXT, KEY_128, IV)
self.assertEqual(IV + expected_ct, actual_ct[:-16])
#test decrypt
actual_pt = Pycrypto_AES_CBC._decrypt(actual_ct, KEY_128)
self.assertEqual(PLAINTEXT, actual_pt)
def test_CBC_AES192(self):
expected_ct = unhexlify('4f021db243bc633d7178183a9fa071e8' +
'b4d9ada9ad7dedf4e5e738763f69145a' +
'571b242012fb7ae07fa9baac3df102e0' +
'08b0e27988598881d920a9e64f5615cd')
actual_ct = Pycrypto_AES_CBC._encrypt(PLAINTEXT, KEY_192, IV)
self.assertEqual(IV + expected_ct, actual_ct[:-16])
#test decrypt
actual_pt = Pycrypto_AES_CBC._decrypt(actual_ct, KEY_192)
self.assertEqual(PLAINTEXT, actual_pt)
def test_CBC_AES256(self):
expected_ct = unhexlify('f58c4c04d6e5f1ba779eabfb5f7bfbd6' +
'9cfc4e967edb808d679f777bc6702c7d' +
'39f23369a9d9bacfa530e26304231461' +
'b2eb05e2c39be9fcda6c19078c6a9d1b')
actual_ct = Pycrypto_AES_CBC._encrypt(PLAINTEXT, KEY_256, IV)
self.assertEqual(IV + expected_ct, actual_ct[:-16])
#test decrypt
actual_pt = Pycrypto_AES_CBC._decrypt(actual_ct, KEY_256)
self.assertEqual(PLAINTEXT, actual_pt)
def test_OFB_AES128(self):
#test encrypt
expected_ct = unhexlify('3b3fd92eb72dad20333449f8e83cfb4a'+
'7789508d16918f03f53c52dac54ed825'+
'9740051e9c5fecf64344f7a82260edcc'+
'304c6528f659c77866a510d9c1d6ae5e')
actual_ct = Pycrypto_AES_OFB._encrypt(PLAINTEXT, KEY_128, IV)
self.assertEqual(IV + expected_ct, actual_ct)
#test decrypt
actual_pt = Pycrypto_AES_OFB._decrypt(actual_ct, KEY_128)
self.assertEqual(PLAINTEXT, actual_pt)
def test_OFB_AES192(self):
#test encrypt
expected_ct = unhexlify('cdc80d6fddf18cab34c25909c99a4174'+
'fcc28b8d4c63837c09e81700c1100401'+
'8d9a9aeac0f6596f559c6d4daf59a5f2'+
'6d9f200857ca6c3e9cac524bd9acc92a')
actual_ct = Pycrypto_AES_OFB._encrypt(PLAINTEXT, KEY_192, IV)
self.assertEqual(IV + expected_ct, actual_ct)
#test decrypt
actual_pt = Pycrypto_AES_OFB._decrypt(actual_ct, KEY_192)
self.assertEqual(PLAINTEXT, actual_pt)
def test_OFB_AES256(self):
#test encrypt
expected_ct = unhexlify('dc7e84bfda79164b7ecd8486985d3860'+
'4febdc6740d20b3ac88f6ad82a4fb08d'+
'71ab47a086e86eedf39d1c5bba97c408'+
'0126141d67f37be8538f5a8be740e484')
actual_ct = Pycrypto_AES_OFB._encrypt(PLAINTEXT, KEY_256, IV)
self.assertEqual(IV + expected_ct, actual_ct)
#test decrypt
actual_pt = Pycrypto_AES_OFB._decrypt(actual_ct, KEY_256)
self.assertEqual(PLAINTEXT, actual_pt)
def test_CTR_AES128(self):
#test encrypt
expected_ct = unhexlify('874d6191b620e3261bef6864990db6ce'+
'9806f66b7970fdff8617187bb9fffdff'+
'5ae4df3edbd5d35e5b4f09020db03eab'+
'1e031dda2fbe03d1792170a0f3009cee')
actual_ct = Pycrypto_AES_CTR._encrypt(PLAINTEXT, KEY_128, INIT_COUNTER)
self.assertEqual(str(INIT_COUNTER) + expected_ct, actual_ct)
#test decrypt
actual_pt = Pycrypto_AES_CTR._decrypt(actual_ct, KEY_128)
self.assertEqual(PLAINTEXT, actual_pt)
def test_CTR_AES192(self):
#test encrypt
expected_ct = unhexlify('1abc932417521ca24f2b0459fe7e6e0b'+
'090339ec0aa6faefd5ccc2c6f4ce8e94'+
'1e36b26bd1ebc670d1bd1d665620abf7'+
'4f78a7f6d29809585a97daec58c6b050')
actual_ct = Pycrypto_AES_CTR._encrypt(PLAINTEXT, KEY_192, INIT_COUNTER)
self.assertEqual(str(INIT_COUNTER) + expected_ct, actual_ct)
#test decrypt
actual_pt = Pycrypto_AES_CTR._decrypt(actual_ct, KEY_192)
self.assertEqual(PLAINTEXT, actual_pt)
def test_CTR_AES256(self):
#test encrypt
expected_ct = unhexlify('601ec313775789a5b7a7f504bbf3d228'+
'f443e3ca4d62b59aca84e990cacaf5c5'+
'2b0930daa23de94ce87017ba2d84988d'+
'dfc9c58db67aada613c2dd08457941a6')
actual_ct = Pycrypto_AES_CTR._encrypt(PLAINTEXT, KEY_256, INIT_COUNTER)
self.assertEqual(str(INIT_COUNTER) + expected_ct, actual_ct)
#test decrypt
actual_pt = Pycrypto_AES_CTR._decrypt(actual_ct, KEY_256)
self.assertEqual(PLAINTEXT, actual_pt)
def test_CFB8_AES128(self):
#test encrypt
plaintext = PLAINTEXT[:18]
expected_ct = unhexlify('3b79424c9c0dd436bace9e0ed4586a4f32b9')
actual_ct = Pycrypto_AES_CFB._encrypt(plaintext, KEY_128, IV)
self.assertEqual(IV + expected_ct, actual_ct)
#test decrypt
actual_pt = Pycrypto_AES_CFB._decrypt(actual_ct, KEY_128)
self.assertEqual(plaintext, actual_pt)
def test_CFB8_AES192(self):
#test encrypt
plaintext = PLAINTEXT[:18]
expected_ct = unhexlify('cda2521ef0a905ca44cd057cbf0d47a0678a')
actual_ct = Pycrypto_AES_CFB._encrypt(plaintext, KEY_192, IV)
self.assertEqual(IV + expected_ct, actual_ct)
#test decrypt
actual_pt = Pycrypto_AES_CFB._decrypt(actual_ct, KEY_192)
self.assertEqual(plaintext, actual_pt)
def test_CFB8_AES256(self):
#test encrypt
plaintext = PLAINTEXT[:18]
expected_ct = unhexlify('dc1f1a8520a64db55fcc8ac554844e889700')
actual_ct = Pycrypto_AES_CFB._encrypt(plaintext, KEY_256, IV)
self.assertEqual(IV + expected_ct, actual_ct)
#test decrypt
actual_pt = Pycrypto_AES_CFB._decrypt(actual_ct, KEY_256)
self.assertEqual(plaintext, actual_pt)
pace/encryption/DEMO.md
+396
View File
@@ -0,0 +1,396 @@
# Confidentiality Demo Commands
This document contains documentation for the commands, primarily in `demo.py`,
for quickly getting up to speed on and running our client-side encryption code
for Accumulo users. This includes field-level encryption, Cryptographically
Enforced Attribute Based Access Control (CEABAC), and the key management
infrastructure that supports CEABAC. Users should read this file if they want a
tour of what our encryption code can accomplish and a look at how this code
works behind the scenes.
## Setup
When trying out our functionality, it is useful to be able to see both the
output shown to the user and the information stored on the server. This is
particularly useful when demonstrating the protections PACE adds, in this case
in terms of how little the server can learn about the users' information it
stores.
For this demo, we will denote the client's view as a Python interpreter shell,
and the server's view as the Accumulo shell. The client view will look as
follows:
```
$ python -i demo.py
>>> run_insert
<function run_insert at 0x...>
>>>
```
We will represent the server view as follows:
```
$ $ACCUMULO_HOME/bin/accumulo shell -u root
Password:
root@server> create_table test_table
root@server test_table> scan
...
root@server test_table>
```
Note that throughout this demo, we provide sample table names for each of these
commands. When running the commands multiple times, it is useful to either vary
the table names or delete the old versions of the tables, to avoid collisions.
## demo.py commands
To simplify the insertion process into Accumulo, we have defined a function
`run_insert()` in demo.py, with arguments as follows:
```python
def run_insert(conn, data, table, config_name, PKI_object):
...
```
- `conn`: the connection to use to connect to Accumulo. Default: `None`, which
will set the connection to `Accumulo(host='localhost', port=42424,
user='root', password='secret')`.
- `data`: a list of five-tuples, where each field in order represents the row,
visibility, column family, column qualifier, and value. This list is the data
to be inserted into `conn`; we provide several sample values for this at the
top of `demo.py`. The default value is `expressive_elems`, one of these sample
values.
- `table`: the name of the table into which to insert `data`. Default: `'demo'`.
- `config_name`: the name of the schema to use to encrypt the data, to look up
in the `schema` dictionary at the top of the file. Default: `'expressive'`.
- `PKI_object`: the PKI object to connect to in order to fetch encryption and
decryption keys. Default is `None`, which defaults to `DummyEncryptionPKI`,
from `encryption_pki.py`. Note that this has hardcoded values stored in it,
and as such will suffice for most of this demo, but a live PKI instance will
also work.
Dual to this, we also define `run_retrieve()`, which retrieves and decrypts
all values from a given Accumulo table:
```python
def run_retrieve(conn, table, config_name, PKI_object):
...
```
These arguments are the same as above, with the exception of `data`, which is
not present as an argument to `run_retrieve()`.
To perform more specific searches, we also provide `run_search()`, which
performs a targeted search for a specific deterministically encrypted row in a
table:
```python
def run_search(conn, table, config_name, PKI_object, row_range):
...
```
Again, the arguments are the same as above, except for the new argument,
`row_range`, which is the name of the row to search for. The default value for
this argument is `'Analytics'`, which works well with some of our default data.
For testing our PKI infrastructure, we provide two more functions,
`run_pki_insert()` and `run_revoke()`:
```python
def run_pki_insert(genpath, user, conn):
...
def run_revoke(user, attr, genpath, conn):
...
```
Each of these functions takes an argument `conn`, as previously defined, and
`genpath`, which denotes a path to a file containing a list of user information.
This defaults to `'../pki/user_info.cfg'` for both functions.
The `user` argument for `run_pki_insert()` does not change the information
inserted into the PKI, but denotes the user to connect to the PKI as in the
return value. This function returns two values: `pki`, an encryption PKI that
can be used as an argument to any of the functions above that take a
`PKI_object` argument, and `fe`, a PKI front-end object that can be used to show
the contents of the PKI, and is documented in the section on using the PKI.
`run_revoke()` is responsible for revoking a user's access to a given attribute
in the key store hosted at `conn`. It takes `user` and `attr` arguments, and
revokes that user's access to that attribute in the key store.
When running the demo, it is important not to interleave calls using a live PKI
and calls using the dummy PKI, as they will interact destructively with one
another. If this happens, it is necessary to reinitialize the PKI object with
`run_pki_insert()`.
## Field-level encryption
To insert elements into an Accumulo table with no encryption, run an insert
using `'none'` as the `config_name` argument:
```
>>> run_insert(table='demo_table_none', config_name='none')
Enrypting with the following schema:
Currently using no encryption scheme
Inserting entry:
Row - MITLL,
Column_Visibility - (a&b)|c,
Column_Family - Analytics,
Column_Qualifier - Rooms,
Value - 1200
...
>>>
```
The server can then read this data as follows:
```
root@server> scan -t demo_table_none
Boston Analytics:Population [(a&b)|c] 3000000
Cambridge Analytics:Population [(a&b)|c] 400000
MITLL Analytics:Employees [(a&b)|c] 4000
MITLL Analytics:Rooms [(a&b)|c] 1200
root@server>
```
In order to hide data from the server, we can encrypt it with a scheme. For
example, the default scheme:
```
>>> run_insert(table='demo_table_default')
Enrypting with the following schema:
Row contains: Row encrypted with Pycrypto_AES_SIV
ColFamily contains: ColFamily & ColQualifier encrypted with Pycrypto_AES_CFB
ColQualifier contains: Nothing
Value contains: Value encrypted with AES_GCM
Inserting entry:
Row - MITLL,
Column_Visibility - (a&b)|c,
Column_Family - Analytics,
Column_Qualifier - Rooms,
Value - 1200
...
>>>
```
Running this server-side, all the server can see beyond the visibility field is
that several of the rows (which are encrypted with deterministic encryption)
have the same value; the rest will have different values each time, as they are
randomized:
```
root@server> scan -t demo_table_default
I\xB3\x82\xF1\xC1\xB3y\x8C\xED\xB6\xD9-\x1B\xD9\x855\xBD\x1D\x9B\x97[+ver1
\x06\xF9\xC8\xDD(w\x98\xECp\xE6\xE70)\x08\x10\xB7\xE3\xFD_r\xD9\x98\xBE\xB9lI^\x91\xF5\x95\x80\x8B\xB4M\x05\x0Aver3:
[(a&b)|c]
"\xF9\xFFE\xE4\x05\xC4\x09\\\x02\xB5\x87\xEE\xA5\xE8d\xCCV\xF7t\xBD$),f=\x98\x0E~\xB8\x8B\xEF\x8C-\xD7\xFAz\xB9\x9A\xA2\x0A\xDC\xA0\xCFxJ:\x92ver3
...
root@server>
```
We can also get a more legible view of the server's state client-side with the
`'print_pretty'` config:
```
>>> run_retrieve(table='demo_table_default', config_name='print_pretty')
Entry:
Row - SbOC8cGzeYztttktG9mFNb0dm5dbK3ZlcjE=,
Column_Visibility - (a&b)|c,
Column_Family - BvnI3Sh3mOxw5ucwKQgQt+P9X3LZmL65bElekfWVgIu0TQUKdmVyMw==,
Column_Qualifier - ,
Value - Ivn/ReQFxAlcArWH7qXoZMxW93S9JCksZj2YDn64i++MLdf6ermaogrcoM94SjqSdmVyMw==
...
>>>
```
This formats the cell by field and base 64 encodes encrypted values for
legibility, but does not decrypt anything. We can also decrypt the values
client-side:
```
>>> run_retrieve(table='demo_table_default')
Entry:
Row - Boston,
Column_Visibility - (a&b)|c,
Column_Family - Analytics,
Column_Qualifier - Population,
Value - 3000000
...
>>>
```
As discussed previously, since the row in this data is encrypted
deterministically, we can also perform a targetted query on specific rows of
data by encrypting the row and searching on the ciphertext, using
`run_search()`:
```
>>> run_search(table='demo_table_default', row_range='Boston')
Entry: (Boston, Analytics, Population, 3000000)
Finished, decrypted 1 total entries.
>>> run_search(table='demo_table_default', row_range='MITLL')
Entry: (MITLL, Analytics, Rooms, 1200)
Entry: (MITLL, Analytics, Employees, 4000)
Finished, decrypted 2 total entries.
>>>
```
## Cryptographically enforced access control
In addition to field-level encryption, as shown above, we also support
cryptographic enforcement of Accumulo's visibility expression-based access
control. This involves two components: a key management system to generate a
different AES key for each attribute that can occur in the Accumulo
installation, and protocol that uses these keys to encrypt data in such a way
that no user without permissions that satisfy a cell's visibility field can
decrypt the value stored in that cell, since they will also not have access to a
set of keys that can decrypt it.
To generate a PKI object, one uses `run_pki_insert()`, as described earlier in
this document:
```
>>> pki, fe = run_pki_insert()
...
>>> pki
<pace.encryption.encryption_pki.EncryptionPKIAccumulo object at 0x...>
>>> fe
<pace.pki.frontend.KeyStoreFrontEnd object at 0x...>
>>>
```
This function has two return values. The first, `pki`, is what we will pass in
as an argument to functions that need to access the keys stored here. Note that
this _cannot_ be used in conjunction with the default PKI generated when no
`PKI_object` argument is provided; if these are accidentally mixed up, the PKI
object must be regenerated with `run_pki_insert()`.
The other object is `fe`, a front end to the key store from which the user
retrieves their keys. It can show all of the attributes that a user has access
to, and a base 64 encoding of their keywraps, for readability. This is done with
the `view()` method:
```
>>> fe.view()
No metadata specified
Using default metadata VIS_AES_CFB
No username specified
Using default username user1
No attribute specified
Showing all attributes
Showing key information for user: user1
Metadata being used: VIS_AES_CFB
========================================
Key attribute: a
Key version : 1
Keywrap:
KelcP0WWpXZkIkcN...
```
Note that this uses certain default values for the metadata, username, and
attribute to view. This is because it has three arguments, as follows:
- `metadata`: the metadata to view in the key store. Defaults to `None`, which
shows keys corresponding to all metadata values.
- `user`: the user whose keys to view. Defaults to `None`, which prints a
message and sets `user` to `'user1'`.
- `attr`: the attribute to view in the key store. Defaults to `None`, which
shows keys corresponding to all attributes.
By default, we generate a key store with three users, `user1`, `user2`, and
`user3`. Users 1 and 2 have access to attributes a, b, c, and d; user 3 only has
access to a, c, and d.
Once this PKI is generated, it can be used when encrypting cells based on their
visibility field, using what we call Cryptographically Enforced Attribute Based
Access Control (CEABAC). The config name `'vis'` uses this scheme to encrypt
values in its table:
```
>>> run_insert(table='ceabac', config_name='vis', data=vis_elems, PKI_object=pki)
...
>>>
```
Note that this protects data from being disclosed to users without permissions
to access it. For example, suppose the server removes the requirement for the
attribute `b` from one of the cells:
```
root@server ceabac> insert Analytics '' '' <a&b ciphertext> -l a&a
root@server ceabac> getauths -u user3
a,c,d
root@server ceabac>
```
If user `user3` attempts to scan this table, they will not receive meaningful
output when attempting to decrypt this element:
```
>>> grant_auths(table='ceabac')
>>> run_retrieve(table='ceabac', config_name='vis', PKI_object=user3_pki)
...
Entry:
Row - Analytics,
Column_Visibility - a&a,
Column_Family - ,
Column_Qualifier - ,
Value - ,C�
...
>>>
```
## Key revocation
We also support the ability to revoke a user's access to certain attributes and
keys. We support forward security (a user can still decrypt old information, but
no new information encrypted under that attribute) by assigning versions to
keys. Each time an attribute is revoked, a new version is generated for its
associated key and distributed to users. Suppose we have a user `user2` whose
permission to the attribute `b` we wish to revoke:
```
>>> user2_pki, user2_fe = run_pki_insert(user='user2')
>>>
```
If we insert elements into a table as root, revoke the key, and perform a query
as `user2`, the data will still be decryptable if `user2` cashed their keys:
```
>>> run_insert(table='revoke', config_name='vis', data=vis_elems, PKI_object=pki)
...
>>> run_revoke('user2', 'b')
>>> run_retrieve(table='revoke', config_name='vis', PKI_object=user2_pki)
...
Entry:
Row - Analytics,
Column_Visibility - a&b,
Column_Family - ,
Column_Qualifier - ,
Value - 200
...
>>>
```
However, if we re-insert the data as root, we use a new key version, thus making
it invisible to `user2`:
```
>>> run_insert(table='revoke', config_name='vis', data=vis_elems, PKI_object=pki)
>>> run_retrieve(table='revoke', config_name='vis', PKI_object=user2_pki)
...
Error: Entry failed to decrypt.
Error message: The key object does not contain keys for the necessary attributes
to decrypt this cell
...
>>>
```
pace/encryption/README.md
+397
View File
@@ -0,0 +1,397 @@
DISTRIBUTION STATEMENT A. Approved for public release: distribution unlimited.
This material is based upon work supported by the Department of Defense under Air Force Contract
No. FA8721-05-C-0002 and/or FA8702-15-D-0001. Any opinions, findings, conclusions or
recommendations expressed in this material are those of the author(s) and do not
necessarily reflect the views of the Department of Defense.
&copy; 2015 Massachusetts Institute of Technology.
MIT Proprietary, Subject to FAR52.227-11 Patent Rights - Ownership by the contractor (May 2014)
The software/firmware is provided to you on an As-Is basis
Delivered to the US Government with Unlimited Rights, as defined in DFARS Part 252.227-7013
or 7014 (Feb 2014). Notwithstanding any copyright notice, U.S. Government rights in this
work are defined by DFARS 252.227-7013 or DFARS 252.227-7014 as detailed above. Use of
this work other than as specifically authorized by the U.S. Government may violate any
copyrights that exist in this work.
# Field Level Encryption and Cryptographic Enforced Attribute Based Access Control
This is a Python library for client-side encryption of Accumulo database entries.
It provides the functionality to encrypt every entry in a pyaccumulo
mutation according to a user defined configuration. Field Level Encryption provides a solution
to the problem of providing confidentiality guarantees against a malicious server.
The extended Cryptographic Enforced Attribute Based Access (CEABAC) library
extends that protection against curious clients in the case the malicious server does
not correctly enforce visibility labels.
## Installation
Installation instructions for the pace module as a whole can be found in the README in the
top level directory.
### Dependencies
This project depends on several other Python libraries and utilities. They are
listed below with links to their project pages. In addition to these packages,
it also requires a running Accumulo instance to run anything beyond the unit tests.
- [nosetests](https://nose.readthedocs.org/en/latest/), version 1.3.3+.
- [pyaccumulo](https://pypi.python.org/pypi/pyaccumulo), version 1.5.0.6+.
- [pycrypto](https://www.dlitz.net/software/pycrypto/). This represents latest experimental release,
2.7a1, as it implements Galios/Counter Mode.
- [Beaker](https://pypi.python.org/pypi/Beaker), version 1.7.0+.
- [enum34](https://pypi.python.org/pypi/enum34), version 1.0.4+.
## Use
The primary interface for the confidentiality library is defined in
`acc_encrypt.py`. This is a high level interface for the encryption of
mutations and the decryption of cells. Our confidentiality library uses
pyaccumulo, a Python wrapper around Accumulo's Thrift interface, to interact with Accumulo
instances.
To encrypt or decrypt, users import the `AccumuloEncrypt`
class:
```python
import pace.encryption.acc_encrypt.AccumuloEncrypt
encrypter = AccumuloEncrypt(config_file, PKI_object)
```
Arguments for the class include a file pointer to a configuration file,
described in the section describing the [configuration file](#configuration-file),
and a handle on a key management object
described in the following [section](#pki-objects). In short, the configuration file specifies
the types of encryption and what portions of the cell are to be encrypted
together, and where the ciphertext will be stored. The key management object
connects with the keystore where the user's encryption and decryption keys are
stored. Note that one configuration file represents the configuration of only
one Accumulo table. If the user wishes to use a different configuration
for another table, another `AccumuloEncrypt` object must be created with
that configuration file as a parameter.
### Configuration File
The configuration files input into the client-side encryption library
specify both the type of encryption desired as well as the formatting
of the cells. It is possible to individually encrypt each field in
the cell or to encrypt subsets of the cell in one ciphertext. In addition to
encryption scheme and configuration information, the config file also
specifies the information needed to be given to the key object in order
to obtain encryption and decryption keys.
This section presents the various options available in the configuration
file as well as several examples of such. More examples can be found in the
`config` directory.
#### Basic syntax
The files follow the basic format expected in Windows INI files and are
parsed with Python’s [configparser](https://docs.python.org/2/library/configparser.html)
tool. The basic syntax takes the form of:
[section]
key_word = some_value
Each entry (a section and corresponding key words) specifies the encryption
algorithm, key information and the parts of the cell to encrypt. The
section title itself denotes the part of the cell where
the encrypted information will end up - the target location. For
example:
[row]
key_id = keyid1234
encryption = Pycrypto_AES_CFB
cell_sections = row
cell_key_length = 16
In this case the row of the cell is to be encrypted with AES in
CFB mode and placed back in the row.
Each entry must have a `key_id` and `encryption` keyword specified,
`cell_sections` and `cell_key_length` is optional.
####Section
The section headers are:
* row
* colFamily
* colQualifier
* colVisibility
* value
At this time it is not possible to encrypt either the `timestamp` or the
`to_delete` portion of an Accumulo cell - it would break functionality
of Accumulo in a significant way.
####Key_id
The `key_id` is the identifying information needed to obtain the keys
needed for encryption/decryption and is handed to the
`key_object` to use (see `pki/abstract_pki.py` for more information).
The `key_id` is treated as a string before it is handed to the `key_object`.
This may be some combination of user information, the algorithm being
used, or whether it is an encryption or decryption query. Note, if the
user wishes to have different keys for each table, this would where it
would be specified i.e. `key_id = table_A`. This value
is what is specified in the metadata in the configuration files used in key generation, see the
[Readme](pki/README.md) under Configuration File Format for more information.
####Encryption
The confidentiality library uses
[Pycrypto](https://www.dlitz.net/software/pycrypto/) to implement
the different modes of AES. For semantically secure encryption we offer several
modes: Cipher Feedback Mode, Cipher Block Chaining Mode, Output Feedback Mode,
and Counter Mode. Of the four of these modes, we recommend Pycrypto's
implementation of Output Feedback Mode, as it is the fastest of the four. For
authenticated semantically secure encryption, we support Galois Counter Mode.
Finally, for deterministic encryption we use Synthentic IV mode
(SIV), which, when used without a nonce, is deterministic.
We also support an `identity` mode, which exercises the confidentiality library's logic for
parsing mutations, but does not actually encrypt the plaintext values.
The names in the configuration files are as follows:
* Identity - Useful for testing, it is simply the identity function
* Pycrypto_AES_CFB - The Pycrypto implementation of AES in CFB mode
* Pycrypto_AES_CBC - The Pycrypto implementation of AES in CBC mode
* Pycrypto_AES_OFB - The Pycrypto implementation of AES in OFB mode
* Pycrypto_AES_CTR - The Pycrypto implementation of AES in CTR mode
* VIS_Identity - Constructs the shares of the cell_key but does not
encrypt them, used primarily for demo purposes
* VIS_AES_CFB - Cryptographically Enforced ABAC that uses AES in CFB mode
to encrypt the shares and value
* VIS_AES_CBC - Cryptographically Enforced ABAC that uses AES in CBC mode
to encrypt the shares and value
* VIS_AES_OFB - Cryptographically Enforced ABAC that uses AES in OFB mode
to encrypt the shares and value
* VIS_AES_CTR - Cryptographically Enforced ABAC that uses AES in CTR mode
to encrypt the shares and value
* VIS_AES_GCM - Cryptographically Enforced ABAC that uses AES in GCM mode
to encrypt the shares and value
####Cell_sections
`cell_sections` specifies the portions of the cell that are to be encrypted
together and placed in the target location. As stated above, it is an
optional field, and if not specified, it is assumed that
the portion of the cell to be encrypted is the one specified in the
section header.
If one wishes to encrypt more than one portion of the cell, the format
is a comma separated list of cell locations, as demonstrated in the
example below:
[colFamily]
key_id = keyid234
encryption = Pycrypto_AES_CFB
cell_sections = colFamily,colQualifier
Possible values for `cell_sections` are the same as those listed in
section headers in any combination:
* row
* colFamily
* colQualifier
* colVisibility
* value
####Cell_key_length
`cell_key_length` specifies the length (in bytes) of the cell key
that is generated to be shared amongst the different attributes.
It is only used for encryption schemes that use Cryptographically
Enforced ABAC - if not specified for these schemes it defaults to
16 bytes. Currently allowed values include 16, 24, and 32 bytes.
If it is present for a non-CEABAC algorithm, the value
will be ignored.
### PKI Objects
The key management object passed into the `AccumuloEncrypt` represents the
interface to acquire keys for encryption and decryption from the keystore. We
provide an abstract interface that it must meet and an instantiation that
communicates with an Accumulo keystore. More information can be found in the
README in the `pki` directory.
At a high-level, there are two types of key: attribute keys used for CEABAC,
and non-attribute keys strictly used for standard AES encryption modes. Symmetric key
cryptography is used throughout the confidentiality library, thus the same key
is used for encryption and decryption. The interface that all encryption key
objects meet inherits from `AbstractPKI`, which both the signature
and encryption key infrastructures are based on. The
abstract class, `EncryptionPKIBase`, is found in
`encryption_pki.py`. Instantiations of this class must implement four
methods:
* `get_current_key(algorithm)`: retrieves the most recent version of
the non-attribute key for the associated algorithm, to be used when new records
are being encrypted and inserted into Accumulo. It is used for non-CEABAC
algorithms.
* `get_key(algorithm, version)`: retrieves that particular version of the non-attribute key for the
associated algorithm. In this case, the algorithms that use this method are non-CEABAC schemes. This method
is used during decryption when the version information is extracted from the ciphertext to obtain the
key to decrypt the rest of the message.
* `get_current_attribute_key(algorithm, attribute)`: retrieves the current version of the
attribute key for the algorithm. Used to acquire the most recent key, which is used to encrypt new
records to insert. This is used for CEABAC schemes, and the `attribute` parameter corresponds to the attributes found in
visibility fields in Accumulo.
* `get_attribute_key(algorithm, attribute, version)`: retrieves that version of the attribute key
for a particular attribute-algorithm pairing. The `attribute` parameter corresponds to the attributes found in
visibility fields in Accumulo. This method is used during decryption in CEABAC algorithms to obtain
the attribute key to decrypt the shares of the cell key.
Algorithms specified as arguments for all these functions are the same name
defined in the configuration file as the `key_id`, such as
`Pycrypto_AES_CBC` or `VIS_AES_CTR`.
We provide an implementation of the interface that communicates with the
Accumulo keystore. Users
can declare a `pki` instance by:
```python
pki = EncryptionPKIAccumulo(conn, user_id, private_rsa_key)
```
The class takes in three arguments. The first, `conn`, is the connection,
as defined in Pyaccumulo, to the Accumulo instances that is storing the keys.
The second, `user_id`, is the string that represents the user currently
encrypting or decrypting records. This value must be synchronized with the key
generation process, as it used during key lookup. Note that in
this implementation a particular instance of `EncryptionPKIAccumulo`
corresponds to one particular user. The final argument is the user's
private RSA key corresponding to the public key used during keywrapping in key
generation. Again, see the README in the `pki` for more information.
There exist two other implementations of the `EncryptionPKIBase` interface.
The first is an extension of `EncryptionPKIAccumulo`, called
`CachingEncryptionPKIAccumulo`, which uses
[beaker](http://beaker.readthedocs.org/en/latest/) to cache the
results of querying the keystore for an hour. This improves performance and
reduces hits on the keystore. The second is `DummyEncryptionPKI`, which
contains hardcoded keys and is useful for unit tests and other demos. Both are
defined in `encryption_pki.py`.
### Encrypting Entries
Once the `encrypter` is created, users can construct encrypted mutations
by passing in a plaintext mutation:
```python
encrypted_mutations = encrypter.encrypt(mutation)
```
The mutation passed in is not changed; instead, it is used to construct a
list of encrypted mutations according to the configuration file passed in at
initialization. A list of mutations is constructed instead of a single mutation
because the row must be encrypted separately for each column/value in the mutation.
The generated mutations only contain a single update. This list
of encrypted mutation can now be written to Accumulo using a
writer as one would with a normal mutation.
### Decrypting Entries
Decrypting the results retrieved from Accumulo also requires an
`AccumuloEncrypt` object. The structure received from a scan is a cell,
which can be directly passed into the decrypt method of the encrypter object:
```python
cell = encrypter.decrypt(encrypted_cell)
```
As with the mutation, the encrypted cell is not changed, but is used to
construct a decrypted version of the cell. The returned cell has its fields that
were encrypted together placed back in their original fields.
### Searching for Entries
The confidentiality library supports several types of scans, depending how the
leading portion of the cell (the row and column) is encrypted. For configurations
that encrypt the row with semantically secure encryption, the only scan supported is
a full-table scan, where individual
records are pulled back all at once and decrypted until the information the user
wants is found:
```python
for entry in conn.scan(table):
cell = encrypter.decrypt(entry)
```
For configuration files that use deterministic
encryption on the cell's row or column values, it is possible
to do a targeted equality scan. For example, if the `row` is
encrypted using deterministic encryption, one can search for all instances of
the row 'a' by doing the following:
```python
enc_row, _ = encrypter.encrypt_search('a')
range = pyaccumulo.Range(srow = enc_row, erow = enc_row)
entries = conn.scan(table, scanrange = range)
```
If the column is encrypted using deterministic encryption as well
it is possible to search over the columns as well. This includes
encrypting both the colFamily and colQualifier as part of the same
ciphertext. An example is shown below:
```python
enc_row, enc_cols = encrypter.encrypt_search('a',[['colFam1','colQual1'],['colFam2',colQual2']])
range = pyaccumulo.Range(srow = enc_row, erow = enc_row)
entries = conn.scan(table, scanrange = range, cols = enc_cols)
```
Finally, when the row in a cell, or leading portion of the cell remains unencrypted,
normal Accumulo range queries can be used.
### Use Example
```python
# Imports
import pyaccumulo
from pace.encryption.acc_encrypt import AccumuloEncrypt
from pace.encryption.encryption_pki import EncryptionPKIAccumulo
# Connection setup
pki_conn = pyaccumulo.Accumulo(host='localhost', port='42424', user='root', password='secret')
conn = pyaccumulo.Accumulo(host='localhost', port='42424', user='root', password='secret')
# Create the key object, encrypter and mutation
# NB: rsa_key must be the key used to wrap keys in the keystore, and user1 must exist with the proper keys
pki = EncryptionPKIAccumulo(pki_conn, 'user1', rsa_key)
encrypter = AccumuloEncrypt('config_file.cfg', pki)
mutation = pyaccumulo.Mutation('test_row')
mutation.put(cf='test_column_family', cq='test_cq', cv='default', val='this is a test value')
# Encrypt the mutation and write it to a table
encrypted_mutations = encrypter.encrypt(mutation)
conn.create_table('demo_table')
wr = conn.create_batch_writer('demo_table')
for mutation in encrypted_mutations
wr.add_mutation(mutation)
wr.close()
# Search for and decrypt the mutation
range = pyaccumulo.Range(srow = 'test_row', erow = 'test_row')
encrypted_range = encrypter.encrypt_search(range)
for entry in conn.scan('demo_table', scanrange = encrypted_range):
cell = encrypter.decrypt(entry)
```
## Testing Code
The unit tests can be run just by running `nosetests` from the
shell in the `pace/encryption/` directory with nosetests installed.
pace/encryption/__init__.py
+11
View File
@@ -0,0 +1,11 @@
## **************
## Copyright 2015 MIT Lincoln Laboratory
## Project: PACE
## Authors: EHS
## Description: Init File for Encryption folder
## Modifications:
## Date Name Modification
## ---- ---- ------------
## 17 May 2015 EHS Original file
## **************
##
pace/encryption/abstract_encrypt.py
+219
View File
@@ -0,0 +1,219 @@
## **************
## Copyright 2014 MIT Lincoln Laboratory
## Project: PACE
## Authors: ATLH
## Description: Abstract class for encryption modules
## Modifications:
## Date Name Modification
## ---- ---- ------------
## 17 Dec 2014 ATLH Original file
## 12 Mar 2015 ATLH Removed AES into own file
## **************
import os
import sys
this_dir = os.path.dirname(os.path.dirname(__file__))
base_dir = os.path.join(this_dir, '../..')
sys.path.append(base_dir)
import fractions
from abc import ABCMeta, abstractmethod
from Crypto.Cipher import AES
from Crypto.Util import Counter, number
from Crypto import Random
from pyaccumulo import Cell
from pace.encryption.enc_mutation import EncMutation, EncCell
from pace.encryption.vars import CELL_MUT_MAPPING
from pace.encryption.encryption_exceptions import EncryptionException, DecryptionException
class AbstractEncrypt(object):
"""Abstract interface for all encryption modules
that (may) interact with the Accumulo Encryption module.
Key management is assumed to be seperate and keys must be
an input into this class.
"""
__metaclass__ = ABCMeta
"""
Name of algorithm as accepted in the configuration file
"""
name = ''
@abstractmethod
def encrypt_mutation(mutation, key, cell_sections):
'''
Arguments:
mutation - mutation as defined in encmutation (based on
pyaccumulo interface
key - key in whatever format is used for particular
encryption scheme
cell_sections - the list of part of the cell to be encrypted.
Options are defined in VALID_KEYS in vars.py.
Returns: The ciphertext. For example if
cell_sections was [colFamily, colQualifier] and
cell_location is 'colFamily'. Let the first update
be cf = 'cf1', and cq = 'cq1', the resulting update
would become cf = 'cf1|cq1' (encrypted) and cq = ''.
This is done for each update in the list of updates.
'''
pass
@abstractmethod
def decrypt_mutation(mutation, dec_mutation, key, cell_location, cell_sections):
'''
Arguments:
mutation - mutation as defined in encmutation (based on
pyaccumulo interface
dec_mutation - the location where the decrypted information
in mutation is stored upon decryption
key - key in whatever format is used for particular
encryption scheme
cell_location - the part of the cell where encrypted data is
to be stored
cell_sections - the target locations for decrypted data that
contained at the cell_location.
Effect: The dec_mutation is modified. The encrypted data
cell_location is decrypted and seperated into
the values associated with each of the cell_sections.
These values are then placed in the cell as
specified by cell_sections.
For example, cell_location = 'cq' and cell_sections =
[colFamily, colQualifier, value] and the cq for
the first update is 'cf1|cq1|val1' the three
values are separated out and the update becomes
cf = 'cf1', cq = 'cq1', and val = 'val1'.
'''
pass
@abstractmethod
def encrypt_row(row, key):
"""
Arguments:
row - (string) row to be encrypted
key - key in whatever format is used for particular
encryption scheme
Returns: encrypted row used to scan the database
for deterministically encrypted row.
"""
pass
@abstractmethod
def encrypt_cols(cols, key_container, cell_sections):
"""
Arugments:
cols: dictionary keyed by 'colFamily' and 'colQualifier'
containing the respective values (defined in the
nested list passed into acc_encrypt) that are
to be encrypted. If colQualifier is not specified
in the input list, the value defaults to ''
key_container - key container to obtain the encryption key
cell_sections - the target locations for decrypted data that
contained at the cell_location.
Returns: encrypted portion of the column as specified
by cell_sections
"""
pass
@abstractmethod
def encrypt_cell(cell, key, cell_sections):
'''
Arguments:
cell - DecCell as defined in enc_mutation (based on
pyaccumulo interface
key - key in whatever format is used for particular
encryption scheme
cell_sections - the list of part of the cell to be encrypted.
Options are defined in VALID_KEYS in vars.py.
Returns: encrypted portion of the cell
'''
pass
@abstractmethod
def decrypt_cell(cell_dict, dec_cell, key, cell_location, cell_sections):
'''
Arguments:
cell_dict - dictionary containing field names as
keys, and the values to be decrypted
dec_cell - dictionary where the decrypted values
are placed
key - key in whatever format is used for particular
encryption scheme
cell_location - the part of the cell where encrypted data is
to be stored
cell_sections - the target locations for decrypted data that
contained at the cell_location.
Returns: The dec_cell is modified. The encrypted data
cell_location is decrypted and seperated into
the values associated with each of the cell_sections.
These values are then placed in the cell as
specified by cell_sections.
'''
pass
class Identity_AccEncrypt(AbstractEncrypt):
'''
Identity function as encryption algorithm, used for
testing
'''
name = 'Identity'
@staticmethod
def encrypt_mutation(mutation, key, cell_sections):
ctexts = EncMutation.concatenate_cell_section_values(mutation, cell_sections)
return ctexts
@staticmethod
def decrypt_mutation(mutation, dec_mutation, key, cell_location, cell_sections):
ptexts = mutation[cell_location]
split_values = EncMutation.split_values(ptexts)
for sec, values in zip(cell_sections, split_values):
dec_mutation[sec] = list(values)
@staticmethod
def encrypt_row(row, key_container):
return row
@staticmethod
def encrypt_cols(cols, key_container, cell_sections):
return cols
@staticmethod
def encrypt_cell(cell, key, cell_sections):
return EncCell.get_value_by_cell_string(cell,cell_sections)
@staticmethod
def decrypt_cell(cell, dec_cell, key, cell_location, cell_sections):
split_value = EncCell.split_value_by_cell_string(cell[CELL_MUT_MAPPING[cell_location]])
for (sec, value) in zip(cell_sections, split_value):
dec_cell[CELL_MUT_MAPPING[sec]] = value
@staticmethod
def _encrypt(plaintext, key):
return plaintext
@staticmethod
def _decrypt(ciphertext, key):
return ciphertext
@staticmethod
def encrypt(plaintext, key):
return plaintext
@staticmethod
def decrypt(ciphertext, key):
return ciphertext
pace/encryption/abstract_encrypt_test.py
+236
View File
@@ -0,0 +1,236 @@
## **************
## Copyright 2014 MIT Lincoln Laboratory
## Project: PACE
## Authors: ATLH
## Description: Unit tests for cell code
## Modifications:
## Date Name Modification
## ---- ---- ------------
## 7 Jan 2014 ATLH Original file
## **************
import os
import sys
this_dir = os.path.dirname(os.path.dirname(__file__))
base_dir = os.path.join(this_dir, '../..')
sys.path.append(base_dir)
import random
import string
import logging
import unittest
import ConfigParser
import StringIO as stringio
from nose.tools import eq_, assert_raises, assert_equal, assert_not_equal, assert_true
from pyaccumulo import Mutation, Cell
from pace.encryption.encryption_pki import DummyEncryptionPKI
from pace.encryption.acc_encrypt import AccumuloEncrypt
from pace.encryption.enc_mutation import EncMutation, EncCell
from pace.encryption.enc_classes import ALGORITHMS, AES_ALGORITHMS, IV_AES_ALGORITHMS,\
LENGTHBOUND_AES_ALGORITHMS, AUTH_ALGORITHMS, DET_ALGORITHMS
from pace.encryption.encryption_exceptions import DecryptionException
from pace.encryption.AES_encrypt import Pycrypto_AES_Base
def _create_encryptor_dict(config):
'''
Helper function that parses a config file and creates a key
object before creating an encryptor dict that can be passed
into and EncMutation.
'''
config_parser = ConfigParser.ConfigParser()
config_parser.readfp(config)
key_object = DummyEncryptionPKI()
return AccumuloEncrypt._config_to_encryptor(config_parser,
key_object)
def _mutations_equal(mut1, mut2):
return mut1.row == mut2.row and mut2.updates == mut1.updates
def _decrypt_mutations_equal(list_mut, mut2):
'''
list_mut - list of mutations
mut2 - the original mutation
Returns if the list of mutations produced by a plaintext
mutation is the same as the original
'''
equal = True
equal &= len(list_mut) == len(mut2.updates)
for (mut, update) in zip(list_mut, mut2.updates):
equal &= mut.row == mut2.row
equal &= mut.updates[0] == update
return equal
def _check_versioning(encClass):
"""
Test the encryptions classes are properly dealing with versions.
Versions are pulled from the DummyEncryptionPKI in encryption_pki.py
"""
groundtruth = {"Pycrypto_AES_CFB": '3',
"Pycrypto_AES_CBC": '1',
"Pycrypto_AES_OFB": '3',
"Pycrypto_AES_CTR": '1',
"Pycrypto_AES_GCM": '2',
"Pycrypto_AES_CFB": '3',
"Pycrypto_AES_SIV": '1'}
config = stringio.StringIO(
'[value]\n'+\
'key_id = '+ encClass.name +'\n'+\
'cell_sections = value\n'+\
'encryption = ' + encClass.name)
encryptor_dict = _create_encryptor_dict(config)
mut = Mutation('abcdefghijklmnopqrstuvwxyz')
mut.put(val='val2')
enc_muts = EncMutation(mut, encryptor_dict).encrypt()
assert_true(len(enc_muts) == 1)
enc_mut = enc_muts[0]
assert_true(enc_mut.updates[0].value.rsplit('ver',1)[-1] == groundtruth[encClass.name],
'Not grabbing the most recent version of the key')
def _check_malformed_ciphertext_version(encClass):
"""
Tests error handling in the case where the ciphertext does
not contain 'ver'
"""
config = stringio.StringIO(
'[colFamily]\n'+\
'key_id = '+ encClass.name +'\n'+\
'cell_sections = colFamily,colQualifier\n'+\
'encryption = ' + encClass.name)
encryptor_dict = _create_encryptor_dict(config)
mut = Mutation('abcdefghijklmnopqrstuvwxyz')
mut.put(cf='cf1',cq='cq1', cv='a&b', ts = '12345', val = 'val1')
mut.put(cv='c|d',cf='cf2',val='val2')
enc_mut = EncMutation(mut, encryptor_dict)
assert_raises(DecryptionException, enc_mut.decrypt)
def _check_encrypt_decrypt_mutation(encClass):
'''
Tests the encrypt then decrypt functionality of the various algorithms
on mutations
'''
config = stringio.StringIO(
'[colFamily]\n'+\
'key_id = '+ encClass.name +'\n'+\
'cell_sections = colFamily,colQualifier\n'+\
'encryption = ' + encClass.name)
encryptor_dict = _create_encryptor_dict(config)
mut = Mutation('abcdefghijklmnopqrstuvwxyz')
mut.put(cf='cf1',cq='cq1', cv='a&b', ts = '12345', val = 'val1')
mut.put(cf='cf2',cq='cq2', cv='a&b', ts = '12345', val = 'val2')
enc_muts = EncMutation(mut, encryptor_dict).encrypt()
dec_muts = []
for enc_mut in enc_muts:
dec_muts.append(EncMutation(enc_mut, encryptor_dict).decrypt())
assert_true(_decrypt_mutations_equal(dec_muts, mut),
"Mutation is not correctly handled during encryption and decryption process")
assert_true(not _mutations_equal(enc_mut, mut),
"Encryption algorithm was identity function.")
def _check_encrypt_decrypt_cell(encClass):
'''
Tests the encrypt then decryption functionality of the various algorithms
on cells
'''
config = stringio.StringIO(
'[colFamily]\n'+\
'key_id = '+ encClass.name +'\n'+\
'encryption = ' + encClass.name)
encryptor_dict = _create_encryptor_dict(config)
cell = Cell('row','cf','cq','(a&b)|c',1234,'val')
enc_cell = EncCell.encrypt(cell, encryptor_dict)
dec_cell = EncCell.decrypt(enc_cell, encryptor_dict)
eq_(cell, dec_cell,
str(dec_cell) + " was not correctly encrypted and decrypted " +\
"should be " + str(cell))
def _check_ciphertext_length(encClass):
key = b'Sixteen byte key'
ciphertext = ''.join(random.choice(string.ascii_uppercase + string.digits)
for _ in range(25))
assert_raises(DecryptionException, encClass._decrypt, ciphertext, key)
def _check_iv_material(encClass):
key = b'Sixteen byte key'
ciphertext = ''.join(random.choice(string.ascii_uppercase + string.digits)
for _ in range(15))
assert_raises(DecryptionException, encClass._decrypt, ciphertext, key)
def _check_key_length(encClass):
k_short = b'ShortKey'
ciphertext = '1'+''.join(random.choice(string.ascii_uppercase + string.digits)
for _ in range(31))
assert_raises(ValueError, encClass._encrypt, ciphertext, k_short)
def _check_authentication(encClass):
key_one = b"Sixteen Byte keySixteen Byte Key"
key_two = b"sIXTEEN BYTE KEYSixteen Byte Key"
plaintext = "This is the sample plaintext"
ciphertext = encClass._encrypt(plaintext, key_one)
assert_raises(ValueError, encClass._decrypt, ciphertext, key_two)
def _check_determinism(encClass):
key = b"Sixteen Byte keySixteen Byte Key"
plaintext_one = "This is a sample, shall we test?"
plaintext_two = "this is a different sample"
ciphertext_one = encClass._encrypt(plaintext_one, key)
ciphertext_two = encClass._encrypt(plaintext_one, key)
ciphertext_three = encClass._encrypt(plaintext_two, key)
assert_equal(ciphertext_one, ciphertext_two)
assert_not_equal(ciphertext_one, ciphertext_three)
def test_all_encryption_algorithms():
for encClass in ALGORITHMS.values():
yield _check_encrypt_decrypt_mutation, encClass
yield _check_encrypt_decrypt_cell, encClass
def test_aes_encryption_algorithms():
#test error handling aes alogirthms
for encClass in LENGTHBOUND_AES_ALGORITHMS.values():
yield _check_ciphertext_length, encClass
for encClass in IV_AES_ALGORITHMS.values():
yield _check_iv_material, encClass
for encClass in AES_ALGORITHMS.values():
yield _check_key_length, encClass
yield _check_versioning, encClass
yield _check_malformed_ciphertext_version, encClass
def test_authenticated_algorithms():
for encClass in AUTH_ALGORITHMS.values():
yield _check_authentication, encClass
def test_deterministic_algorithms():
for encClass in DET_ALGORITHMS.values():
yield _check_determinism, encClass
def test_aes_padding():
'''
Tests the padding and unpadding functions of AES class
'''
for n in xrange(1,100):
s = ''.join(random.choice(string.ascii_uppercase + string.digits)
for _ in range(n))
padded_string = Pycrypto_AES_Base._pad(s)
eq_(len(padded_string) % 16, 0,
'Padded string %s of length %d is not a multiple of 16.' % (padded_string, len(padded_string)))
eq_(s, Pycrypto_AES_Base._strip_pad(padded_string),
"Padding is not correctly stripped from padded string")
pace/encryption/acc_encrypt.py
+199
View File
@@ -0,0 +1,199 @@
## **************
## Copyright 2014 MIT Lincoln Laboratory
## Project: PACE
## Authors: ATLH
## Description: High level module for encrypting records
## Modifications:
## Date Name Modification
## ---- ---- ------------
## 18 Dec 2014 ATLH Original file
## **************
import os
import sys
this_dir = os.path.dirname(os.path.dirname(__file__))
base_dir = os.path.join(this_dir, '../..')
sys.path.append(base_dir)
import ConfigParser
from StringIO import StringIO
from collections import namedtuple
from pace.encryption.vars import VALID_KEYS
from pace.encryption.enc_classes import ALGORITHMS
from pace.encryption.enc_mutation import EncMutation, EncCell, EncRange
Keytor = namedtuple('Keytor',['key_id','key_object','cell_key_length'])
"""
Key container object, used to encapsulate the information needed to obtain a key for a
particular user and algorithm and/or generate a cell_key if the algorithm is a VIS_*
key_id - the identifier passed into the key object, usually the algorithm name
key_object - a handle on an instance of an EncryptionPKI object
cell_key_length - (int) for VIS_* algorithm, length in bytes
(not bits) of the cell_key to be generated.
"""
Encryptor = namedtuple('Encryptor', ['encryption', 'cell_sections', 'key_container'])
"""
Container object that is used to contain all the information to encrypt a portion of the
cell.
encryption - class instance to be used for that particular portion of the cell
cell_sections - the portions of the cell to encrypt, can be multiple portions
of the cell
key_container - contains a (Keytor) object which is used to extract the necessary
key material
"""
class ConfigurationException(Exception):
""" Exception raised when unable to process configuration file
Attributes:
msg - error message for situation
"""
def __init__(self, msg):
self.msg = msg
def __str__(self):
return self.msg
class AccumuloEncrypt(object):
"""High level interface for encrypting/decrypting accumulo cells.
Parametrized by a configuration file and a separate
key management object.
"""
def __init__(self, config, key_object):
"""
Arguments:
config - configuration file in the format specified
by INI files, see README for more details.
key_object - key management object that meets the
interface outlined in encryption_pki.py
"""
self.config_parser = ConfigParser.ConfigParser()
#support for StringIO for testing
if isinstance(config, StringIO):
self.config_parser.readfp(config)
else:
self.config_parser.read(config)
self.key_object = key_object
self.encrypt_dict = self._config_to_encryptor(self.config_parser,
self.key_object)
@staticmethod
def _config_to_encryptor(config_parser, key_object):
'''
Converts configuration file into associated encryptor
objects for each part of key
'''
for sec in config_parser.sections():
#make sure all sections correspond to part of key
if sec not in VALID_KEYS:
raise ConfigurationException(
"%s is not a valid part of the key" % sec)
#check to see if encryption specified
if not config_parser.has_option(sec, 'encryption'):
raise ConfigurationException(
"No encryption set for %s" % sec)
#make sure identifier information for key_object
if not config_parser.has_option(sec, 'key_id'):
raise ConfigurationException(
"There is no key idenfier information for"+
"key_object")
#make sure algorithm is supported
algorithm = config_parser.get(sec,'encryption')
if algorithm not in ALGORITHMS.keys():
raise ConfigurationException(
"%s is not a supported encryption algorithm" % algorithm)
keys = config_parser.sections()
encryptors = []
for sec in config_parser.sections():
#check if cell_sections exists, if not, just use the current section
if config_parser.has_option(sec,'cell_sections'):
cell_sections = config_parser.get(sec,'cell_sections').split(',')
else:
cell_sections = [sec]
#check if cell_key_length has been specified, if not, use 16 bytes
if config_parser.has_option(sec, 'cell_key_length'):
cell_key_length = config_parser.get(sec,'cell_key_length')
try:
cell_key_length = int(cell_key_length)
except ValueError:
raise ConfigurationException('%s is not an valid integer for key_length' % cell_key_length)
if cell_key_length not in [16,24,32]:
raise ConfigurationException('Key_length must be 16,24,or 32 bytes long')
else:
cell_key_length=16
if not all([cell_sec in VALID_KEYS for cell_sec in cell_sections]):
raise ConfigurationException("At least one of cell_sections ("+\
str(cell_sections)+\
") are not a valid part of a cell")
encryptors.append(Encryptor(ALGORITHMS[config_parser.get(sec,'encryption')],
cell_sections,
Keytor(config_parser.get(sec,'key_id'),
key_object,
cell_key_length)))
if keys == [] or encryptors == []:
raise ConfigurationException("Configuration was not properly parsed")
return dict(zip(keys, encryptors))
def encrypt(self, mutation):
"""
Arugments:
mutation - an plaintext mutation as defined in the
pyaccumulo interface
Returns: A list of new mutations containing the encrypted data
as specified in the configuration file
"""
enc_mut = EncMutation(mutation, self.encrypt_dict)
return enc_mut.encrypt()
def encrypt_search(self, row, columns = None):
'''
Functionality to help users to search over
deterministically encrypted data. Can only
be used to retrieve one row at a time.
Because order is not guarenteed in the supported
deterministic modes
Arguments:
row - the row to search for, this will be passed in as both
the start and end row into 'scan'
columns - double nested list of what columns
to look for, where the first element in the nested list is
the column_family, the second is the column_qualifier.
It is possible to just specify the column_family.
Defaults to None. Example:
[['cf1','cq1'],['cf2']]
Returns: A tuple containing two things:
1) the encrypted row value if is deterministically encrypted
2) A similarly formatted list of columns if they
are deterministically encrypted and the value
passed in is not None. If the columns parameter
is None (as in the default case) None is returned
as the second part of the tuple.
'''
return EncRange.encrypt(row, columns, self.encrypt_dict)
def decrypt(self, cell):
"""
Arugments:
mutation - an cell as defined in pyaccumulo
Returns: New cell containing the decrypted data
as specified in the configuration file
"""
return EncCell.decrypt(cell, self.encrypt_dict)
pace/encryption/acc_encrypt_test.py
+372
View File
@@ -0,0 +1,372 @@
## **************
## Copyright 2014 MIT Lincoln Laboratory
## Project: PACE
## Authors: ATLH
## Description: Unit tests for acc_encrypt
## Modifications:
## Date Name Modification
## ---- ---- ------------
## 23 Dec 2014 ATLH Original file
## **************
import os
import sys
this_dir = os.path.dirname(os.path.dirname(__file__))
base_dir = os.path.join(this_dir, '../..')
sys.path.append(base_dir)
import random
import logging
import unittest
from StringIO import StringIO
from pyaccumulo import Mutation, Cell, Range
from pace.encryption.encryption_exceptions import EncryptionException, DecryptionException
from pace.encryption.acc_encrypt import AccumuloEncrypt, ConfigurationException
from pace.encryption.encryption_pki import DummyEncryptionPKI
from pace.pki.accumulo_keystore import AccumuloKeyStore
from pace.encryption.AES_encrypt import Pycrypto_AES_CFB
from pace.common.fakeconn import FakeConnection
class AccumuloCryptTest(unittest.TestCase):
def setUp(self):
# Keep the same PKI around, since it generates a new RSA key
# for key wraps each time
self.pki = DummyEncryptionPKI()
def test_error_handling(self):
'''
Tests error handling of configuration sections
'''
not_valid_section = StringIO('[row]\n'+\
'key_id = Pycrypto_AES_CFB\n'+\
'encryption = Pycrypto_AES_CFB\n'+\
'[colSection]\n'+\
'key_id = Pycrypto_AES_CFB\n'+\
'encryption = Pycrypto_AES_CFB'
'[colFamily]\n'+\
'key_id = Pycrypto_AES_CFB\n'+\
'encryption = Pycrypto_AES_CFB')
self.assertRaises(ConfigurationException, AccumuloEncrypt, not_valid_section, self.pki)
no_encryption = StringIO('[row]\n'+\
'key_id = Pycrypto_AES_CFB\n'+\
'[colFamily]\n'+\
'key_id = Identity\n'+\
'encryption = Identity')
self.assertRaises(ConfigurationException, AccumuloEncrypt, no_encryption, self.pki)
no_key_id = StringIO('[row]\n'+\
'key_id = Pycrypto_AES_CFB\n'+\
'encryption = Pycrypto_AES_CFB\n'+\
'[colFamily]\n'+\
'encryption = Pycrypto_AES_CFB')
self.assertRaises(ConfigurationException, AccumuloEncrypt, no_key_id, self.pki)
algorithm_not_supported = StringIO('[row]\n'+\
'key_id = Pycrypto_AES_CFB\n'+\
'encryption = Pycrypto_RSA\n'+\
'[colFamily]\n'+\
'key_id = Pycrypto_AES_CFB\n'+\
'encryption = Pycrypto_AES_CFB')
self.assertRaises(ConfigurationException, AccumuloEncrypt, algorithm_not_supported, self.pki)
def test_encryptor_dict(self):
'''
Tests the format of the created encryptor_dict
'''
all_sections = StringIO('[row]\n'+\
'key_id = Pycrypto_AES_CFB\n'+\
'encryption = Pycrypto_AES_CFB\n'+\
'[colQualifier]\n'+\
'key_id = Pycrypto_AES_CFB\n'+\
'encryption = Pycrypto_AES_CFB\n'+\
'[colFamily]\n'+\
'key_id = Pycrypto_AES_CFB\n'+\
'encryption = Pycrypto_AES_CFB\n'+\
'[colVisibility]\n'+\
'key_id = Pycrypto_AES_CFB\n'+\
'encryption = Pycrypto_AES_CFB\n'+\
'[value]\n'+\
'key_id = Pycrypto_AES_CFB\n'+\
'encryption = Pycrypto_AES_CFB')
ac = AccumuloEncrypt(all_sections, self.pki)
encryptor_dict = ac.encrypt_dict
keys = ['row','colQualifier','colFamily','colVisibility',
'value']
for k in keys:
self.assertTrue(k in encryptor_dict, '%s is not in dictionary' % k)
encryptor = encryptor_dict[k]
self.assertEqual(encryptor.encryption, Pycrypto_AES_CFB )
self.assertEqual(encryptor.cell_sections, [k])
def test_with_accumulo_conn(self):
'''
Tests the interplay with a fake accumulo connection
'''
all_sections = '[row]\n'+\
'key_id = table1\n'+\
'encryption = Pycrypto_AES_CFB\n'+\
'[colQualifier]\n'+\
'key_id = table1\n'+\
'encryption = Pycrypto_AES_CFB\n'+\
'[colFamily]\n'+\
'key_id = Pycrypto_AES_CFB\n'+\
'encryption = Pycrypto_AES_CFB\n'+\
'[colVisibility]\n'+\
'key_id = table1\n'+\
'encryption = Pycrypto_AES_CFB\n'+\
'[value]\n'+\
'key_id = Pycrypto_AES_CFB\n'+\
'encryption = Pycrypto_AES_CFB'
#create mutation
mut = Mutation('row1')
mut.put(cf='cf1',cq='cq1', cv='cv1', ts = 12345, val = 'val1')
mut.put(cf='cf2',cq='cq2', cv='', ts = 67890, val = 'val2')
ae = AccumuloEncrypt(StringIO(all_sections), self.pki)
enc_muts = ae.encrypt(mut)
#write mutation along fake connection
conn = FakeConnection()
conn.create_table('enc_test')
conn.write('enc_test', enc_muts[0])
conn.write('enc_test', enc_muts[1])
#create ground truth
conn.create_table('ground')
conn.write('ground', mut)
#retrieve encrypted mutation
dec_cells = []
for c in conn.scan('enc_test'):
dec_cells.append(ae.decrypt(c))
gt_cells = []
for c in conn.scan('ground'):
gt_cells.append(c)
self.assertEqual(sorted(gt_cells), sorted(dec_cells))
def _run_search(self, config, row, cols, correct_cells):
'''
Tests the encrypting search functionality
'''
#create range & mutation to search for
mut1 = Mutation('arow')
mut1.put(cf='cf1',cq='cq1', cv='', ts = 1, val = 'val1')
mut1.put(cf='cf2',cq='cq2', cv='', ts = 2, val = 'val2')
mut1.put(cf='cf1',cq='cq1', cv='', ts = 3, val = 'val3')
mut1.put(cf='cf2',cq='cq3', cv='', ts = 4, val = 'val4')
mut1.put(cf='cf3',cq='cq4', cv='', ts = 5, val = 'val5')
mut2 = Mutation('brow')
mut2.put(cf='cf1',cq='cq1', cv='', ts = 6, val = 'val1')
mut2.put(cf='cf2',cq='cq2', cv='', ts = 7, val = 'val2')
ae = AccumuloEncrypt(StringIO(config), self.pki)
enc_muts1 = ae.encrypt(mut1)
enc_muts2 = ae.encrypt(mut2)
enc_row, enc_cols = ae.encrypt_search(row, cols)
#write mutation along fake connection
conn = FakeConnection()
conn.create_table('enc_test')
for mut in enc_muts1 + enc_muts2:
conn.write('enc_test', mut)
#retrieve encrypted mutation with search
dec_cells = []
for c in conn.scan('enc_test',
scanrange=Range(srow=enc_row, erow=enc_row,
sinclude=True, einclude=True),
cols=enc_cols):
dec_cells.append(ae.decrypt(c))
self.assertEqual(sorted(dec_cells), sorted(correct_cells))
def test_det_row_search(self):
config = '[row]\n'+\
'key_id = Pycrypto_AES_SIV\n'+\
'encryption = Pycrypto_AES_SIV\n'
self._run_search(config,
'brow',
None,
[Cell('brow','cf1','cq1','',6,'val1'),
Cell('brow','cf2','cq2','',7,'val2')])
def test_unencrypted_search(self):
config = '[colFamily]\n'+\
'key_id = Pycrypto_AES_CBC\n'+\
'cell_sections = colFamily\n'+\
'encryption = Pycrypto_AES_CBC\n'
self._run_search(config,
'brow',
None,
[Cell('brow','cf1','cq1','',6,'val1'),
Cell('brow','cf2','cq2','',7,'val2')])
def test_det_row_cf_search(self):
config = '[row]\n'+\
'key_id = Pycrypto_AES_SIV\n'+\
'encryption = Pycrypto_AES_SIV\n'+\
'[colFamily]\n'+\
'key_id = Pycrypto_AES_SIV\n'+\
'cell_sections = colFamily\n'+\
'encryption = Pycrypto_AES_SIV\n'
self._run_search(config,
'brow',
[['cf1']],
[Cell('brow','cf1','cq1','',6,'val1')])
self._run_search(config,
'arow',
[['cf1'],['cf3']],
[Cell('arow','cf1','cq1','',1,'val1'),
Cell('arow','cf1','cq1','',3,'val3'),
Cell('arow','cf3','cq4','',5,'val5')])
def test_det_cf_search(self):
config = '[colFamily]\n'+\
'key_id = Pycrypto_AES_SIV\n'+\
'cell_sections = colFamily\n'+\
'encryption = Pycrypto_AES_SIV\n'
self._run_search(config,
'brow',
[['cf1']],
[Cell('brow','cf1','cq1','',6,'val1')])
self._run_search(config,
'brow',
[['cf1','cq1']],
[Cell('brow','cf1','cq1','',6,'val1')])
self._run_search(config,
'arow',
[['cf1'],['cf3']],
[Cell('arow','cf1','cq1','',1,'val1'),
Cell('arow','cf1','cq1','',3,'val3'),
Cell('arow','cf3','cq4','',5,'val5')])
def test_det_cq_search(self):
config = '[colQualifier]\n'+\
'key_id = Pycrypto_AES_SIV\n'+\
'cell_sections = colQualifier\n'+\
'encryption = Pycrypto_AES_SIV\n'
self._run_search(config,
'brow',
[['cf1','cq1']],
[Cell('brow','cf1','cq1','',6,'val1')])
def test_det_row_cf_cq_search(self):
config = '[row]\n'+\
'key_id = Pycrypto_AES_SIV\n'+\
'encryption = Pycrypto_AES_SIV\n'+\
'[colFamily]\n'+\
'key_id = Pycrypto_AES_SIV\n'+\
'cell_sections = colFamily\n'+\
'encryption = Pycrypto_AES_SIV\n'+\
'[colQualifier]\n'+\
'key_id = Pycrypto_AES_SIV\n'+\
'cell_sections = colQualifier\n'+\
'encryption = Pycrypto_AES_SIV\n'
self._run_search(config,
'brow',
[['cf1','cq1']],
[Cell('brow','cf1','cq1','',6,'val1')])
def test_det_cf_cq_search(self):
config = '[row]\n'+\
'key_id = Pycrypto_AES_SIV\n'+\
'encryption = Pycrypto_AES_SIV\n'+\
'[colFamily]\n'+\
'key_id = Pycrypto_AES_SIV\n'+\
'cell_sections = colFamily,colQualifier\n'+\
'encryption = Pycrypto_AES_SIV\n'
self._run_search(config,
'brow',
[['cf1','cq1']],
[Cell('brow','cf1','cq1','',6,'val1')])
def test_det_cq_cf_search(self):
config = '[row]\n'+\
'key_id = Pycrypto_AES_SIV\n'+\
'encryption = Pycrypto_AES_SIV\n'+\
'[colQualifier]\n'+\
'key_id = Pycrypto_AES_SIV\n'+\
'cell_sections = colFamily,colQualifier\n'+\
'encryption = Pycrypto_AES_SIV\n'
self._run_search(config,
'brow',
[['cf1','cq1']],
[Cell('brow','cf1','cq1','',6,'val1')])
def test_det_cf_cq_switch_search(self):
config = '[colQualifier]\n'+\
'key_id = Pycrypto_AES_SIV\n'+\
'cell_sections = colFamily\n'+\
'encryption = Pycrypto_AES_SIV\n'+\
'[colFamily]\n'+\
'key_id = Pycrypto_AES_SIV\n'+\
'cell_sections = colQualifier\n'+\
'encryption = Pycrypto_AES_SIV\n'
self._run_search(config,
'brow',
[['cf1','cq1']],
[Cell('brow','cf1','cq1','',6,'val1')])
def test_non_det_row(self):
config = '[row]\n'+\
'key_id = Pycrypto_AES_CBC\n'+\
'encryption = Pycrypto_AES_CBC\n'
ae = AccumuloEncrypt(StringIO(config), self.pki)
self.assertRaises(EncryptionException, ae.encrypt_search, 'arow', [['cf1']])
def test_det_non_det_search(self):
config = '[row]\n'+\
'key_id = Pycrypto_AES_SIV\n'+\
'encryption = Pycrypto_AES_SIV\n'+\
'[colFamily]\n'+\
'key_id = Pycrypto_AES_CBC\n'+\
'cell_sections = colFamily\n'+\
'encryption = Pycrypto_AES_CBC\n'+\
'[colQualifier]\n'+\
'key_id = Pycrypto_AES_SIV\n'+\
'cell_sections = colQualifier\n'+\
'encryption = Pycrypto_AES_CBC\n'
self._run_search(config,
'brow',
None,
[Cell('brow','cf1','cq1','',6,'val1'),
Cell('brow','cf2','cq2','',7,'val2')])
pace/encryption/benchmarker.py
+265
View File
@@ -0,0 +1,265 @@
## **************
## Copyright 2015 MIT Lincoln Laboratory
## Project: PACE
## Authors: ATLH
## Description: Benchmark framework for encryption code
## Modifications:
## Date Name Modification
## ---- ---- ------------
## 20 Jan 2015 ATLH Original file - Based on signature benchmarker
## **************
import os
import sys
this_dir = os.path.dirname(os.path.dirname(__file__))
base_dir = os.path.join(this_dir, '../..')
sys.path.append(base_dir)
import random
from pyaccumulo import Accumulo, Mutation, Range
import time, datetime
import string
import cProfile
import StringIO
import pstats
from pace.encryption.acc_encrypt import AccumuloEncrypt
from pace.encryption.enc_classes import ALGORITHMS
from pace.encryption.encryption_pki import DummyEncryptionPKI
from pace.common.common_utils import generate_data, sanitize, Timer
class Benchmarker(object):
"""
Class that contains logic for benchmarking PACE encryption code
"""
def __init__(self,
conn=None,
pki=None,
logger=None):
"""
Arguments:
conn - connection to the Accumulo instance, defaults to localhost
pki - pki object used, default is DummyEncryptionPKI
logger - the output of the benchmarking is logged here
"""
if conn is None:
self.conn = Accumulo(host='localhost',
port=42424,
user='root',
password='secret')
else:
self.conn = conn
if pki is None:
self.pki = DummyEncryptionPKI(conn=self.conn)
else:
self.pki = pki
self.logger = logger
def _process_profile(self, profiler):
buffer_for_results = StringIO.StringIO()
stats = pstats.Stats(profiler, stream=buffer_for_results)
stats.strip_dirs().sort_stats('time').print_stats(10)
msg = buffer_for_results.getvalue()
return msg
def _random_string(self, length=10):
"""
Generates a random string of length lengths
"""
return ''.join(random.choice(string.ascii_uppercase + string.digits) for _ in range(length))
def _generate_data(self, vis, num_entries, num_rows):
"""
Arguments:
vis - Visibility label used for the cells
num_entries - the number of total cells generated
num_rows - the number of rows that num_entries are broken up into
Returns: a list of num_rows mutations with total number of cells
of num_entries
"""
mutations = []
#row size
rsize = num_entries/num_rows
for r in range(num_rows):
mutation = Mutation(self._random_string())
for r in range(rsize):
mutation.put(cf=self._random_string(),
cq=self._random_string(),
cv=vis,
val=self._random_string(20))
mutations.append(mutation)
return mutations
def _write_and_encrypt_data(self,
table,
encryptor,
vis,
encrypt=True,
num_entries=50000,
num_rows=1000,
profile = False):
"""
Arguments:
table - the table to write the data to
encryptor - the encryption object used to encrypt the mutations
vis - the visibility label used in the mutations
encrypt - to encrypt the data or not, if False it represents the baseline
num_entries - the total number of cells generated
num_rows - the number of rows that num_entries that are broken up into
profile - to profile the code or not
Results:
Logs the timing into the output file
"""
#delete table if it exists and recreate it
if self.conn.table_exists(table):
self.conn.delete_table(table)
self.conn.create_table(table)
mutations = self._generate_data(vis, num_entries, num_rows)
wr = self.conn.create_batch_writer(table)
encryption_time = 0
communication_time = 0
profiler_string = ''
if encrypt:
if profile:
pr = cProfile.Profile()
pr.enable()
enc_mutations = []
for m in mutations:
with Timer() as t:
enc_mut = encryptor.encrypt(m)
enc_mutations += enc_mut
encryption_time += t.msecs
if profile:
pr.disable()
profiler_string = self._process_profile(pr)
else:
enc_mutations = mutations
for mut in enc_mutations:
with Timer() as t:
wr.add_mutation(mut)
communication_time += t.msecs
wr.close()
return (encryption_time, communication_time, profiler_string)
def _decrypt_data(self, table, decryptor, decrypt=True, profile=False):
"""
Arguments:
table - the table to read from
decryptor - the decryption object used to encrypt the mutations
decrypt - to decrypt the data or not, if False it represents the baseline
profile - to profile code or not
Results:
Logs the timing into the output file
"""
if not self.conn.table_exists(table):
return (False, 0, 0)
cells = []
with Timer() as t:
for entry in self.conn.scan(table, auths=['a','b','c','d','e']):
cells.append(entry)
communication_time = t.msecs
decryption_time = 0
dec_error = 0
profiler_string = ''
if decrypt:
if profile:
pr = cProfile.Profile()
pr.enable()
for cell in cells:
with Timer() as t:
try:
dec_entry = decryptor.decrypt(entry)
except DecryptionException as ve:
dec_error += 1
decryption_time += t.msecs
if profile:
pr.disable()
profiler_string = self._process_profile(pr)
success = dec_error < 1 and len(cells) > 0
return (success, decryption_time, communication_time, profiler_string)
def run_test(self,
config_file,
table="encrypt_test",
vis='a&b',
num_entries=1000,
num_rows=100,
encrypt=True,
decrypt=True,
profile=False):
"""
Arguments:
config_file - configuration file to be used
table - the table to write the data to
vis - the visibility label used in the mutations
decrypt - to decrypt the data or not, if False it represents the baseline
encrypt - to encrypt the data or not, if False it represents the baseline
num_entries - the total number of cells generated
num_rows - the number of rows that num_entries that are broken up into
profile - whether to profile the code
Results:
Logs the timing into the output file
"""
table = sanitize(table)
self.logger.info("Running with %d cells and %d rows" % (num_entries, num_rows))
if encrypt:
encryptor = AccumuloEncrypt(config_file, self.pki)
(encryption_time, communication_time, profile_string) = self._write_and_encrypt_data(table,
encryptor,
vis,
True,
num_entries,
num_rows,
profile)
self.logger.info('Encryption & communication timing for %s: %d, %d' %
(config_file, encryption_time, communication_time))
if profile:
self.logger.info("Profile for %s is : %s" % (config_file, profile_string))
if decrypt:
decryptor = AccumuloEncrypt(config_file, self.pki)
(sucess, decryption_time, communication_time, profile_string) = self._decrypt_data(table, decryptor, True,
profile)
if sucess:
self.logger.info('Decryption & communication timing for %s: %d, %d' %
(config_file, decryption_time, communication_time))
if profile:
self.logger.info("Profile for %s is : %s" % (config_file, profile_string))
else:
self.logger.info("Was not able to sucessfully decrypt for %s" % (config_file))
if not encrypt and not decrypt:
(encryption_time, communication_time, _) = self._write_and_encrypt_data(table,
None,
vis,
False,
num_entries,
num_rows)
self.logger.info('Communication timing for writing to baseline: %d' % (communication_time))
(sucess, decryption_time, communication_time, _) = self._decrypt_data(table, None, False)
self.logger.info('Communication timing for reading from baseline: %d' % (communication_time))
pace/encryption/config/ceabac/VIS_AES_CBC.cfg
+4
View File
@@ -0,0 +1,4 @@
[value]
key_id = VIS_AES_CBC
encryption = VIS_AES_CBC
cell_sections = value
pace/encryption/config/ceabac/VIS_AES_CFB.cfb
+4
View File
@@ -0,0 +1,4 @@
[value]
key_id = VIS_AES_CFB
encryption = VIS_AES_CFB
cell_sections = value
pace/encryption/config/ceabac/VIS_AES_CTR.cfg
+4
View File
@@ -0,0 +1,4 @@
[value]
key_id = VIS_AES_CTR
encryption = VIS_AES_CTR
cell_sections = value
pace/encryption/config/ceabac/VIS_AES_GCM.cfg
+4
View File
@@ -0,0 +1,4 @@
[value]
key_id = VIS_AES_GCM
encryption = VIS_AES_GCM
cell_sections = value
pace/encryption/config/ceabac/VIS_AES_OFB.cfg
+4
View File
@@ -0,0 +1,4 @@
[value]
key_id = VIS_AES_OFB
encryption = VIS_AES_OFB
cell_sections = value
pace/encryption/config/config_examp.cfg
+13
View File
@@ -0,0 +1,13 @@
[row]
key_id = Pycrypto_AES_GCM
encryption = Pycrypto_AES_GCM
cell_sections = row
[colFamily]
key_id = VIS_AES_CFB
encryption = VIS_AES_CFB
cell_sections = colFamily,colQualifier
[value]
key_id = VIS_AES_CBC
encryption = VIS_AES_CBC
cell_sections = value
cell_key_length=16
pace/encryption/config/different_schemas_ceabac/ceabac_cq_value.cfg
+11
View File
@@ -0,0 +1,11 @@
[colFamily]
key_id= VIS_AES_CBC
encryption = VIS_AES_CBC
cell_sections = colFamily
cell_key_length=16
[value]
key_id = VIS_AES_CBC
encryption = VIS_AES_CBC
cell_sections = value
cell_key_length=16
pace/encryption/config/different_schemas_ceabac/ceabac_row_cf_cq_value.cfg
+21
View File
@@ -0,0 +1,21 @@
[row]
key_id = VIS_AES_CBC
encryption = VIS_AES_CBC
cell_sections = row
cell_key_length=16
[colFamily]
key_id = VIS_AES_CBC
encryption = VIS_AES_CBC
cell_sections = colFamily
cell_key_length=16
[colQualifier]
key_id = VIS_AES_CBC
encryption = VIS_AES_CBC
cell_sections = colQualifier
cell_key_length=16
[value]
key_id = VIS_AES_CBC
encryption = VIS_AES_CBC
cell_sections = value
cell_key_length=16
pace/encryption/config/different_schemas_ceabac/ceabac_row_cfcq_value.cfg
+16
View File
@@ -0,0 +1,16 @@
[row]
key_id = VIS_AES_CBC
encryption = VIS_AES_CBC
cell_sections = row
cell_key_length=16
[colFamily]
key_id = VIS_AES_CBC
encryption = VIS_AES_CBC
cell_sections = colFamily,colQualifier
cell_key_length=16
[value]
key_id = VIS_AES_CBC
encryption = VIS_AES_CBC
cell_sections = value
cell_key_length=16
pace/encryption/config/different_schemas_ceabac/ceabac_row_cfcqvalue.cfg
+11
View File
@@ -0,0 +1,11 @@
[row]
key_id = VIS_AES_CBC
encryption = VIS_AES_CBC
cell_sections = row
cell_key_length=16
[value]
key_id = VIS_AES_CBC
encryption = VIS_AES_CBC
cell_sections = colFamily,colQualifier,value
cell_key_length=16
pace/encryption/config/different_schemas_ceabac/ceabac_row_value.cfg
+10
View File
@@ -0,0 +1,10 @@
[row]
key_id = VIS_AES_CBC
encryption = VIS_AES_CBC
cell_sections = row
cell_key_length=16
[value]
key_id = VIS_AES_CBC
encryption = VIS_AES_CBC
cell_sections = value
cell_key_length=16
pace/encryption/config/different_schemas_ceabac/ceabac_value.cfg
+5
View File
@@ -0,0 +1,5 @@
[value]
key_id = VIS_AES_CBC
encryption = VIS_AES_CBC
cell_sections = value
cell_key_length=16
pace/encryption/config/different_schemas_non_ceabac/cq_value.cfg
+11
View File
@@ -0,0 +1,11 @@
[colFamily]
key_id = Pycrypto_AES_CBC
encryption = Pycrypto_AES_CBC
cell_sections = colFamily
cell_key_length=16
[value]
key_id = Pycrypto_AES_CBC
encryption = Pycrypto_AES_CBC
cell_sections = value
cell_key_length=16
pace/encryption/config/different_schemas_non_ceabac/row_cf_cq_value.cfg
+21
View File
@@ -0,0 +1,21 @@
[row]
key_id = Pycrypto_AES_CBC
encryption = Pycrypto_AES_CBC
cell_sections = row
cell_key_length=16
[colFamily]
key_id = Pycrypto_AES_CBC
encryption = Pycrypto_AES_CBC
cell_sections = colFamily
cell_key_length=16
[colQualifier]
key_id = Pycrypto_AES_CBC
encryption = Pycrypto_AES_CBC
cell_sections = colQualifier
cell_key_length=16
[value]
key_id = Pycrypto_AES_CBC
encryption = Pycrypto_AES_CBC
cell_sections = value
cell_key_length=16
pace/encryption/config/different_schemas_non_ceabac/row_cfcq_value.cfg
+16
View File
@@ -0,0 +1,16 @@
[row]
key_id = Pycrypto_AES_CBC
encryption = Pycrypto_AES_CBC
cell_sections = row
cell_key_length=16
[colFamily]
key_id = Pycrypto_AES_CBC
encryption = Pycrypto_AES_CBC
cell_sections = colFamily,colQualifier
cell_key_length=16
[value]
key_id = Pycrypto_AES_CBC
encryption = Pycrypto_AES_CBC
cell_sections = value
cell_key_length=16
pace/encryption/config/different_schemas_non_ceabac/row_cfcqvalue.cfg
+11
View File
@@ -0,0 +1,11 @@
[row]
key_id = Pycrypto_AES_CBC
encryption = Pycrypto_AES_CBC
cell_sections = row
cell_key_length=16
[value]
key_id = Pycrypto_AES_CBC
encryption = Pycrypto_AES_CBC
cell_sections = colFamily,colQualifier,value
cell_key_length=16
pace/encryption/config/different_schemas_non_ceabac/row_value.cfg
+10
View File
@@ -0,0 +1,10 @@
[row]
key_id = Pycrypto_AES_CBC
encryption = Pycrypto_AES_CBC
cell_sections = row
cell_key_length=16
[value]
key_id = Pycrypto_AES_CBC
encryption = Pycrypto_AES_CBC
cell_sections = value
cell_key_length=16
pace/encryption/config/different_schemas_non_ceabac/value.cfg
+5
View File
@@ -0,0 +1,5 @@
[value]
key_id = Pycrypto_AES_CBC
encryption = Pycrypto_AES_CBC
cell_sections = value
cell_key_length=16
pace/encryption/config/mixed_schemas/SSE_auth_VIS.cfg
+13
View File
@@ -0,0 +1,13 @@
[row]
key_id = Pycrypto_AES_CBC
encryption = Pycrypto_AES_CBC
cell_sections = row
[colFamily]
key_id = Pycrypto_AES_GCM
encryption = Pycrypto_AES_GCM
cell_sections = colFamily,colQualifier
[value]
key_id = VIS_AES_CBC
encryption = VIS_AES_CBC
cell_sections = value
cell_key_length=16
pace/encryption/config/mixed_schemas/det_SSE_VIS.cfg
+13
View File
@@ -0,0 +1,13 @@
[row]
key_id = Pycrypto_AES_SIV
encryption = Pycrypto_AES_SIV
cell_sections = row
[colFamily]
key_id = Pycrypto_AES_GCM
encryption = Pycrypto_AES_GCM
cell_sections = colFamily,colQualifier
[value]
key_id = VIS_AES_CBC
encryption = VIS_AES_CBC
cell_sections = value
cell_key_length=16
pace/encryption/config/mixed_schemas/det_sse_auth.cfg
+13
View File
@@ -0,0 +1,13 @@
[row]
key_id = Pycrypto_AES_SIV
encryption = Pycrypto_AES_SIV
cell_sections = row
[colFamily]
key_id = Pycrypto_AES_CFB
encryption = Pycrypto_AES_CFB
cell_sections = colFamily,colQualifier
[value]
key_id = Pycrypto_AES_GCM
encryption = Pycrypto_AES_GCM
cell_sections = value
cell_key_length=16
pace/encryption/config/non-ceabac/AES_CBC.cfg
+4
View File
@@ -0,0 +1,4 @@
[value]
key_id = Pycrypto_AES_CFB
encryption = Pycrypto_AES_CFB
cell_sections = value
pace/encryption/config/non-ceabac/AES_CFB.cfg
+4
View File
@@ -0,0 +1,4 @@
[value]
key_id = Pycrypto_AES_CFB
encryption = Pycrypto_AES_CFB
cell_sections = value
pace/encryption/config/non-ceabac/AES_CTR.cfg
+4
View File
@@ -0,0 +1,4 @@
[value]
key_id = Pycrypto_AES_OFB
encryption = Pycrypto_AES_OFB
cell_sections = value
pace/encryption/config/non-ceabac/AES_GCM.cfg
+4
View File
@@ -0,0 +1,4 @@
[value]
key_id = Pycrypto_AES_GCM
encryption = Pycrypto_AES_GCM
cell_sections = value
pace/encryption/config/non-ceabac/AES_OFB.cfg
+4
View File
@@ -0,0 +1,4 @@
[value]
key_id = Pycrypto_AES_CTR
encryption = Pycrypto_AES_CTR
cell_sections = value
pace/encryption/config/non-ceabac/AES_SIV.cfg
+4
View File
@@ -0,0 +1,4 @@
[value]
key_id = Pycrypto_AES_SIV
encryption = Pycrypto_AES_SIV
cell_sections = value
pace/encryption/demo.py
+342
View File
@@ -0,0 +1,342 @@
## **************
## Copyright 2015 MIT Lincoln Laboratory
## Project: PACE
## Authors: ATLH
## Description: Framework for demoing the encryption code
## Modifications:
## Date Name Modification
## ---- ---- ------------
## 09 Apr 2015 ATLH Original file
## **************
import os
import sys
this_dir = os.path.dirname(os.path.dirname(__file__))
base_dir = os.path.join(this_dir, '../..')
sys.path.append(base_dir)
import pyaccumulo
from StringIO import StringIO
import base64
from Crypto.PublicKey import RSA
import subprocess
from pace.encryption.acc_encrypt import AccumuloEncrypt, Encryptor
from pace.encryption.encryption_pki import DummyEncryptionPKI, EncryptionPKIAccumulo
from pace.encryption.AES_encrypt import Pycrypto_AES_CFB
from pace.encryption.encryption_exceptions import DecryptionException
from pace.pki.keygen import KeyGen
from pace.pki.accumulo_keystore import AccumuloKeyStore, AccumuloAttrKeyStore
from pace.pki.frontend import KeyStoreFrontEnd
ABS_PATH = os.path.dirname(__file__)
DEMO_USERS = ABS_PATH + '../pki/user_info.cfg'
PRIV_KEYS = ABS_PATH + '../pki/'
# Default elements to use
expressive_elems = [('MITLL', '(a&b)|c','Analytics','Rooms','1200'),
('MITLL', '(a&b)|c','Analytics','Employees','4000'),
('Boston', '(a&b)|c','Analytics','Population','3000000'),
('Cambridge', '(a&b)|c','Analytics','Population','400000')]
#Tuples of accumulo elements: row, vis, colFam, colQual, and value
vis_elems = [('Analytics', 'a','','','120'),
('Average', 'a&b','','','200'),
('Median', '(a&b)|c','','','160'),
('Total', '(a&b)|(c&d)|(b&d)|(b&c)','','','320')]
new_vis_elems = [('Sum', 'c&d','','','1200'),
('Minimum', 'b','','','120')]
empty_vis = [('Sum', '','','','1200')]
descriptions = {'expressive' : "Row contains: Row encrypted with Pycrypto_AES_SIV\n"+\
"ColFamily contains: ColFamily & ColQualifier encrypted with Pycrypto_AES_CFB\n"+\
"ColQualifier contains: Nothing\n"+\
"Value contains: Value encrypted with AES_GCM\n",
'identity' : "Row contains: Row unecrypted\n"+\
"ColFamily contains: ColFamily & ColQualifier concatentated\n"+\
"ColQualifier contains: Nothing\n"+\
"Value contains: Value unencrypted \n",
'vis' : "Value contains: Value encrypted with VIS_AES_CFB",
'print' : "",
'vis_print':'',
'none': "Currently using no encryption scheme"}
schema = { 'expressive' : '[row]\n'+\
'key_id = Pycrypto_AES_SIV\n'+\
'encryption = Pycrypto_AES_SIV\n'+\
'[colFamily]\n'+\
'key_id = Pycrypto_AES_CFB\n'+\
'cell_sections = colFamily,colQualifier\n'+\
'encryption = Pycrypto_AES_CFB\n'+\
'[value]\n'+\
'key_id = Pycrypto_AES_CFB\n'+\
'encryption = Pycrypto_AES_GCM',
'identity': '[row]\n'+\
'key_id = Identity\n'+\
'encryption = Identity\n'+\
'[colFamily]\n'+\
'key_id = Identity\n'+\
'cell_sections = colFamily,colQualifier\n'+\
'encryption = Identity\n'+\
'[value]\n'+\
'key_id = Identity\n'+\
'encryption = Identity',
'vis' : '[value]\n'+\
'key_id = VIS_AES_CFB\n'+\
'cell_key_length = 32\n'+\
'encryption = VIS_AES_CFB',
'none': '[row]\n'+\
'key_id = Identity\n'+\
'encryption = Identity\n',
'print':'[row]\n'+\
'key_id = Identity\n'+\
'encryption = Identity\n',
'vis_print':'[row]\n'+\
'key_id = Identity\n'+\
'encryption = Identity\n'}
def mutation_from_kv_tuple(tup):
""" Define a pyaccumulo mutation from a (key, value) formatted tuple.
"""
row, cv, cf, cq, val = tup
print 'Inserting entry: \n Row - %s,\n Column_Visibility - %s,\n Column_Family - %s,\n Column_Qualifier - %s,\n Value - %s' % (row,cv,cf,cq,val)
m = pyaccumulo.Mutation(row)
m.put(cf=cf, cq=cq, cv=cv, val=val)
return m
def grant_auths(table='demo', user='user3'):
"""
Grants read/write priviledges for the user
for the necessary tables in the demo, also sets
user's attributes to a,c,d,e
"""
print "Setting ", user, "'s attributes to a,c,d,e"
_ = subprocess.call("/usr/local/accumulo-1.7.0/bin/accumulo shell -u root -p secret -e 'setauths -u "+user+" -s a,c,d,e'", stdout=subprocess.PIPE, shell=True)
tables = ['__VERSION_METADATA__','__KEYWRAP_METADATA__',"VIS_AES_CFB"]
tables.append(table)
print "Granting read/write access to tables:"
for tab in tables:
print " Table ", tab
_ = subprocess.call("/usr/local/accumulo-1.7.0/bin/accumulo shell -u root -p secret -e 'grant Table.READ -t "+tab+" -u "+user+"'", stdout=subprocess.PIPE, shell=True)
_ = subprocess.call("/usr/local/accumulo-1.7.0/bin/accumulo shell -u root -p secret -e 'grant Table.WRITE -t "+tab+" -u "+user+"'", stdout=subprocess.PIPE, shell=True)
def write_list_to_table(conn, encrypter, table, data):
""" Write a list to an Accumulo table.
Arguments:
conn - a pyaccumulo connection to an Accumulo instance
encrypter - an AccumuloEncrypt instance
table - the name of the table to write to
data - the list to write to the table, formatted as a list of
(key, cf, cq, value) tuples
"""
if not conn.table_exists(table):
conn.create_table(table)
wr = conn.create_batch_writer(table)
for tup in data:
m = mutation_from_kv_tuple(tup)
enc_mut = encrypter.encrypt(m)
for mut in enc_mut:
wr.add_mutation(mut)
wr.close()
def run_insert(conn=None,
data=expressive_elems,
table='demo',
config_name = 'expressive',
PKI_object=None):
""" Insert a list into the accumulo table specified.
Arguments:
conn - the Accumulo connection to use
data - the list to insert into the Accumulo table
table - the name of the table to insert to
default_vis - the default visibility label to use. default: '(a&b)|c'
config_filept - file pointer to the configuration file for encryption
PKI_object - matches the interface on encryption PKI OBJECT, default is
DummyEncryptionPKI
"""
if conn is None:
conn = pyaccumulo.Accumulo(host='localhost',
port=42424,
user='root',
password='secret')
if PKI_object is None:
PKI_object = DummyEncryptionPKI(conn=conn)
config_filept = StringIO(schema[config_name])
encrypter = AccumuloEncrypt(config_filept, PKI_object)
print "\nEnrypting with the following schema: \n\n" + descriptions[config_name]
write_list_to_table(conn, encrypter, table, data)
def run_retrieve(conn=None,
table='demo',
config_name = 'expressive',
PKI_object=None):
""" Retrieves and decrypts values from the specified table,
outputting the appropriate error messages if not.
Arguments:
conn - the Accumulo connection to use
data - the list to insert into the Accumulo table
table - the name of the table to insert to
config_filept - file pointer to the configuration file for encryption
PKI_object - matches the interface on encryption PKI OBJECT, default is
DummyEncryptionPKI
"""
if conn is None:
conn = pyaccumulo.Accumulo(host='localhost',
port=42424,
user='root',
password='secret')
if PKI_object is None:
PKI_object = DummyEncryptionPKI(conn=conn)
total = 0
config_filept = StringIO(schema[config_name])
decrypter = AccumuloEncrypt(config_filept, PKI_object)
for entry in conn.scan(table):
total = total + 1
try:
cell = decrypter.decrypt(entry)
if config_name == 'print':
print 'Entry: \n Row - %s,\n Column_Visibility - %s,\n Column_Family - %s,\n Column_Qualifier - %s,\n Value - %s\n' %(base64.b64encode(cell.row),
cell.cv,
base64.b64encode(cell.cf),
base64.b64encode(cell.cq),
base64.b64encode(cell.val))
elif config_name == 'vis_print':
print 'Entry: \n Row - %s,\n Column_Visibility - %s,\n Column_Family - %s,\n Column_Qualifier - %s,\n Value - %s\n' %(cell.row,
cell.cv,
cell.cf,
cell.cq,
base64.b64encode(cell.val))
else:
print 'Entry: \n Row - %s,\n Column_Visibility - %s,\n Column_Family - %s,\n Column_Qualifier - %s,\n Value - %s\n' % (cell.row, cell.cv,cell.cf,cell.cq,cell.val)
except DecryptionException as ve:
print 'Error: Entry failed to decrypt.'
print 'Error message:', ve.msg
print
print 'Finished, decrypted %d total entries.' %(total)
def run_search(conn=None,
table='demo',
config_name = 'expressive',
PKI_object=None,
row_range='Analytics'):
""" Retrieves and decrypts values from the specified table,
outputting the appropriate error messages if not.
Arguments:
conn - the Accumulo connection to use
data - the list to insert into the Accumulo table
table - the name of the table to insert to
config_filept - file pointer to the configuration file for encryption
PKI_object - matches the interface on encryption PKI OBJECT, default is
DummyEncryptionPKI
row_range - the keyword to search for
"""
if conn is None:
conn = pyaccumulo.Accumulo(host='localhost',
port=42424,
user='root',
password='secret')
if PKI_object is None:
PKI_object = DummyEncryptionPKI(conn=conn)
total = 0
enc_config_filept = StringIO(schema[config_name])
dec_config_filept = StringIO(schema[config_name])
encrypter = AccumuloEncrypt(enc_config_filept, PKI_object)
enc_row, _ = encrypter.encrypt_search(row_range, None)
range = pyaccumulo.Range(srow = enc_row, sinclude = True,
erow = enc_row, einclude = True)
for entry in conn.scan(table, scanrange=range):
total = total + 1
try:
cell = encrypter.decrypt(entry)
print "Entry: (%s, %s, %s, %s)" % (cell.row, cell.cf, cell.cq, cell.val)
except DecryptionException as ve:
print 'Error: Entry failed to decrypt.'
print 'Error message:', ve.msg
print
print 'Finished, decrypted %d total entries.' %(total)
def run_pki_insert(genpath=DEMO_USERS, user='user1', conn=None):
""" Insert elements into the PKI as specified by the config
file located at genpath.
Returns:
ks - the keystore object initialized this way
"""
if conn is None:
print 'Logging in to Accumulo...'
conn = pyaccumulo.Accumulo(host='localhost', port=42424,
user='root', password='secret')
print 'Logged in. Creating key store...'
print 'Deleting existing metadata tables'
for table_name in ['__VERSION_METADATA__','__ATTR_TABLE__','__KEYWRAP_METADATA__']:
if conn.table_exists(table_name):
print 'Deleting table ', table_name
conn.delete_table(table_name)
ks = AccumuloAttrKeyStore(conn)
print 'Key store created.'
print
master_secret = 'master secret key'
print 'Initializing key store with dummy master secret "master secret key"...'
kg = KeyGen(master_secret)
users = KeyGen.file_to_dict(genpath)
for _, infos in users.itervalues():
for _, _, metadata, _ in infos:
if conn.table_exists(metadata):
print ' Found existing table:', metadata
print ' Deleting to have a clean slate.'
conn.delete_table(metadata)
kg.initialize_users(users, ks)
#get private keys to unwrap things
f = open(PRIV_KEYS+user+'_privkey.pem','r')
priv_key = RSA.importKey(f.read())
print 'Key store initialized.'
return EncryptionPKIAccumulo(conn, user, priv_key), KeyStoreFrontEnd(ks)
def run_revoke(user, attr, genpath=DEMO_USERS, conn=None):
if conn is None:
conn = pyaccumulo.Accumulo(host='localhost', port=42424,
user='root', password='secret')
ks = AccumuloAttrKeyStore(conn)
master_secret = 'master secret key'
kg = KeyGen(master_secret)
users = KeyGen.file_to_dict(genpath)
user_keys = {}
for k_user, k_tuple in users.iteritems():
user_keys[k_user] = k_tuple[0]
print "Revoking user %s's access to attribute %s." %(user, attr)
kg.revoke(user, attr, ks, ks, ks, user_keys)
pace/encryption/enc_classes.py
+63
View File
@@ -0,0 +1,63 @@
## **************
## Copyright 2014 MIT Lincoln Laboratory
## Project: PACE
## Authors: ATLH
## Description: All the different types of algorithms
## Modifications:
## Date Name Modification
## ---- ---- ------------
## 19 Dec 2014 ATLH Original file
## **************
import os
import sys
this_dir = os.path.dirname(os.path.dirname(__file__))
base_dir = os.path.join(this_dir, '../..')
sys.path.append(base_dir)
from pace.encryption.AES_encrypt import Pycrypto_AES_CTR, Pycrypto_AES_OFB, \
Pycrypto_AES_CFB, Pycrypto_AES_CBC, Pycrypto_AES_GCM, Pycrypto_AES_SIV
from pace.encryption.visibility.vis_encrypt import VIS_AES_CTR, VIS_AES_OFB, \
VIS_AES_CFB, VIS_AES_CBC, VIS_AES_GCM, VIS_Identity
from pace.encryption.abstract_encrypt import Identity_AccEncrypt
ALGORITHMS = {"Identity" : Identity_AccEncrypt,
"Pycrypto_AES_CFB" : Pycrypto_AES_CFB,
"Pycrypto_AES_CBC" : Pycrypto_AES_CBC,
"Pycrypto_AES_OFB" : Pycrypto_AES_OFB,
"Pycrypto_AES_CTR" : Pycrypto_AES_CTR,
"Pycrypto_AES_GCM" : Pycrypto_AES_GCM,
"Pycrypto_AES_SIV" : Pycrypto_AES_SIV,
"VIS_Identity" : VIS_Identity,
"VIS_AES_CFB" : VIS_AES_CFB,
"VIS_AES_CBC" : VIS_AES_CBC,
"VIS_AES_OFB" : VIS_AES_OFB,
"VIS_AES_CTR" : VIS_AES_CTR,
"VIS_AES_GCM" : VIS_AES_GCM}
AES_ALGORITHMS = {"Pycrypto_AES_CFB" : Pycrypto_AES_CFB,
"Pycrypto_AES_CBC" : Pycrypto_AES_CBC,
"Pycrypto_AES_OFB" : Pycrypto_AES_OFB,
"Pycrypto_AES_CTR" : Pycrypto_AES_CTR,
"Pycrypto_AES_GCM" : Pycrypto_AES_GCM,
"Pycrypto_AES_SIV" : Pycrypto_AES_SIV}
#modes of operation that require plaintext to be a multiple of a block length
LENGTHBOUND_AES_ALGORITHMS = {"Pycrypto_AES_CBC" : Pycrypto_AES_CBC}
IV_AES_ALGORITHMS = { "Pycrypto_AES_CFB" : Pycrypto_AES_CFB,
"Pycrypto_AES_CBC" : Pycrypto_AES_CBC,
"Pycrypto_AES_OFB" : Pycrypto_AES_OFB,
"Pycrypto_AES_GCM" : Pycrypto_AES_GCM}
AUTH_ALGORITHMS = {"Pycrypto_AES_GCM" : Pycrypto_AES_GCM}
DET_ALGORITHMS = {"Pycrypto_AES_SIV" : Pycrypto_AES_SIV}
VIS_ALGORITHMS = {"VIS_Identity" : VIS_Identity,
"VIS_AES_CFB" : VIS_AES_CFB,
"VIS_AES_CBC" : VIS_AES_CBC,
"VIS_AES_OFB" : VIS_AES_OFB,
"VIS_AES_CTR" : VIS_AES_CTR,
"VIS_AES_GCM" : VIS_AES_GCM}
pace/encryption/enc_mutation.py
+424
View File
@@ -0,0 +1,424 @@
## **************
## Copyright 2014 MIT Lincoln Laboratory
## Project: PACE
## Authors: ATLH
## Description: Definition of encrypted mutation and cells
## Modifications:
## Date Name Modification
## ---- ---- ------------
## 19 Dec 2014 ATLH Original file
## 30 Dec 2014 ATLH Added EncCell
## **************
import os
import sys
this_dir = os.path.dirname(os.path.dirname(__file__))
base_dir = os.path.join(this_dir, '../..')
sys.path.append(base_dir)
import itertools
import copy
from collections import defaultdict
from pyaccumulo import Mutation, Cell, Range
from operator import methodcaller
from pace.encryption.encryption_exceptions import EncryptionException, DecryptionException
from pace.encryption.vars import DELIN_CHAR, CELL_MUT_MAPPING, DET_ALGORITHMS, VALID_KEYS, CELL_ORDER
class EncMutation(Mutation):
'''
Contains the data structures and logic for
applying encryptors to a mutation
'''
def __init__(self, mut, encryptor_dict):
'''
Arguments
mut - mutation as defined in pyaccumulo
interface
encryptor_dict - dictionary keyed by cell location
containing a tuple of the encryptor class and
key_id used for encryption or decryption. Also
contains the key object for retrieving the key
'''
self._encrypted = False
self._num_updates = len(mut.updates)
self.encryptor_dict = encryptor_dict
#lists of updates grouped by cell location
self.update_dict = {}
self.update_dict['row'] = [mut.row] * len(mut.updates)
self.update_dict['colFamily'] = [u.colFamily for u in mut.updates]
self.update_dict['colQualifier'] = [u.colQualifier for u in mut.updates]
self.update_dict['colVisibility'] = [u.colVisibility for u in mut.updates]
self.update_dict['timestamp'] = [u.timestamp for u in mut.updates]
self.update_dict['value'] = [u.value for u in mut.updates]
self.update_dict['deleteCell'] = [u.deleteCell for u in mut.updates]
#methods to allow object to be treated as dict
def __getitem__(self, key):
return self.update_dict[key]
def __setitem__(self, key, value):
self.update_dict[key] = value
def __iter__(self):
for list in [self.update_dict['row'],
self.update_dict['colFamily'],
self.update_dict['colQualifier'],
self.update_dict['colVisibility'],
self.update_dict['timestamp'],
self.update_dict['value'],
self.update_dict['deleteCell']]:
yield list
def __eq__(self, other):
if isinstance(other, self.__class__):
return self.update_dict == other.update_dict
else:
return False
def __ne__(self, other):
return not self.__eq__(other)
@staticmethod
def concatenate_cell_section_values(mutation, cell_sections):
'''
Arguments:
mutation - Mutation which one wants to extract
the list of all the same cell locations
cell_sections - the location(s) that need to be
extracted from the mutation
Returns: a list of all of the cell locations
for that mutation (taking into account all of
the cell's updates). If there is more then one
location, it is a list of the value concatenated
by the DELIN_CHAR character.
For example:
mut = Mutation('abcd')
mut.put(cf='cf1',cq='cq1', cv='cv1', ts = '12345', val = 'val1')
mut.put(cf='cf2',val='val2')
and
cell_sections = [colFamily, value]
this function would return:
['cf1|val1','cf2|val2']
'''
sections = [mutation[sec] for sec in cell_sections]
return [DELIN_CHAR.join(map(str, u)) for u in zip(*sections)]
@staticmethod
def split_values(values):
'''
Arguments:
values - A list of values constructed by
get_values_by_cell_string in which individual entries
are made of cell values concatenated by the DELIN_CHAR
Returns: A list of tuples, the first list consisting of
the first values marked by the DELIN_CHAR, the second list
the second values in the entries, and so on. For example
the list ['a|b|c', "1|2|3"] would return [(a,1), (b, 2),
(c, 3)].
'''
return zip(*map(methodcaller('split', DELIN_CHAR), values))
def _remove_unencrypted_cell_sections(self, updates):
"""
Compares the list of targeted locations (where encrypted
values are stored) to the locations that are actually
encrypted as part of subsets. If there are any locations
that were included in part of a subset that weren't over
written with encrypted data, it removes the original,
unencrypted data, and replaces it with a list of empty
strings as long as the original list of updates.
"""
source_locations = set()
target_locations = set()
for (cell_string, encryptor) in self.encryptor_dict.items():
source_locations.update(encryptor.cell_sections)
target_locations.add(cell_string)
for sec in (source_locations - target_locations):
updates[sec] = list(itertools.repeat("", self._num_updates))
return updates
def _get_key(self, key_id):
"""
Helper function that acquires the key from the key object for
encryption/decryption. Raises PKI Lookup error if the key_id
does not return a key for this particular user
"""
return self._key_object.get_key(key_id)
def encrypt(self):
'''
Returns a list of new mutations. Each portion of the cell that
has an associated encryptor is encrypted.
'''
#only want to encrypt the values once
if not self._encrypted:
self._encrypted = True
enc_updates = self.update_dict.copy()
for (cell_string, encryptor) in self.encryptor_dict.items():
enc_updates[cell_string] = encryptor.encryption.encrypt_mutation(self,
encryptor.key_container,
encryptor.cell_sections)
self.update_dict = self._remove_unencrypted_cell_sections(enc_updates)
#TODO: in the case where the row is deterministically encrypted
# update to only produce one mutation
muts = []
for (row, cf,cq,cv,ts,v,dc) in zip(*self):
mut = Mutation(row)
mut.put(cf,cq,cv,ts,v,dc)
muts.append(mut)
return muts
def decrypt(self):
'''
Returns a new mutation. Each portion of the cell that
has an associated encryptor is decrypted. Mostly
used for testing.
'''
dec_updates = self.update_dict.copy()
for (cell_string, encryptor) in self.encryptor_dict.items():
encryptor.encryption.decrypt_mutation(self,
dec_updates,
encryptor.key_container,
cell_string,
encryptor.cell_sections)
self.update_dict = dec_updates
#only should be one cell since each encrypted mutation only contains one cell
assert len(self.update_dict['row']) == 1
mut = Mutation(self.update_dict['row'][0])
for (row, cf,cq,cv,ts,v,dc) in zip(*self):
mut.put(cf,cq,cv,ts,v,dc)
return mut
#create a default empty dictionary with the empty string as default
#used in EncRange
def constant_empty():
return itertools.repeat('').next
class EncRange(Range):
'''
Contains the logic for encrypting and decrypting a range object only
in the case where the algorithm is deterministic, otherwise does not
change the contents of the range
'''
@staticmethod
def get_value_by_cell_string(cols, cell_sections):
'''
Arguments:
cols - col dictionary which one wants to extract
the list of all the same cell locations
cell_sections - the location(s) that need to be
extracted from the dict
Returns: a string of all of the cell_sections
for the range concatenated by the DELIN_CHAR
'''
return DELIN_CHAR.join([str(cols[sec]) for sec in cell_sections])
@staticmethod
def split_value_by_cell_string(value):
return value.split(DELIN_CHAR)
@staticmethod
def _remove_unencrypted_cell_sections(encryptor_dict, col_dict):
"""
Compares the list of targeted locations (where encrypted
values are stored) to the locations that are actually
encrypted as part of subsets. If there are any locations
that were included in part of a subset that weren't over
written with encrypted data, it removes the original,
unencrypted data, and replaces it with a list of empty
strings as long as the original list of updates.
"""
source_locations = set()
target_locations = set()
for (cell_string, encryptor) in encryptor_dict.items():
source_locations.update(encryptor.cell_sections)
target_locations.add(cell_string)
for sec in (source_locations - target_locations):
col_dict[sec] = ''
@staticmethod
def _valid_configuration(encryptor_dict):
"""
Arguments:
encryptor_dict - dictionary keyed by cell location
containing a tuple of the encryptor class and also
contains the key object for retrieving the key
Returns: True/False whether the configuration is valid
for searching. For example, in the case where
non-deterministic encryption appears
in the ordering before either deterministic or no encryption
an false is returned. For example:
Row: no encryption
ColFamily: Deterministic
ColQualifier: CTR_MODE
would return true, but:
Row: CTR_MODE
ColFamily: Deterministc
ColQualifier: No encryption
would.
"""
det_encryptions = 0
for cell_string in VALID_KEYS:
#case of no encryption
try:
encryptor = encryptor_dict[cell_string]
except KeyError:
det_encryptions += 1
continue
#case of non-deterministic encryption
if encryptor.encryption.name not in DET_ALGORITHMS:
break
det_encryptions +=1
return det_encryptions != 0
@staticmethod
def encrypt(row, cols, encryptor_dict):
'''
Arguments:
row - the row to search for, this will be passed in as both
the start and end row into 'scan'
cols - double nested list of what columns
to look for, where the first element in the
column_family, the second is the column_qualifier.
Defaults to None. Example:
[['cf1','cq1'],['cf2','cq2']]
encryptor_dict - dictionary keyed by cell location
containing a tuple of the encryptor class and also
contains the key object for retrieving the key
Returns: (encrypted_start_row, encrypt_end_row, encrypted_col_list)
as per the the configuration specifies. If it does not specify
any of the above values, they are left unchanged.
In the case where non-deterministic encryption appears
in the ordering before either deterministic or no encryption
an error is raised. For example:
Row: no encryption
ColFamily: Deterministic
ColQualifier: CTR_MODE
would not raise an error, but:
Row: CTR_MODE
ColFamily: Deterministc
ColQualifier: No encryption
would.
'''
if not EncRange._valid_configuration(encryptor_dict):
raise EncryptionException('Cannot encrypt a range object in which the configuration '+\
'has the leading portion of the key not either unencrypted ' +\
'or deterministically encrypted.')
#check to see if the row is being encrypted
if encryptor_dict.has_key('row'):
encryptor = encryptor_dict['row']
row = encryptor.encryption.encrypt_row(row, encryptor.key_container)
enc_cols = []
if cols == None:
return (row, None)
for col in cols:
col_dict = defaultdict(constant_empty())
for (key, value) in zip(['colFamily','colQualifier'], col):
col_dict[key]=value
enc_col = col_dict.copy()
for cell_string in ['colFamily', 'colQualifier']:
try:
encryptor = encryptor_dict[cell_string]
except KeyError:
continue
enc_col[cell_string] = encryptor.encryption.encrypt_cols(col_dict,
encryptor.key_container,
encryptor.cell_sections)
#convert back into a list
EncRange._remove_unencrypted_cell_sections(encryptor_dict, enc_col)
if not enc_col.has_key('colQualifier'):
enc_cols.append([enc_col['colFamily']])
else:
enc_cols.append([enc_col['colFamily'], enc_col['colQualifier']])
return (row, enc_cols)
class EncCell(Cell):
'''
Contains the logic for encrypting and decrypting a single cell
'''
@staticmethod
def get_value_by_cell_string(cell, cell_sections):
return DELIN_CHAR.join([str(cell[CELL_MUT_MAPPING[sec]]) for sec in cell_sections])
@staticmethod
def split_value_by_cell_string(value):
return value.split(DELIN_CHAR)
@staticmethod
def encrypt(cell, encryptor_dict):
'''
Arguments:
cell - cell to be encrypted
encryptor_dict - dictionary keyed by cell location
containing a tuple of the encryptor class and also
contains the key object for retrieving the key
Returns new cell with all locations in cell encrypted
that are noted in encryptor_dict
'''
cell_dict = cell._asdict()
enc_cell = cell_dict.copy()
for (cell_string, encryptor) in encryptor_dict.items():
enc_cell[CELL_MUT_MAPPING[cell_string]] = encryptor.encryption.encrypt_cell(cell_dict,
encryptor.key_container,
encryptor.cell_sections)
return Cell(*[enc_cell[field] for field in CELL_ORDER])
@staticmethod
def decrypt(cell, encryptor_dict):
'''
Arguments:
cell - cell to be encrypted
encryptor_dict - dictionary keyed by cell location
containing a tuple of the encryptor class and also
contains the key object for retrieving the key
Returns new cell with all locations in cell decrypted
that are noted in encryptor_dict
'''
cell_dict = cell._asdict()
dec_cell = cell_dict.copy()
for (cell_string, encryptor) in encryptor_dict.items():
encryptor.encryption.decrypt_cell(cell_dict,
dec_cell,
encryptor.key_container,
cell_string,
encryptor.cell_sections)
return Cell(*[dec_cell[field] for field in CELL_ORDER])
pace/encryption/enc_mutation_test.py
+203
View File
@@ -0,0 +1,203 @@
## **************
## Copyright 2014 MIT Lincoln Laboratory
## Project: PACE
## Authors: ATLH
## Description: Unit tests for cell code
## Modifications:
## Date Name Modification
## ---- ---- ------------
## 19 Dec 2014 ATLH Original file
## **************
import os
import sys
this_dir = os.path.dirname(os.path.dirname(__file__))
base_dir = os.path.join(this_dir, '../..')
sys.path.append(base_dir)
import random
import logging
import unittest
import ConfigParser
import StringIO as stringio
from pyaccumulo import Mutation, Cell, Range
from pace.encryption.enc_mutation import EncMutation, EncCell, EncRange
from pace.encryption.acc_encrypt import AccumuloEncrypt
from pace.encryption.encryption_pki import DummyEncryptionPKI
from pace.encryption.encryption_exceptions import EncryptionException, DecryptionException
class EncStructureTest(unittest.TestCase):
def setUp(self):
#Set up encryptor_dict
config = stringio.StringIO(
'[row]\n'+\
'key_id = Pycrypto_AES_CFB\n'+\
'encryption = Pycrypto_AES_CFB\n'+\
'cell_sections = row\n'+\
'[colFamily]\n'+\
'key_id = Pycrypto_AES_CFB\n'+\
'cell_sections = colFamily,colQualifier\n'+\
'encryption = Pycrypto_AES_CFB')
config_identity = stringio.StringIO(
'[row]\n'+\
'key_id = Identity\n'+\
'encryption = Identity\n'+\
'cell_sections = row\n'+\
'[colFamily]\n'+\
'key_id = Identity\n'+\
'cell_sections = colFamily\n'+\
'encryption = Identity')
config_det = stringio.StringIO(
'[row]\n'+\
'key_id = Pycrypto_AES_SIV\n'+\
'encryption = Pycrypto_AES_SIV\n'+\
'cell_sections = row\n'+\
'[colFamily]\n'+\
'key_id = Pycrypto_AES_SIV\n'+\
'cell_sections = colFamily,colQualifier\n'+\
'encryption = Pycrypto_AES_SIV')
# Use a consistent PKI
self.pki = DummyEncryptionPKI()
self.encryptor_dict = self._create_encryptor_dict(config)
self.encryptor_dict_identity = self._create_encryptor_dict(config_identity)
self.encryptor_dict_det = self._create_encryptor_dict(config_det)
#Sample mutation
mut = Mutation('abcd')
mut.put(cf='cf1',cq='cq1', cv='cv1', ts = '12345', val = 'val1')
self.mut = mut
#complex Sample mutation
c_mut = Mutation('abcd')
c_mut.put(cf='cf1',cq='cq1', cv='cv1', ts = '12345', val = 'val1')
c_mut.put(cf='cf2', val='val2')
c_mut.put(cf='cf3',cq='cq3', val = 'val3')
c_mut.put(val = 'val4')
self.c_mut = c_mut
def _create_encryptor_dict(self, config):
config_parser = ConfigParser.ConfigParser()
config_parser.readfp(config)
key_object = self.pki
return AccumuloEncrypt._config_to_encryptor(config_parser,
key_object)
@staticmethod
def _mutations_equal(list_mut, mut2):
'''
list_mut - list of mutations
mut2 - the original mutation
Returns if the list of mutations produced by a plaintext
mutation is the same as the original
'''
equal = True
equal &= len(list_mut) == len(mut2.updates)
for (mut, update) in zip(list_mut, mut2.updates):
equal &= mut.row == mut2.row
equal &= mut.updates[0] == update
return equal
def test_encrypt_mutation(self):
'''
Tests encryption functionality of EncMutation
when encryption is identity function
'''
enc_mut = EncMutation(self.mut,self.encryptor_dict_identity).encrypt()
self.assertTrue(self._mutations_equal(enc_mut, self.mut),
"Mutation is not correctly handled during encryption process")
def test_encrypt_overwrite(self):
"""
Tests the fact information is overwritten if encrypted and not
a target location
"""
enc_mut = EncMutation(self.mut, self.encryptor_dict).encrypt()
self.assertTrue(len(enc_mut) == 1)
for u in enc_mut[0].updates:
self.assertEqual(u.colQualifier, '', "colQualifier was not overwritten correctly")
def test_row_different(self):
"""
Tests that the row value produced is different if deterministic encryption is not
used
"""
enc_muts = EncMutation(self.c_mut, self.encryptor_dict).encrypt()
self.assertTrue(len(enc_muts) == 4)
row_ids = set()
for mut in enc_muts:
row_ids.add(mut.row)
self.assertTrue(len(row_ids) == 4)
def test_row_same(self):
"""
Tests that the row value produced is same if deterministic encryption is
used
"""
enc_muts = EncMutation(self.c_mut, self.encryptor_dict_det).encrypt()
self.assertTrue(len(enc_muts) == 4)
row_ids = set()
for mut in enc_muts:
row_ids.add(mut.row)
self.assertTrue(len(row_ids) == 1)
def test_encrypt_decrypt_mutation(self):
'''
Tests encrypt then decrypt functionality of EncMutation
'''
enc_mut = EncMutation(self.mut, self.encryptor_dict).encrypt()
self.assertTrue(len(enc_mut) == 1)
dec_mut = EncMutation(enc_mut[0], self.encryptor_dict).decrypt()
self.assertTrue(self._mutations_equal([dec_mut], self.mut),
"Mutation is not correctly handled during encryption and decryption process")
def test_decrypt_cell(self):
'''
Tests decryption functionality of EncCell when
decryption is identity function
'''
cell = Cell('row','cf','cq','cv',1234,'val')
dec_cell = EncCell.decrypt(cell, self.encryptor_dict_identity)
self.assertEqual(cell, dec_cell,
str(dec_cell) + " was not correctly handled during decyption process " +\
"should be " + str(cell))
def test_encrypt_decrypt_cell(self):
'''
Tests encrypt then decrypt functionality of EncCell
'''
cell = Cell('row','cf','cq','cv',1234,'val')
enc_cell = EncCell.encrypt(cell, self.encryptor_dict)
dec_cell = EncCell.decrypt(enc_cell, self.encryptor_dict)
self.assertEqual(cell, dec_cell,
str(dec_cell) + " was not correctly encrypted and decrypted " +\
"should be " + str(cell))
def test_get_and_split_values(self):
'''
Tests get_values_by_cell_string and split_values_by_cell_string
'''
mut = EncMutation(self.mut,self.encryptor_dict)
values = EncMutation.concatenate_cell_section_values(mut, ['colFamily','colQualifier'])
self.assertEqual(values, ['cf1+cq1'])
split_values = EncMutation.split_values(values)
self.assertEqual(split_values, [('cf1',),('cq1',)])
cell = Cell('row','cf','cq','cv',1234,'val')
value = EncCell.get_value_by_cell_string(cell._asdict(), ['colFamily', 'colQualifier'])
self.assertEqual(value, 'cf+cq')
split_value = EncCell.split_value_by_cell_string(value)
self.assertEqual(split_value,['cf','cq'])
pace/encryption/enc_utils.py
+210
View File
@@ -0,0 +1,210 @@
## **************
## Copyright 2015 MIT Lincoln Laboratory
## Project: PACE
## Authors: ATLH
## Description: Utility functions for testing the encryption code
## Modifications:
## Date Name Modification
## ---- ---- ------------
## 20 Jan 2015 ATLH Original file - based on sign_utils
## **************
""" Utility functions and tests for mass encryption/decryption of rows written to
Accumulo.
Any randomized test should use the same seed so that they are equivalent.
"""
import os
import sys
this_dir = os.path.dirname(os.path.dirname(__file__))
base_dir = os.path.join(this_dir, '../..')
sys.path.append(base_dir)
import random
import time, datetime
from pyaccumulo import Accumulo, Mutation, Range
from pace.encryption.encryption_exceptions import DecryptionException
from pace.common.common_utils import generate_data, sanitize
from pace.common import common_utils
def sanitize(string):
return common_utils.sanitize(string)
def generate_data(filename,
seed,
num_entries=50000,
num_rows=1000,
vis_length=9,
row_length=9):
common_utils.generate_data(filename, seed, default_vis = 'a&b',
num_entries = num_entries,
num_rows = num_rows,
vis_length = vis_length,
row_length = row_length)
def write_and_encrypt_data(file_in, conn, table, encryptor, benchmark=False):
#Get the lines from the file
print "Opening file", file_in
f = open(file_in, 'r')
lines = f.readlines()
f.close()
assert lines
print "Lines (%d) are now in memory." %len(lines)
#create the table if it doesn't exist yet
if not conn.table_exists(table):
conn.create_table(table)
print table, 'table created'
print "Beginning write."
wr = conn.create_batch_writer(table)
row = '340930563???poitapeoita'
m = None
if benchmark:
print "Starting encrypting..."
start = time.clock()
mutations = []
for l in lines:
pieces = l.split('\t')
if (row != pieces[0]):
if m:
mutations.append(m)
row = pieces[0]
m = Mutation(row)
vis = pieces[3]
m.put(cf=pieces[1], cq=pieces[2], cv=vis, val=pieces[4][:-1])
mutations.append(m)
if benchmark:
print "Starting encrypting..."
start = time.clock()
for m in mutations:
wr.add_mutation(encryptor.encrypt(m))
if benchmark:
end = time.clock()
print "Encrypting finished!"
wr.close()
print "Write completed."
if benchmark:
return (start, end)
def write_data(file_in, conn, table, benchmark=False):
""" Just writes the data without signing it.
"""
#Get the lines from the file
f = open(file_in, 'r')
lines = f.readlines()
f.close()
#create the table if it doesn't exist yet
if not conn.table_exists(table):
conn.create_table(table)
wr = conn.create_batch_writer(table)
m = None
mutations = []
for l in lines:
pieces = l.split('\t')
row = pieces[0]
m = Mutation(row)
vis = None
m.put(cf=pieces[1], cq=pieces[2], cv=pieces[3], val=pieces[4][:-1])
mutations.append(m)
if benchmark:
print "Starting writing..."
start = time.clock()
for m in mutations:
wr.add_mutation(m)
if benchmark:
print "Done writing ..."
end = time.clock()
wr.close()
if benchmark:
return (start,end)
def decrypt_data(conn, table, decryptor, benchmark=False):
""" Verify the signed data in a table.
Arguments:
conn - the connection to the accumulo server
table - the name of the table on the server to decrypt
decryptor - AcccumuloDecrypt object that will decrypt mutations
benchmark - whether we're benchmarking decryption (default: False)
"""
success = True
if benchmark:
print "Starting decryption..."
start = time.clock()
count = 0
dec_error = 0
for entry in conn.scan(table):
count += 1
try:
dec_entry = decryptor.decrypt(entry)
except DecryptionException as ve:
dec_error += 1
if dec_error < 11:
print "Decryption error:", ve.msg
print "Entry was %s" % (str(entry))
success = False
if benchmark:
print "Decryption finished!"
end = time.clock()
if success and count > 0:
print "Decryption of all (%d) entries succeeded." %count
elif count <= 0:
print "WARNING: no entries to decrypt"
else:
print "FAILURE: %d entries failed to decrypt." % dec_error
if benchmark:
return (success, start, end)
else:
return success
def read_data(conn, table, benchmark=False):
""" Read the data in a table.
Arguments:
conn - the connection to the accumulo server
table - the name of the table on the server to decrypt
benchmark - whether we're benchmarking decryption (default: False)
"""
success = True
if benchmark:
print "Starting reading..."
start = time.clock()
count = 0
dec_error = 0
for entry in conn.scan(table):
count += 1
if benchmark:
print "Reading finished!"
end = time.clock()
if benchmark:
return (None, start, end)
Loaded 100 of 174 files, more files were not shown because too many files have changed in this diff. Show more