Magic-Mirror/gecko-dev
1
0
Fork 0
mirror of https://github.com/mozilla/gecko-dev.git synced 2024-10-08 19:04:45 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

Bug 1492128: Vendor pathlib2==2.3.2; r=firefox-build-system-reviewers,gps

Browse Source 
Differential Revision: https://phabricator.services.mozilla.com/D10145

--HG--
extra : moz-landing-system : lando
This commit is contained in:
Tom Prince 2018-10-30 18:01:06 +00:00
parent a65c91f935
commit de09f5861b
28 changed files with 8393 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
build/virtualenv_packages.txt
View File

@ -20,6 +20,7 @@ mozilla.pth:third_party/python/funcsigs
mozilla.pth:third_party/python/futures
mozilla.pth:third_party/python/more-itertools
mozilla.pth:third_party/python/mozilla-version
mozilla.pth:third_party/python/pathlib2
mozilla.pth:third_party/python/gyp/pylib
mozilla.pth:third_party/python/python-hglib
mozilla.pth:third_party/python/pluggy
@ -33,6 +34,7 @@ mozilla.pth:third_party/python/pystache
mozilla.pth:third_party/python/pyyaml/lib
mozilla.pth:third_party/python/requests
mozilla.pth:third_party/python/requests-unixsocket
mozilla.pth:third_party/python/scandir
mozilla.pth:third_party/python/slugid
mozilla.pth:third_party/python/py
mozilla.pth:third_party/python/pytest/src

137
third_party/python/pathlib2/CHANGELOG.rst vendored Normal file
View File

@ -0,0 +1,137 @@
History
-------
Version 2.3.2
^^^^^^^^^^^^^
- Hotfix for broken setup.py.
Version 2.3.1
^^^^^^^^^^^^^
- Fix tests for systems where filesystem encoding only supports ascii
(reported by yurivict, fixed with help of honnibal, see issue #30).
- Use modern setuptools syntax for specifying conditional scandir
dependency (see issue #31).
- Remove legacy use of support module from old pathlib module (see
issue #39). This fixes the tests for Python 3.6.
- Drop the "from __future__ import unicode_literals" and -Qnew tests
as it introduced subtle bugs in the tests, and maintaining separate
test modules for these legacy features seems not worth the effort.
- Drop Python 3.2 support, as scandir no longer supports it.
Version 2.3.0
^^^^^^^^^^^^^
- Sync with upstream pathlib from CPython 3.6.1 (7d1017d).
Version 2.2.1
^^^^^^^^^^^^^
- Fix conditional scandir dependency in wheel (reported by AvdN, see
issue #20 and pull request #21).
Version 2.2.0
^^^^^^^^^^^^^
- Sync with upstream pathlib from CPython 3.5.2 and 3.6.0: fix various
exceptions, empty glob pattern, scandir, __fspath__.
- Support unicode strings to be used to construct paths in Python 2
(reported by native-api, see issue #13 and pull request #15).
Version 2.1.0
^^^^^^^^^^^^^
- Sync with upstream pathlib from CPython 3.5.0: gethomedir, home,
expanduser.
Version 2.0.1
^^^^^^^^^^^^^
- Fix TypeError exceptions in write_bytes and write_text (contributed
by Emanuele Gaifas, see pull request #2).
Version 2.0
^^^^^^^^^^^
- Sync with upstream pathlib from CPython: read_text, write_text,
read_bytes, write_bytes, __enter__, __exit__, samefile.
- Use travis and appveyor for continuous integration.
- Fixed some bugs in test code.
Version 1.0.1
^^^^^^^^^^^^^
- Pull request #4: Python 2.6 compatibility by eevee.
Version 1.0
^^^^^^^^^^^
This version brings ``pathlib`` up to date with the official Python 3.4
release, and also fixes a couple of 2.7-specific issues.
- Python issue #20765: Add missing documentation for PurePath.with_name()
and PurePath.with_suffix().
- Fix test_mkdir_parents when the working directory has additional bits
set (such as the setgid or sticky bits).
- Python issue #20111: pathlib.Path.with_suffix() now sanity checks the
given suffix.
- Python issue #19918: Fix PurePath.relative_to() under Windows.
- Python issue #19921: When Path.mkdir() is called with parents=True, any
missing parent is created with the default permissions, ignoring the mode
argument (mimicking the POSIX "mkdir -p" command).
- Python issue #19887: Improve the Path.resolve() algorithm to support
certain symlink chains.
- Make pathlib usable under Python 2.7 with unicode pathnames (only pure
ASCII, though).
- Issue #21: fix TypeError under Python 2.7 when using new division.
- Add tox support for easier testing.
Version 0.97
^^^^^^^^^^^^
This version brings ``pathlib`` up to date with the final API specified
in :pep:`428`. The changes are too long to list here, it is recommended
to read the `documentation <https://pathlib.readthedocs.org/>`_.
.. warning::
The API in this version is partially incompatible with pathlib 0.8 and
earlier. Be sure to check your code for possible breakage!
Version 0.8
^^^^^^^^^^^
- Add PurePath.name and PurePath.anchor.
- Add Path.owner and Path.group.
- Add Path.replace().
- Add Path.as_uri().
- Issue #10: when creating a file with Path.open(), don't set the executable
bit.
- Issue #11: fix comparisons with non-Path objects.
Version 0.7
^^^^^^^^^^^
- Add '**' (recursive) patterns to Path.glob().
- Fix openat() support after the API refactoring in Python 3.3 beta1.
- Add a *target_is_directory* argument to Path.symlink_to()
Version 0.6
^^^^^^^^^^^
- Add Path.is_file() and Path.is_symlink()
- Add Path.glob() and Path.rglob()
- Add PurePath.match()
Version 0.5
^^^^^^^^^^^
- Add Path.mkdir().
- Add Python 2.7 compatibility by Michele Lacchia.
- Make parent() raise ValueError when the level is greater than the path
length.

23
third_party/python/pathlib2/LICENSE.rst vendored Normal file
View File

@ -0,0 +1,23 @@
The MIT License (MIT)
Copyright (c) 2014-2017 Matthias C. M. Troffaes
Copyright (c) 2012-2014 Antoine Pitrou and contributors
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

10
third_party/python/pathlib2/MANIFEST.in vendored Normal file
View File

@ -0,0 +1,10 @@
include *.py
recursive-include pathlib2 *.py
recursive-include tests *.py
include *.rst
include VERSION
include requirements.txt
exclude .travis.yml
exclude appveyor.yml
exclude codecov.yml
prune appveyor

72
third_party/python/pathlib2/PKG-INFO vendored Normal file
View File

@ -0,0 +1,72 @@
Metadata-Version: 1.1
Name: pathlib2
Version: 2.3.2
Summary: Object-oriented filesystem paths
Home-page: https://pypi.python.org/pypi/pathlib2/
Author: Matthias C. M. Troffaes
Author-email: matthias.troffaes@gmail.com
License: MIT
Download-URL: https://pypi.python.org/pypi/pathlib2/
Description-Content-Type: UNKNOWN
Description: The `old pathlib <https://bitbucket.org/pitrou/pathlib>`_
module on bitbucket is in bugfix-only mode.
The goal of pathlib2 is to provide a backport of
`standard pathlib <http://docs.python.org/dev/library/pathlib.html>`_
module which tracks the standard library module,
so all the newest features of the standard pathlib can be
used also on older Python versions.
Download
--------
Standalone releases are available on PyPI:
http://pypi.python.org/pypi/pathlib2/
Development
-----------
The main development takes place in the Python standard library: see
the `Python developer's guide <http://docs.python.org/devguide/>`_.
In particular, new features should be submitted to the
`Python bug tracker <http://bugs.python.org/>`_.
Issues that occur in this backport, but that do not occur not in the
standard Python pathlib module can be submitted on
the `pathlib2 bug tracker <https://github.com/mcmtroffaes/pathlib2/issues>`_.
Documentation
-------------
Refer to the
`standard pathlib <http://docs.python.org/dev/library/pathlib.html>`_
documentation.
.. |travis| image:: https://travis-ci.org/mcmtroffaes/pathlib2.png?branch=develop
:target: https://travis-ci.org/mcmtroffaes/pathlib2
:alt: travis-ci
.. |appveyor| image:: https://ci.appveyor.com/api/projects/status/baddx3rpet2wyi2c?svg=true
:target: https://ci.appveyor.com/project/mcmtroffaes/pathlib2
:alt: appveyor
.. |codecov| image:: https://codecov.io/gh/mcmtroffaes/pathlib2/branch/develop/graph/badge.svg
:target: https://codecov.io/gh/mcmtroffaes/pathlib2
:alt: codecov
Platform: UNKNOWN
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 2
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 2.6
Classifier: Programming Language :: Python :: 2.7
Classifier: Programming Language :: Python :: 3.3
Classifier: Programming Language :: Python :: 3.4
Classifier: Programming Language :: Python :: 3.5
Classifier: Programming Language :: Python :: 3.6
Classifier: Topic :: Software Development :: Libraries
Classifier: Topic :: System :: Filesystems

52
third_party/python/pathlib2/README.rst vendored Normal file
View File

@ -0,0 +1,52 @@
pathlib2
========
|appveyor| |travis| |codecov|
Fork of pathlib aiming to support the full stdlib Python API.
The `old pathlib <https://bitbucket.org/pitrou/pathlib>`_
module on bitbucket is in bugfix-only mode.
The goal of pathlib2 is to provide a backport of
`standard pathlib <http://docs.python.org/dev/library/pathlib.html>`_
module which tracks the standard library module,
so all the newest features of the standard pathlib can be
used also on older Python versions.
Download
--------
Standalone releases are available on PyPI:
http://pypi.python.org/pypi/pathlib2/
Development
-----------
The main development takes place in the Python standard library: see
the `Python developer's guide <http://docs.python.org/devguide/>`_.
In particular, new features should be submitted to the
`Python bug tracker <http://bugs.python.org/>`_.
Issues that occur in this backport, but that do not occur not in the
standard Python pathlib module can be submitted on
the `pathlib2 bug tracker <https://github.com/mcmtroffaes/pathlib2/issues>`_.
Documentation
-------------
Refer to the
`standard pathlib <http://docs.python.org/dev/library/pathlib.html>`_
documentation.
.. |travis| image:: https://travis-ci.org/mcmtroffaes/pathlib2.png?branch=develop
:target: https://travis-ci.org/mcmtroffaes/pathlib2
:alt: travis-ci
.. |appveyor| image:: https://ci.appveyor.com/api/projects/status/baddx3rpet2wyi2c?svg=true
:target: https://ci.appveyor.com/project/mcmtroffaes/pathlib2
:alt: appveyor
.. |codecov| image:: https://codecov.io/gh/mcmtroffaes/pathlib2/branch/develop/graph/badge.svg
:target: https://codecov.io/gh/mcmtroffaes/pathlib2
:alt: codecov

1
third_party/python/pathlib2/VERSION vendored Normal file
View File

@ -0,0 +1 @@
2.3.2

1670
third_party/python/pathlib2/pathlib2/__init__.py vendored Normal file
View File

File diff suppressed because it is too large Load Diff

3
third_party/python/pathlib2/requirements.txt vendored Normal file
View File

@ -0,0 +1,3 @@
six
scandir; python_version < '3.5'
mock; python_version < '3.3'

13
third_party/python/pathlib2/setup.cfg vendored Normal file
View File

@ -0,0 +1,13 @@
[nosetests]
with-coverage = 1
cover-package = pathlib2
cover-branches = 1
cover-html = 1
[wheel]
universal = 1
[egg_info]
tag_build =
tag_date = 0

49
third_party/python/pathlib2/setup.py vendored Normal file
View File

@ -0,0 +1,49 @@
# Copyright (c) 2014-2017 Matthias C. M. Troffaes
# Copyright (c) 2012-2014 Antoine Pitrou and contributors
# Distributed under the terms of the MIT License.
import io
from setuptools import setup, find_packages
def readfile(filename):
with io.open(filename, encoding="utf-8") as stream:
return stream.read().split("\n")
readme = readfile("README.rst")[5:] # skip title and badges
version = readfile("VERSION")[0].strip()
setup(
name='pathlib2',
version=version,
packages=find_packages(),
license='MIT',
description='Object-oriented filesystem paths',
long_description="\n".join(readme[2:]),
author='Matthias C. M. Troffaes',
author_email='matthias.troffaes@gmail.com',
classifiers=[
'Development Status :: 5 - Production/Stable',
'Intended Audience :: Developers',
'License :: OSI Approved :: MIT License',
'Operating System :: OS Independent',
'Programming Language :: Python',
'Programming Language :: Python :: 2',
'Programming Language :: Python :: 3',
'Programming Language :: Python :: 2.6',
'Programming Language :: Python :: 2.7',
'Programming Language :: Python :: 3.3',
'Programming Language :: Python :: 3.4',
'Programming Language :: Python :: 3.5',
'Programming Language :: Python :: 3.6',
'Topic :: Software Development :: Libraries',
'Topic :: System :: Filesystems',
],
download_url='https://pypi.python.org/pypi/pathlib2/',
url='https://pypi.python.org/pypi/pathlib2/',
install_requires=[
'six',
'scandir;python_version<"3.5"',
],
)

2401
third_party/python/pathlib2/tests/test_pathlib2.py vendored Normal file
View File

File diff suppressed because it is too large Load Diff

1
third_party/python/requirements.in vendored
View File

@ -3,6 +3,7 @@ blessings==1.7
jsmin==2.1.0
json-e==2.7.0
mozilla-version==0.3.0
pathlib2==2.3.2
pip-tools==3.0.0
pipenv==2018.5.18
psutil==5.4.3

16
third_party/python/requirements.txt vendored
View File

@ -38,6 +38,9 @@ more-itertools==4.3.0 \
# via pytest
mozilla-version==0.3.0 \
--hash=sha256:97f428f6a87f1a0569e03c39e446eeed87c3ec5d8300319d41e8348ef832e8ea
pathlib2==2.3.2 \
--hash=sha256:8eb170f8d0d61825e09a95b38be068299ddeda82f35e96c3301a8a5e7604cb83 \
--hash=sha256:d1aa2a11ba7b8f7b21ab852b1fb5afb277e1bb99d5dfc663380b5015c0d80c5a
pip-tools==3.0.0 \
--hash=sha256:4a94997602848f77ff02f660c0fcdfeaf316924ebb236c865f9742ce212aa6f9 \
--hash=sha256:e45e5198ce3799068642ebb0e7c9be5520bcff944c0186f79c1199a2759c970a
@ -71,6 +74,19 @@ python-hglib==2.4 \
requests==2.9.1 \
--hash=sha256:113fbba5531a9e34945b7d36b33a084e8ba5d0664b703c81a7c572d91919a5b8 \
--hash=sha256:c577815dd00f1394203fc44eb979724b098f88264a9ef898ee45b8e5e9cf587f
scandir==1.9.0 \
--hash=sha256:04b8adb105f2ed313a7c2ef0f1cf7aff4871aa7a1883fa4d8c44b5551ab052d6 \
--hash=sha256:1444134990356c81d12f30e4b311379acfbbcd03e0bab591de2696a3b126d58e \
--hash=sha256:1b5c314e39f596875e5a95dd81af03730b338c277c54a454226978d5ba95dbb6 \
--hash=sha256:346619f72eb0ddc4cf355ceffd225fa52506c92a2ff05318cfabd02a144e7c4e \
--hash=sha256:44975e209c4827fc18a3486f257154d34ec6eaec0f90fef0cca1caa482db7064 \
--hash=sha256:61859fd7e40b8c71e609c202db5b0c1dbec0d5c7f1449dec2245575bdc866792 \
--hash=sha256:a5e232a0bf188362fa00123cc0bb842d363a292de7126126df5527b6a369586a \
--hash=sha256:c14701409f311e7a9b7ec8e337f0815baf7ac95776cc78b419a1e6d49889a383 \
--hash=sha256:c7708f29d843fc2764310732e41f0ce27feadde453261859ec0fca7865dfc41b \
--hash=sha256:c9009c527929f6e25604aec39b0a43c3f831d2947d89d6caaab22f057b7055c8 \
--hash=sha256:f5c71e29b4e2af7ccdc03a020c626ede51da471173b4a6ad1e904f2b2e04b4bd \
# via pathlib2
six==1.10.0 \
--hash=sha256:0ff78c403d9bccf5a425a6d31a12aa6b47f1c21ca4dc2573a7e2f32a97335eb1 \
--hash=sha256:105f8d68616f8248e24bf0e9372ef04d3cc10104f1980f54d57b2ce73a5ad56a

27
third_party/python/scandir/LICENSE.txt vendored Normal file
View File

@ -0,0 +1,27 @@
Copyright (c) 2012, Ben Hoyt
All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
* Redistributions of source code must retain the above copyright notice, this
list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation
and/or other materials provided with the distribution.
* Neither the name of Ben Hoyt nor the names of its contributors may be used
to endorse or promote products derived from this software without specific
prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

6
third_party/python/scandir/MANIFEST.in vendored Normal file
View File

@ -0,0 +1,6 @@
include *.py
include *.c
include *.h
include *.txt
include *.rst
include test/*.py

238
third_party/python/scandir/PKG-INFO vendored Normal file
View File

@ -0,0 +1,238 @@
Metadata-Version: 1.1
Name: scandir
Version: 1.9.0
Summary: scandir, a better directory iterator and faster os.walk()
Home-page: https://github.com/benhoyt/scandir
Author: Ben Hoyt
Author-email: benhoyt@gmail.com
License: New BSD License
Description-Content-Type: UNKNOWN
Description:
scandir, a better directory iterator and faster os.walk()
=========================================================
.. image:: https://img.shields.io/pypi/v/scandir.svg
:target: https://pypi.python.org/pypi/scandir
:alt: scandir on PyPI (Python Package Index)
.. image:: https://travis-ci.org/benhoyt/scandir.svg?branch=master
:target: https://travis-ci.org/benhoyt/scandir
:alt: Travis CI tests (Linux)
.. image:: https://ci.appveyor.com/api/projects/status/github/benhoyt/scandir?branch=master&svg=true
:target: https://ci.appveyor.com/project/benhoyt/scandir
:alt: Appveyor tests (Windows)
``scandir()`` is a directory iteration function like ``os.listdir()``,
except that instead of returning a list of bare filenames, it yields
``DirEntry`` objects that include file type and stat information along
with the name. Using ``scandir()`` increases the speed of ``os.walk()``
by 2-20 times (depending on the platform and file system) by avoiding
unnecessary calls to ``os.stat()`` in most cases.
Now included in a Python near you!
----------------------------------
``scandir`` has been included in the Python 3.5 standard library as
``os.scandir()``, and the related performance improvements to
``os.walk()`` have also been included. So if you're lucky enough to be
using Python 3.5 (release date September 13, 2015) you get the benefit
immediately, otherwise just
`download this module from PyPI <https://pypi.python.org/pypi/scandir>`_,
install it with ``pip install scandir``, and then do something like
this in your code:
.. code-block:: python
# Use the built-in version of scandir/walk if possible, otherwise
# use the scandir module version
try:
from os import scandir, walk
except ImportError:
from scandir import scandir, walk
`PEP 471 <https://www.python.org/dev/peps/pep-0471/>`_, which is the
PEP that proposes including ``scandir`` in the Python standard library,
was `accepted <https://mail.python.org/pipermail/python-dev/2014-July/135561.html>`_
in July 2014 by Victor Stinner, the BDFL-delegate for the PEP.
This ``scandir`` module is intended to work on Python 2.6+ and Python
3.2+ (and it has been tested on those versions).
Background
----------
Python's built-in ``os.walk()`` is significantly slower than it needs to be,
because -- in addition to calling ``listdir()`` on each directory -- it calls
``stat()`` on each file to determine whether the filename is a directory or not.
But both ``FindFirstFile`` / ``FindNextFile`` on Windows and ``readdir`` on Linux/OS
X already tell you whether the files returned are directories or not, so
no further ``stat`` system calls are needed. In short, you can reduce the number
of system calls from about 2N to N, where N is the total number of files and
directories in the tree.
In practice, removing all those extra system calls makes ``os.walk()`` about
**7-50 times as fast on Windows, and about 3-10 times as fast on Linux and Mac OS
X.** So we're not talking about micro-optimizations. See more benchmarks
in the "Benchmarks" section below.
Somewhat relatedly, many people have also asked for a version of
``os.listdir()`` that yields filenames as it iterates instead of returning them
as one big list. This improves memory efficiency for iterating very large
directories.
So as well as a faster ``walk()``, scandir adds a new ``scandir()`` function.
They're pretty easy to use, but see "The API" below for the full docs.
Benchmarks
----------
Below are results showing how many times as fast ``scandir.walk()`` is than
``os.walk()`` on various systems, found by running ``benchmark.py`` with no
arguments:
==================== ============== =============
System version Python version Times as fast
==================== ============== =============
Windows 7 64-bit 2.7.7 64-bit 10.4
Windows 7 64-bit SSD 2.7.7 64-bit 10.3
Windows 7 64-bit NFS 2.7.6 64-bit 36.8
Windows 7 64-bit SSD 3.4.1 64-bit 9.9
Windows 7 64-bit SSD 3.5.0 64-bit 9.5
CentOS 6.2 64-bit 2.6.6 64-bit 3.9
Ubuntu 14.04 64-bit 2.7.6 64-bit 5.8
Mac OS X 10.9.3 2.7.5 64-bit 3.8
==================== ============== =============
All of the above tests were done using the fast C version of scandir
(source code in ``_scandir.c``).
Note that the gains are less than the above on smaller directories and greater
on larger directories. This is why ``benchmark.py`` creates a test directory
tree with a standardized size.
The API
-------
walk()
~~~~~~
The API for ``scandir.walk()`` is exactly the same as ``os.walk()``, so just
`read the Python docs <https://docs.python.org/3.5/library/os.html#os.walk>`_.
scandir()
~~~~~~~~~
The full docs for ``scandir()`` and the ``DirEntry`` objects it yields are
available in the `Python documentation here <https://docs.python.org/3.5/library/os.html#os.scandir>`_.
But below is a brief summary as well.
scandir(path='.') -> iterator of DirEntry objects for given path
Like ``listdir``, ``scandir`` calls the operating system's directory
iteration system calls to get the names of the files in the given
``path``, but it's different from ``listdir`` in two ways:
* Instead of returning bare filename strings, it returns lightweight
``DirEntry`` objects that hold the filename string and provide
simple methods that allow access to the additional data the
operating system may have returned.
* It returns a generator instead of a list, so that ``scandir`` acts
as a true iterator instead of returning the full list immediately.
``scandir()`` yields a ``DirEntry`` object for each file and
sub-directory in ``path``. Just like ``listdir``, the ``'.'``
and ``'..'`` pseudo-directories are skipped, and the entries are
yielded in system-dependent order. Each ``DirEntry`` object has the
following attributes and methods:
* ``name``: the entry's filename, relative to the scandir ``path``
argument (corresponds to the return values of ``os.listdir``)
* ``path``: the entry's full path name (not necessarily an absolute
path) -- the equivalent of ``os.path.join(scandir_path, entry.name)``
* ``is_dir(*, follow_symlinks=True)``: similar to
``pathlib.Path.is_dir()``, but the return value is cached on the
``DirEntry`` object; doesn't require a system call in most cases;
don't follow symbolic links if ``follow_symlinks`` is False
* ``is_file(*, follow_symlinks=True)``: similar to
``pathlib.Path.is_file()``, but the return value is cached on the
``DirEntry`` object; doesn't require a system call in most cases;
don't follow symbolic links if ``follow_symlinks`` is False
* ``is_symlink()``: similar to ``pathlib.Path.is_symlink()``, but the
return value is cached on the ``DirEntry`` object; doesn't require a
system call in most cases
* ``stat(*, follow_symlinks=True)``: like ``os.stat()``, but the
return value is cached on the ``DirEntry`` object; does not require a
system call on Windows (except for symlinks); don't follow symbolic links
(like ``os.lstat()``) if ``follow_symlinks`` is False
* ``inode()``: return the inode number of the entry; the return value
is cached on the ``DirEntry`` object
Here's a very simple example of ``scandir()`` showing use of the
``DirEntry.name`` attribute and the ``DirEntry.is_dir()`` method:
.. code-block:: python
def subdirs(path):
"""Yield directory names not starting with '.' under given path."""
for entry in os.scandir(path):
if not entry.name.startswith('.') and entry.is_dir():
yield entry.name
This ``subdirs()`` function will be significantly faster with scandir
than ``os.listdir()`` and ``os.path.isdir()`` on both Windows and POSIX
systems, especially on medium-sized or large directories.
Further reading
---------------
* `The Python docs for scandir <https://docs.python.org/3.5/library/os.html#os.scandir>`_
* `PEP 471 <https://www.python.org/dev/peps/pep-0471/>`_, the
(now-accepted) Python Enhancement Proposal that proposed adding
``scandir`` to the standard library -- a lot of details here,
including rejected ideas and previous discussion
Flames, comments, bug reports
-----------------------------
Please send flames, comments, and questions about scandir to Ben Hoyt:
http://benhoyt.com/
File bug reports for the version in the Python 3.5 standard library
`here <https://docs.python.org/3.5/bugs.html>`_, or file bug reports
or feature requests for this module at the GitHub project page:
https://github.com/benhoyt/scandir
Platform: UNKNOWN
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: License :: OSI Approved :: BSD License
Classifier: Programming Language :: Python
Classifier: Topic :: System :: Filesystems
Classifier: Topic :: System :: Operating System
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 2
Classifier: Programming Language :: Python :: 2.6
Classifier: Programming Language :: Python :: 2.7
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.4
Classifier: Programming Language :: Python :: 3.5
Classifier: Programming Language :: Python :: 3.6
Classifier: Programming Language :: Python :: Implementation :: CPython

211
third_party/python/scandir/README.rst vendored Normal file
View File

@ -0,0 +1,211 @@
scandir, a better directory iterator and faster os.walk()
=========================================================
.. image:: https://img.shields.io/pypi/v/scandir.svg
:target: https://pypi.python.org/pypi/scandir
:alt: scandir on PyPI (Python Package Index)
.. image:: https://travis-ci.org/benhoyt/scandir.svg?branch=master
:target: https://travis-ci.org/benhoyt/scandir
:alt: Travis CI tests (Linux)
.. image:: https://ci.appveyor.com/api/projects/status/github/benhoyt/scandir?branch=master&svg=true
:target: https://ci.appveyor.com/project/benhoyt/scandir
:alt: Appveyor tests (Windows)
``scandir()`` is a directory iteration function like ``os.listdir()``,
except that instead of returning a list of bare filenames, it yields
``DirEntry`` objects that include file type and stat information along
with the name. Using ``scandir()`` increases the speed of ``os.walk()``
by 2-20 times (depending on the platform and file system) by avoiding
unnecessary calls to ``os.stat()`` in most cases.
Now included in a Python near you!
----------------------------------
``scandir`` has been included in the Python 3.5 standard library as
``os.scandir()``, and the related performance improvements to
``os.walk()`` have also been included. So if you're lucky enough to be
using Python 3.5 (release date September 13, 2015) you get the benefit
immediately, otherwise just
`download this module from PyPI <https://pypi.python.org/pypi/scandir>`_,
install it with ``pip install scandir``, and then do something like
this in your code:
.. code-block:: python
# Use the built-in version of scandir/walk if possible, otherwise
# use the scandir module version
try:
from os import scandir, walk
except ImportError:
from scandir import scandir, walk
`PEP 471 <https://www.python.org/dev/peps/pep-0471/>`_, which is the
PEP that proposes including ``scandir`` in the Python standard library,
was `accepted <https://mail.python.org/pipermail/python-dev/2014-July/135561.html>`_
in July 2014 by Victor Stinner, the BDFL-delegate for the PEP.
This ``scandir`` module is intended to work on Python 2.6+ and Python
3.2+ (and it has been tested on those versions).
Background
----------
Python's built-in ``os.walk()`` is significantly slower than it needs to be,
because -- in addition to calling ``listdir()`` on each directory -- it calls
``stat()`` on each file to determine whether the filename is a directory or not.
But both ``FindFirstFile`` / ``FindNextFile`` on Windows and ``readdir`` on Linux/OS
X already tell you whether the files returned are directories or not, so
no further ``stat`` system calls are needed. In short, you can reduce the number
of system calls from about 2N to N, where N is the total number of files and
directories in the tree.
In practice, removing all those extra system calls makes ``os.walk()`` about
**7-50 times as fast on Windows, and about 3-10 times as fast on Linux and Mac OS
X.** So we're not talking about micro-optimizations. See more benchmarks
in the "Benchmarks" section below.
Somewhat relatedly, many people have also asked for a version of
``os.listdir()`` that yields filenames as it iterates instead of returning them
as one big list. This improves memory efficiency for iterating very large
directories.
So as well as a faster ``walk()``, scandir adds a new ``scandir()`` function.
They're pretty easy to use, but see "The API" below for the full docs.
Benchmarks
----------
Below are results showing how many times as fast ``scandir.walk()`` is than
``os.walk()`` on various systems, found by running ``benchmark.py`` with no
arguments:
==================== ============== =============
System version Python version Times as fast
==================== ============== =============
Windows 7 64-bit 2.7.7 64-bit 10.4
Windows 7 64-bit SSD 2.7.7 64-bit 10.3
Windows 7 64-bit NFS 2.7.6 64-bit 36.8
Windows 7 64-bit SSD 3.4.1 64-bit 9.9
Windows 7 64-bit SSD 3.5.0 64-bit 9.5
CentOS 6.2 64-bit 2.6.6 64-bit 3.9
Ubuntu 14.04 64-bit 2.7.6 64-bit 5.8
Mac OS X 10.9.3 2.7.5 64-bit 3.8
==================== ============== =============
All of the above tests were done using the fast C version of scandir
(source code in ``_scandir.c``).
Note that the gains are less than the above on smaller directories and greater
on larger directories. This is why ``benchmark.py`` creates a test directory
tree with a standardized size.
The API
-------
walk()
~~~~~~
The API for ``scandir.walk()`` is exactly the same as ``os.walk()``, so just
`read the Python docs <https://docs.python.org/3.5/library/os.html#os.walk>`_.
scandir()
~~~~~~~~~
The full docs for ``scandir()`` and the ``DirEntry`` objects it yields are
available in the `Python documentation here <https://docs.python.org/3.5/library/os.html#os.scandir>`_.
But below is a brief summary as well.
scandir(path='.') -> iterator of DirEntry objects for given path
Like ``listdir``, ``scandir`` calls the operating system's directory
iteration system calls to get the names of the files in the given
``path``, but it's different from ``listdir`` in two ways:
* Instead of returning bare filename strings, it returns lightweight
``DirEntry`` objects that hold the filename string and provide
simple methods that allow access to the additional data the
operating system may have returned.
* It returns a generator instead of a list, so that ``scandir`` acts
as a true iterator instead of returning the full list immediately.
``scandir()`` yields a ``DirEntry`` object for each file and
sub-directory in ``path``. Just like ``listdir``, the ``'.'``
and ``'..'`` pseudo-directories are skipped, and the entries are
yielded in system-dependent order. Each ``DirEntry`` object has the
following attributes and methods:
* ``name``: the entry's filename, relative to the scandir ``path``
argument (corresponds to the return values of ``os.listdir``)
* ``path``: the entry's full path name (not necessarily an absolute
path) -- the equivalent of ``os.path.join(scandir_path, entry.name)``
* ``is_dir(*, follow_symlinks=True)``: similar to
``pathlib.Path.is_dir()``, but the return value is cached on the
``DirEntry`` object; doesn't require a system call in most cases;
don't follow symbolic links if ``follow_symlinks`` is False
* ``is_file(*, follow_symlinks=True)``: similar to
``pathlib.Path.is_file()``, but the return value is cached on the
``DirEntry`` object; doesn't require a system call in most cases;
don't follow symbolic links if ``follow_symlinks`` is False
* ``is_symlink()``: similar to ``pathlib.Path.is_symlink()``, but the
return value is cached on the ``DirEntry`` object; doesn't require a
system call in most cases
* ``stat(*, follow_symlinks=True)``: like ``os.stat()``, but the
return value is cached on the ``DirEntry`` object; does not require a
system call on Windows (except for symlinks); don't follow symbolic links
(like ``os.lstat()``) if ``follow_symlinks`` is False
* ``inode()``: return the inode number of the entry; the return value
is cached on the ``DirEntry`` object
Here's a very simple example of ``scandir()`` showing use of the
``DirEntry.name`` attribute and the ``DirEntry.is_dir()`` method:
.. code-block:: python
def subdirs(path):
"""Yield directory names not starting with '.' under given path."""
for entry in os.scandir(path):
if not entry.name.startswith('.') and entry.is_dir():
yield entry.name
This ``subdirs()`` function will be significantly faster with scandir
than ``os.listdir()`` and ``os.path.isdir()`` on both Windows and POSIX
systems, especially on medium-sized or large directories.
Further reading
---------------
* `The Python docs for scandir <https://docs.python.org/3.5/library/os.html#os.scandir>`_
* `PEP 471 <https://www.python.org/dev/peps/pep-0471/>`_, the
(now-accepted) Python Enhancement Proposal that proposed adding
``scandir`` to the standard library -- a lot of details here,
including rejected ideas and previous discussion
Flames, comments, bug reports
-----------------------------
Please send flames, comments, and questions about scandir to Ben Hoyt:
http://benhoyt.com/
File bug reports for the version in the Python 3.5 standard library
`here <https://docs.python.org/3.5/bugs.html>`_, or file bug reports
or feature requests for this module at the GitHub project page:
https://github.com/benhoyt/scandir

1833
third_party/python/scandir/_scandir.c vendored Normal file
View File

File diff suppressed because it is too large Load Diff

192
third_party/python/scandir/benchmark.py vendored Normal file
View File

@ -0,0 +1,192 @@
"""Simple benchmark to compare the speed of scandir.walk() with os.walk()."""
import optparse
import os
import stat
import sys
import timeit
import warnings
with warnings.catch_warnings(record=True):
import scandir
DEPTH = 4
NUM_DIRS = 5
NUM_FILES = 50
def os_walk_pre_35(top, topdown=True, onerror=None, followlinks=False):
"""Pre Python 3.5 implementation of os.walk() that doesn't use scandir."""
islink, join, isdir = os.path.islink, os.path.join, os.path.isdir
try:
names = os.listdir(top)
except OSError as err:
if onerror is not None:
onerror(err)
return
dirs, nondirs = [], []
for name in names:
if isdir(join(top, name)):
dirs.append(name)
else:
nondirs.append(name)
if topdown:
yield top, dirs, nondirs
for name in dirs:
new_path = join(top, name)
if followlinks or not islink(new_path):
for x in os_walk_pre_35(new_path, topdown, onerror, followlinks):
yield x
if not topdown:
yield top, dirs, nondirs
def create_tree(path, depth=DEPTH):
"""Create a directory tree at path with given depth, and NUM_DIRS and
NUM_FILES at each level.
"""
os.mkdir(path)
for i in range(NUM_FILES):
filename = os.path.join(path, 'file{0:03}.txt'.format(i))
with open(filename, 'wb') as f:
f.write(b'foo')
if depth <= 1:
return
for i in range(NUM_DIRS):
dirname = os.path.join(path, 'dir{0:03}'.format(i))
create_tree(dirname, depth - 1)
def get_tree_size(path):
"""Return total size of all files in directory tree at path."""
size = 0
try:
for entry in scandir.scandir(path):
if entry.is_symlink():
pass
elif entry.is_dir():
size += get_tree_size(os.path.join(path, entry.name))
else:
size += entry.stat().st_size
except OSError:
pass
return size
def benchmark(path, get_size=False):
sizes = {}
if get_size:
def do_os_walk():
size = 0
for root, dirs, files in os.walk(path):
for filename in files:
fullname = os.path.join(root, filename)
st = os.lstat(fullname)
if not stat.S_ISLNK(st.st_mode):
size += st.st_size
sizes['os_walk'] = size
def do_scandir_walk():
sizes['scandir_walk'] = get_tree_size(path)
else:
def do_os_walk():
for root, dirs, files in os.walk(path):
pass
def do_scandir_walk():
for root, dirs, files in scandir.walk(path):
pass
# Run this once first to cache things, so we're not benchmarking I/O
print("Priming the system's cache...")
do_scandir_walk()
# Use the best of 3 time for each of them to eliminate high outliers
os_walk_time = 1000000
scandir_walk_time = 1000000
N = 3
for i in range(N):
print('Benchmarking walks on {0}, repeat {1}/{2}...'.format(
path, i + 1, N))
os_walk_time = min(os_walk_time, timeit.timeit(do_os_walk, number=1))
scandir_walk_time = min(scandir_walk_time,
timeit.timeit(do_scandir_walk, number=1))
if get_size:
if sizes['os_walk'] == sizes['scandir_walk']:
equality = 'equal'
else:
equality = 'NOT EQUAL!'
print('os.walk size {0}, scandir.walk size {1} -- {2}'.format(
sizes['os_walk'], sizes['scandir_walk'], equality))
print('os.walk took {0:.3f}s, scandir.walk took {1:.3f}s -- {2:.1f}x as fast'.format(
os_walk_time, scandir_walk_time, os_walk_time / scandir_walk_time))
if __name__ == '__main__':
usage = """Usage: benchmark.py [-h] [tree_dir]
Create a large directory tree named "benchtree" (relative to this script) and
benchmark os.walk() versus scandir.walk(). If tree_dir is specified, benchmark
using it instead of creating a tree."""
parser = optparse.OptionParser(usage=usage)
parser.add_option('-s', '--size', action='store_true',
help='get size of directory tree while walking')
parser.add_option('-c', '--scandir', type='choice', choices=['best', 'generic', 'c', 'python', 'os'], default='best',
help='version of scandir() to use, default "%default"')
options, args = parser.parse_args()
if args:
tree_dir = args[0]
else:
tree_dir = os.path.join(os.path.dirname(__file__), 'benchtree')
if not os.path.exists(tree_dir):
print('Creating tree at {0}: depth={1}, num_dirs={2}, num_files={3}'.format(
tree_dir, DEPTH, NUM_DIRS, NUM_FILES))
create_tree(tree_dir)
if options.scandir == 'generic':
scandir.scandir = scandir.scandir_generic
elif options.scandir == 'c':
if scandir.scandir_c is None:
print("ERROR: Compiled C version of scandir not found!")
sys.exit(1)
scandir.scandir = scandir.scandir_c
elif options.scandir == 'python':
if scandir.scandir_python is None:
print("ERROR: Python version of scandir not found!")
sys.exit(1)
scandir.scandir = scandir.scandir_python
elif options.scandir == 'os':
if not hasattr(os, 'scandir'):
print("ERROR: Python 3.5's os.scandir() not found!")
sys.exit(1)
scandir.scandir = os.scandir
elif hasattr(os, 'scandir'):
scandir.scandir = os.scandir
if scandir.scandir == getattr(os, 'scandir', None):
print("Using Python 3.5's builtin os.scandir()")
elif scandir.scandir == scandir.scandir_c:
print('Using fast C version of scandir')
elif scandir.scandir == scandir.scandir_python:
print('Using slower ctypes version of scandir')
elif scandir.scandir == scandir.scandir_generic:
print('Using very slow generic version of scandir')
else:
print('ERROR: Unsure which version of scandir we are using!')
sys.exit(1)
if hasattr(os, 'scandir'):
os.walk = os_walk_pre_35
print('Comparing against pre-Python 3.5 version of os.walk()')
else:
print('Comparing against builtin version of os.walk()')
benchmark(tree_dir, get_size=options.size)

48
third_party/python/scandir/osdefs.h vendored Normal file
View File

@ -0,0 +1,48 @@
// from CPython
#ifndef Py_OSDEFS_H
#define Py_OSDEFS_H
#ifdef __cplusplus
extern "C" {
#endif
/* Operating system dependencies */
#ifdef MS_WINDOWS
#define SEP L'\\'
#define ALTSEP L'/'
#define MAXPATHLEN 256
#define DELIM L';'
#endif
/* Filename separator */
#ifndef SEP
#define SEP L'/'
#endif
/* Max pathname length */
#ifdef __hpux
#include <sys/param.h>
#include <limits.h>
#ifndef PATH_MAX
#define PATH_MAX MAXPATHLEN
#endif
#endif
#ifndef MAXPATHLEN
#if defined(PATH_MAX) && PATH_MAX > 1024
#define MAXPATHLEN PATH_MAX
#else
#define MAXPATHLEN 1024
#endif
#endif
/* Search path entry delimiter */
#ifndef DELIM
#define DELIM L':'
#endif
#ifdef __cplusplus
}
#endif
#endif /* !Py_OSDEFS_H */

693
third_party/python/scandir/scandir.py vendored Normal file
View File

@ -0,0 +1,693 @@
"""scandir, a better directory iterator and faster os.walk(), now in the Python 3.5 stdlib
scandir() is a generator version of os.listdir() that returns an
iterator over files in a directory, and also exposes the extra
information most OSes provide while iterating files in a directory
(such as type and stat information).
This module also includes a version of os.walk() that uses scandir()
to speed it up significantly.
See README.md or https://github.com/benhoyt/scandir for rationale and
docs, or read PEP 471 (https://www.python.org/dev/peps/pep-0471/) for
more details on its inclusion into Python 3.5
scandir is released under the new BSD 3-clause license. See
LICENSE.txt for the full license text.
"""
from __future__ import division
from errno import ENOENT
from os import listdir, lstat, stat, strerror
from os.path import join, islink
from stat import S_IFDIR, S_IFLNK, S_IFREG
import collections
import sys
try:
import _scandir
except ImportError:
_scandir = None
try:
import ctypes
except ImportError:
ctypes = None
if _scandir is None and ctypes is None:
import warnings
warnings.warn("scandir can't find the compiled _scandir C module "
"or ctypes, using slow generic fallback")
__version__ = '1.9.0'
__all__ = ['scandir', 'walk']
# Windows FILE_ATTRIBUTE constants for interpreting the
# FIND_DATA.dwFileAttributes member
FILE_ATTRIBUTE_ARCHIVE = 32
FILE_ATTRIBUTE_COMPRESSED = 2048
FILE_ATTRIBUTE_DEVICE = 64
FILE_ATTRIBUTE_DIRECTORY = 16
FILE_ATTRIBUTE_ENCRYPTED = 16384
FILE_ATTRIBUTE_HIDDEN = 2
FILE_ATTRIBUTE_INTEGRITY_STREAM = 32768
FILE_ATTRIBUTE_NORMAL = 128
FILE_ATTRIBUTE_NOT_CONTENT_INDEXED = 8192
FILE_ATTRIBUTE_NO_SCRUB_DATA = 131072
FILE_ATTRIBUTE_OFFLINE = 4096
FILE_ATTRIBUTE_READONLY = 1
FILE_ATTRIBUTE_REPARSE_POINT = 1024
FILE_ATTRIBUTE_SPARSE_FILE = 512
FILE_ATTRIBUTE_SYSTEM = 4
FILE_ATTRIBUTE_TEMPORARY = 256
FILE_ATTRIBUTE_VIRTUAL = 65536
IS_PY3 = sys.version_info >= (3, 0)
if IS_PY3:
unicode = str # Because Python <= 3.2 doesn't have u'unicode' syntax
class GenericDirEntry(object):
__slots__ = ('name', '_stat', '_lstat', '_scandir_path', '_path')
def __init__(self, scandir_path, name):
self._scandir_path = scandir_path
self.name = name
self._stat = None
self._lstat = None
self._path = None
@property
def path(self):
if self._path is None:
self._path = join(self._scandir_path, self.name)
return self._path
def stat(self, follow_symlinks=True):
if follow_symlinks:
if self._stat is None:
self._stat = stat(self.path)
return self._stat
else:
if self._lstat is None:
self._lstat = lstat(self.path)
return self._lstat
# The code duplication below is intentional: this is for slightly
# better performance on systems that fall back to GenericDirEntry.
# It avoids an additional attribute lookup and method call, which
# are relatively slow on CPython.
def is_dir(self, follow_symlinks=True):
try:
st = self.stat(follow_symlinks=follow_symlinks)
except OSError as e:
if e.errno != ENOENT:
raise
return False # Path doesn't exist or is a broken symlink
return st.st_mode & 0o170000 == S_IFDIR
def is_file(self, follow_symlinks=True):
try:
st = self.stat(follow_symlinks=follow_symlinks)
except OSError as e:
if e.errno != ENOENT:
raise
return False # Path doesn't exist or is a broken symlink
return st.st_mode & 0o170000 == S_IFREG
def is_symlink(self):
try:
st = self.stat(follow_symlinks=False)
except OSError as e:
if e.errno != ENOENT:
raise
return False # Path doesn't exist or is a broken symlink
return st.st_mode & 0o170000 == S_IFLNK
def inode(self):
st = self.stat(follow_symlinks=False)
return st.st_ino
def __str__(self):
return '<{0}: {1!r}>'.format(self.__class__.__name__, self.name)
__repr__ = __str__
def _scandir_generic(path=unicode('.')):
"""Like os.listdir(), but yield DirEntry objects instead of returning
a list of names.
"""
for name in listdir(path):
yield GenericDirEntry(path, name)
if IS_PY3 and sys.platform == 'win32':
def scandir_generic(path=unicode('.')):
if isinstance(path, bytes):
raise TypeError("os.scandir() doesn't support bytes path on Windows, use Unicode instead")
return _scandir_generic(path)
scandir_generic.__doc__ = _scandir_generic.__doc__
else:
scandir_generic = _scandir_generic
scandir_c = None
scandir_python = None
if sys.platform == 'win32':
if ctypes is not None:
from ctypes import wintypes
# Various constants from windows.h
INVALID_HANDLE_VALUE = ctypes.c_void_p(-1).value
ERROR_FILE_NOT_FOUND = 2
ERROR_NO_MORE_FILES = 18
IO_REPARSE_TAG_SYMLINK = 0xA000000C
# Numer of seconds between 1601-01-01 and 1970-01-01
SECONDS_BETWEEN_EPOCHS = 11644473600
kernel32 = ctypes.windll.kernel32
# ctypes wrappers for (wide string versions of) FindFirstFile,
# FindNextFile, and FindClose
FindFirstFile = kernel32.FindFirstFileW
FindFirstFile.argtypes = [
wintypes.LPCWSTR,
ctypes.POINTER(wintypes.WIN32_FIND_DATAW),
]
FindFirstFile.restype = wintypes.HANDLE
FindNextFile = kernel32.FindNextFileW
FindNextFile.argtypes = [
wintypes.HANDLE,
ctypes.POINTER(wintypes.WIN32_FIND_DATAW),
]
FindNextFile.restype = wintypes.BOOL
FindClose = kernel32.FindClose
FindClose.argtypes = [wintypes.HANDLE]
FindClose.restype = wintypes.BOOL
Win32StatResult = collections.namedtuple('Win32StatResult', [
'st_mode',
'st_ino',
'st_dev',
'st_nlink',
'st_uid',
'st_gid',
'st_size',
'st_atime',
'st_mtime',
'st_ctime',
'st_atime_ns',
'st_mtime_ns',
'st_ctime_ns',
'st_file_attributes',
])
def filetime_to_time(filetime):
"""Convert Win32 FILETIME to time since Unix epoch in seconds."""
total = filetime.dwHighDateTime << 32 | filetime.dwLowDateTime
return total / 10000000 - SECONDS_BETWEEN_EPOCHS
def find_data_to_stat(data):
"""Convert Win32 FIND_DATA struct to stat_result."""
# First convert Win32 dwFileAttributes to st_mode
attributes = data.dwFileAttributes
st_mode = 0
if attributes & FILE_ATTRIBUTE_DIRECTORY:
st_mode |= S_IFDIR | 0o111
else:
st_mode |= S_IFREG
if attributes & FILE_ATTRIBUTE_READONLY:
st_mode |= 0o444
else:
st_mode |= 0o666
if (attributes & FILE_ATTRIBUTE_REPARSE_POINT and
data.dwReserved0 == IO_REPARSE_TAG_SYMLINK):
st_mode ^= st_mode & 0o170000
st_mode |= S_IFLNK
st_size = data.nFileSizeHigh << 32 | data.nFileSizeLow
st_atime = filetime_to_time(data.ftLastAccessTime)
st_mtime = filetime_to_time(data.ftLastWriteTime)
st_ctime = filetime_to_time(data.ftCreationTime)
# Some fields set to zero per CPython's posixmodule.c: st_ino, st_dev,
# st_nlink, st_uid, st_gid
return Win32StatResult(st_mode, 0, 0, 0, 0, 0, st_size,
st_atime, st_mtime, st_ctime,
int(st_atime * 1000000000),
int(st_mtime * 1000000000),
int(st_ctime * 1000000000),
attributes)
class Win32DirEntryPython(object):
__slots__ = ('name', '_stat', '_lstat', '_find_data', '_scandir_path', '_path', '_inode')
def __init__(self, scandir_path, name, find_data):
self._scandir_path = scandir_path
self.name = name
self._stat = None
self._lstat = None
self._find_data = find_data
self._path = None
self._inode = None
@property
def path(self):
if self._path is None:
self._path = join(self._scandir_path, self.name)
return self._path
def stat(self, follow_symlinks=True):
if follow_symlinks:
if self._stat is None:
if self.is_symlink():
# It's a symlink, call link-following stat()
self._stat = stat(self.path)
else:
# Not a symlink, stat is same as lstat value
if self._lstat is None:
self._lstat = find_data_to_stat(self._find_data)
self._stat = self._lstat
return self._stat
else:
if self._lstat is None:
# Lazily convert to stat object, because it's slow
# in Python, and often we only need is_dir() etc
self._lstat = find_data_to_stat(self._find_data)
return self._lstat
def is_dir(self, follow_symlinks=True):
is_symlink = self.is_symlink()
if follow_symlinks and is_symlink:
try:
return self.stat().st_mode & 0o170000 == S_IFDIR
except OSError as e:
if e.errno != ENOENT:
raise
return False
elif is_symlink:
return False
else:
return (self._find_data.dwFileAttributes &
FILE_ATTRIBUTE_DIRECTORY != 0)
def is_file(self, follow_symlinks=True):
is_symlink = self.is_symlink()
if follow_symlinks and is_symlink:
try:
return self.stat().st_mode & 0o170000 == S_IFREG
except OSError as e:
if e.errno != ENOENT:
raise
return False
elif is_symlink:
return False
else:
return (self._find_data.dwFileAttributes &
FILE_ATTRIBUTE_DIRECTORY == 0)
def is_symlink(self):
return (self._find_data.dwFileAttributes &
FILE_ATTRIBUTE_REPARSE_POINT != 0 and
self._find_data.dwReserved0 == IO_REPARSE_TAG_SYMLINK)
def inode(self):
if self._inode is None:
self._inode = lstat(self.path).st_ino
return self._inode
def __str__(self):
return '<{0}: {1!r}>'.format(self.__class__.__name__, self.name)
__repr__ = __str__
def win_error(error, filename):
exc = WindowsError(error, ctypes.FormatError(error))
exc.filename = filename
return exc
def _scandir_python(path=unicode('.')):
"""Like os.listdir(), but yield DirEntry objects instead of returning
a list of names.
"""
# Call FindFirstFile and handle errors
if isinstance(path, bytes):
is_bytes = True
filename = join(path.decode('mbcs', 'strict'), '*.*')
else:
is_bytes = False
filename = join(path, '*.*')
data = wintypes.WIN32_FIND_DATAW()
data_p = ctypes.byref(data)
handle = FindFirstFile(filename, data_p)
if handle == INVALID_HANDLE_VALUE:
error = ctypes.GetLastError()
if error == ERROR_FILE_NOT_FOUND:
# No files, don't yield anything
return
raise win_error(error, path)
# Call FindNextFile in a loop, stopping when no more files
try:
while True:
# Skip '.' and '..' (current and parent directory), but
# otherwise yield (filename, stat_result) tuple
name = data.cFileName
if name not in ('.', '..'):
if is_bytes:
name = name.encode('mbcs', 'replace')
yield Win32DirEntryPython(path, name, data)
data = wintypes.WIN32_FIND_DATAW()
data_p = ctypes.byref(data)
success = FindNextFile(handle, data_p)
if not success:
error = ctypes.GetLastError()
if error == ERROR_NO_MORE_FILES:
break
raise win_error(error, path)
finally:
if not FindClose(handle):
raise win_error(ctypes.GetLastError(), path)
if IS_PY3:
def scandir_python(path=unicode('.')):
if isinstance(path, bytes):
raise TypeError("os.scandir() doesn't support bytes path on Windows, use Unicode instead")
return _scandir_python(path)
scandir_python.__doc__ = _scandir_python.__doc__
else:
scandir_python = _scandir_python
if _scandir is not None:
scandir_c = _scandir.scandir
DirEntry_c = _scandir.DirEntry
if _scandir is not None:
scandir = scandir_c
DirEntry = DirEntry_c
elif ctypes is not None:
scandir = scandir_python
DirEntry = Win32DirEntryPython
else:
scandir = scandir_generic
DirEntry = GenericDirEntry
# Linux, OS X, and BSD implementation
elif sys.platform.startswith(('linux', 'darwin', 'sunos5')) or 'bsd' in sys.platform:
have_dirent_d_type = (sys.platform != 'sunos5')
if ctypes is not None and have_dirent_d_type:
import ctypes.util
DIR_p = ctypes.c_void_p
# Rather annoying how the dirent struct is slightly different on each
# platform. The only fields we care about are d_name and d_type.
class Dirent(ctypes.Structure):
if sys.platform.startswith('linux'):
_fields_ = (
('d_ino', ctypes.c_ulong),
('d_off', ctypes.c_long),
('d_reclen', ctypes.c_ushort),
('d_type', ctypes.c_byte),
('d_name', ctypes.c_char * 256),
)
elif 'openbsd' in sys.platform:
_fields_ = (
('d_ino', ctypes.c_uint64),
('d_off', ctypes.c_uint64),
('d_reclen', ctypes.c_uint16),
('d_type', ctypes.c_uint8),
('d_namlen', ctypes.c_uint8),
('__d_padding', ctypes.c_uint8 * 4),
('d_name', ctypes.c_char * 256),
)
else:
_fields_ = (
('d_ino', ctypes.c_uint32), # must be uint32, not ulong
('d_reclen', ctypes.c_ushort),
('d_type', ctypes.c_byte),
('d_namlen', ctypes.c_byte),
('d_name', ctypes.c_char * 256),
)
DT_UNKNOWN = 0
DT_DIR = 4
DT_REG = 8
DT_LNK = 10
Dirent_p = ctypes.POINTER(Dirent)
Dirent_pp = ctypes.POINTER(Dirent_p)
libc = ctypes.CDLL(ctypes.util.find_library('c'), use_errno=True)
opendir = libc.opendir
opendir.argtypes = [ctypes.c_char_p]
opendir.restype = DIR_p
readdir_r = libc.readdir_r
readdir_r.argtypes = [DIR_p, Dirent_p, Dirent_pp]
readdir_r.restype = ctypes.c_int
closedir = libc.closedir
closedir.argtypes = [DIR_p]
closedir.restype = ctypes.c_int
file_system_encoding = sys.getfilesystemencoding()
class PosixDirEntry(object):
__slots__ = ('name', '_d_type', '_stat', '_lstat', '_scandir_path', '_path', '_inode')
def __init__(self, scandir_path, name, d_type, inode):
self._scandir_path = scandir_path
self.name = name
self._d_type = d_type
self._inode = inode
self._stat = None
self._lstat = None
self._path = None
@property
def path(self):
if self._path is None:
self._path = join(self._scandir_path, self.name)
return self._path
def stat(self, follow_symlinks=True):
if follow_symlinks:
if self._stat is None:
if self.is_symlink():
self._stat = stat(self.path)
else:
if self._lstat is None:
self._lstat = lstat(self.path)
self._stat = self._lstat
return self._stat
else:
if self._lstat is None:
self._lstat = lstat(self.path)
return self._lstat
def is_dir(self, follow_symlinks=True):
if (self._d_type == DT_UNKNOWN or
(follow_symlinks and self.is_symlink())):
try:
st = self.stat(follow_symlinks=follow_symlinks)
except OSError as e:
if e.errno != ENOENT:
raise
return False
return st.st_mode & 0o170000 == S_IFDIR
else:
return self._d_type == DT_DIR
def is_file(self, follow_symlinks=True):
if (self._d_type == DT_UNKNOWN or
(follow_symlinks and self.is_symlink())):
try:
st = self.stat(follow_symlinks=follow_symlinks)
except OSError as e:
if e.errno != ENOENT:
raise
return False
return st.st_mode & 0o170000 == S_IFREG
else:
return self._d_type == DT_REG
def is_symlink(self):
if self._d_type == DT_UNKNOWN:
try:
st = self.stat(follow_symlinks=False)
except OSError as e:
if e.errno != ENOENT:
raise
return False
return st.st_mode & 0o170000 == S_IFLNK
else:
return self._d_type == DT_LNK
def inode(self):
return self._inode
def __str__(self):
return '<{0}: {1!r}>'.format(self.__class__.__name__, self.name)
__repr__ = __str__
def posix_error(filename):
errno = ctypes.get_errno()
exc = OSError(errno, strerror(errno))
exc.filename = filename
return exc
def scandir_python(path=unicode('.')):
"""Like os.listdir(), but yield DirEntry objects instead of returning
a list of names.
"""
if isinstance(path, bytes):
opendir_path = path
is_bytes = True
else:
opendir_path = path.encode(file_system_encoding)
is_bytes = False
dir_p = opendir(opendir_path)
if not dir_p:
raise posix_error(path)
try:
result = Dirent_p()
while True:
entry = Dirent()
if readdir_r(dir_p, entry, result):
raise posix_error(path)
if not result:
break
name = entry.d_name
if name not in (b'.', b'..'):
if not is_bytes:
name = name.decode(file_system_encoding)
yield PosixDirEntry(path, name, entry.d_type, entry.d_ino)
finally:
if closedir(dir_p):
raise posix_error(path)
if _scandir is not None:
scandir_c = _scandir.scandir
DirEntry_c = _scandir.DirEntry
if _scandir is not None:
scandir = scandir_c
DirEntry = DirEntry_c
elif ctypes is not None:
scandir = scandir_python
DirEntry = PosixDirEntry
else:
scandir = scandir_generic
DirEntry = GenericDirEntry
# Some other system -- no d_type or stat information
else:
scandir = scandir_generic
DirEntry = GenericDirEntry
def _walk(top, topdown=True, onerror=None, followlinks=False):
"""Like Python 3.5's implementation of os.walk() -- faster than
the pre-Python 3.5 version as it uses scandir() internally.
"""
dirs = []
nondirs = []
# We may not have read permission for top, in which case we can't
# get a list of the files the directory contains. os.walk
# always suppressed the exception then, rather than blow up for a
# minor reason when (say) a thousand readable directories are still
# left to visit. That logic is copied here.
try:
scandir_it = scandir(top)
except OSError as error:
if onerror is not None:
onerror(error)
return
while True:
try:
try:
entry = next(scandir_it)
except StopIteration:
break
except OSError as error:
if onerror is not None:
onerror(error)
return
try:
is_dir = entry.is_dir()
except OSError:
# If is_dir() raises an OSError, consider that the entry is not
# a directory, same behaviour than os.path.isdir().
is_dir = False
if is_dir:
dirs.append(entry.name)
else:
nondirs.append(entry.name)
if not topdown and is_dir:
# Bottom-up: recurse into sub-directory, but exclude symlinks to
# directories if followlinks is False
if followlinks:
walk_into = True
else:
try:
is_symlink = entry.is_symlink()
except OSError:
# If is_symlink() raises an OSError, consider that the
# entry is not a symbolic link, same behaviour than
# os.path.islink().
is_symlink = False
walk_into = not is_symlink
if walk_into:
for entry in walk(entry.path, topdown, onerror, followlinks):
yield entry
# Yield before recursion if going top down
if topdown:
yield top, dirs, nondirs
# Recurse into sub-directories
for name in dirs:
new_path = join(top, name)
# Issue #23605: os.path.islink() is used instead of caching
# entry.is_symlink() result during the loop on os.scandir() because
# the caller can replace the directory entry during the "yield"
# above.
if followlinks or not islink(new_path):
for entry in walk(new_path, topdown, onerror, followlinks):
yield entry
else:
# Yield after recursion if going bottom up
yield top, dirs, nondirs
if IS_PY3 or sys.platform != 'win32':
walk = _walk
else:
# Fix for broken unicode handling on Windows on Python 2.x, see:
# https://github.com/benhoyt/scandir/issues/54
file_system_encoding = sys.getfilesystemencoding()
def walk(top, topdown=True, onerror=None, followlinks=False):
if isinstance(top, bytes):
top = top.decode(file_system_encoding)
return _walk(top, topdown, onerror, followlinks)

4
third_party/python/scandir/setup.cfg vendored Normal file
View File

@ -0,0 +1,4 @@
[egg_info]
tag_build =
tag_date = 0

80
third_party/python/scandir/setup.py vendored Normal file
View File

@ -0,0 +1,80 @@
"""Run "python setup.py install" to install scandir."""
try:
from setuptools import setup, Extension
from setuptools.command.build_ext import build_ext as base_build_ext
except ImportError:
import warnings
import sys
val = sys.exc_info()[1]
warnings.warn("import of setuptools failed %r" % val)
from distutils.core import setup, Extension
from distutils.command.build_ext import build_ext as base_build_ext
import os
import re
import sys
import logging
# Get version without importing scandir because that will lock the
# .pyd file (if scandir is already installed) so it can't be
# overwritten during the install process
with open(os.path.join(os.path.dirname(__file__), 'scandir.py')) as f:
for line in f:
match = re.match(r"__version__.*'([0-9.]+)'", line)
if match:
version = match.group(1)
break
else:
raise Exception("Couldn't find version in setup.py")
with open('README.rst') as f:
long_description = f.read()
class BuildExt(base_build_ext):
# the extension is optional since in case of lack of c the api
# there is a ctypes fallback and a slow python fallback
def build_extension(self, ext):
try:
base_build_ext.build_extension(self, ext)
except Exception:
exception = sys.exc_info()[0]
logging.warn("building the %s failed with %s", ext.name, exception)
extension = Extension('_scandir', ['_scandir.c'], optional=True)
setup(
name='scandir',
version=version,
author='Ben Hoyt',
author_email='benhoyt@gmail.com',
url='https://github.com/benhoyt/scandir',
license='New BSD License',
description='scandir, a better directory iterator and faster os.walk()',
long_description=long_description,
py_modules=['scandir'],
ext_modules=[extension],
classifiers=[
'Development Status :: 5 - Production/Stable',
'Intended Audience :: Developers',
'Operating System :: OS Independent',
'License :: OSI Approved :: BSD License',
'Programming Language :: Python',
'Topic :: System :: Filesystems',
'Topic :: System :: Operating System',
'Programming Language :: Python',
'Programming Language :: Python :: 2',
'Programming Language :: Python :: 2.6',
'Programming Language :: Python :: 2.7',
'Programming Language :: Python :: 3',
'Programming Language :: Python :: 3.4',
'Programming Language :: Python :: 3.5',
'Programming Language :: Python :: 3.6',
'Programming Language :: Python :: Implementation :: CPython',
], cmdclass={'build_ext': BuildExt},
)

25
third_party/python/scandir/test/run_tests.py vendored Normal file
View File

@ -0,0 +1,25 @@
"""Run all unit tests."""
import glob
import os
import sys
if sys.version_info[:2] < (2, 7):
import unittest2 as unittest
else:
import unittest
def main():
test_dir = os.path.dirname(os.path.abspath(__file__))
test_files = glob.glob(os.path.join(test_dir, 'test_*.py'))
test_names = [os.path.basename(f)[:-3] for f in test_files]
sys.path.insert(0, os.path.join(test_dir, '..'))
suite = unittest.defaultTestLoader.loadTestsFromNames(test_names)
result = unittest.TextTestRunner(verbosity=2).run(suite)
sys.exit(1 if (result.errors or result.failures) else 0)
if __name__ == '__main__':
main()

320
third_party/python/scandir/test/test_scandir.py vendored Normal file
View File

@ -0,0 +1,320 @@
"""Tests for scandir.scandir()."""
from __future__ import unicode_literals
import os
import shutil
import sys
import time
if sys.version_info[:2] < (2, 7):
import unittest2 as unittest
else:
import unittest
try:
import scandir
has_scandir = True
except ImportError:
has_scandir = False
FILE_ATTRIBUTE_DIRECTORY = 16
TEST_PATH = os.path.abspath(os.path.join(os.path.dirname(__file__), 'testdir'))
IS_PY3 = sys.version_info >= (3, 0)
if IS_PY3:
int_types = int
else:
int_types = (int, long)
str = unicode
if hasattr(os, 'symlink'):
try:
link_name = os.path.join(os.path.dirname(__file__), '_testlink')
os.symlink(__file__, link_name)
os.remove(link_name)
symlinks_supported = True
except NotImplementedError:
# Windows versions before Vista don't support symbolic links
symlinks_supported = False
else:
symlinks_supported = False
def create_file(path, contents='1234'):
with open(path, 'w') as f:
f.write(contents)
def setup_main():
join = os.path.join
os.mkdir(TEST_PATH)
os.mkdir(join(TEST_PATH, 'subdir'))
create_file(join(TEST_PATH, 'file1.txt'))
create_file(join(TEST_PATH, 'file2.txt'), contents='12345678')
os.mkdir(join(TEST_PATH, 'subdir', 'unidir\u018F'))
create_file(join(TEST_PATH, 'subdir', 'file1.txt'))
create_file(join(TEST_PATH, 'subdir', 'unicod\u018F.txt'))
create_file(join(TEST_PATH, 'subdir', 'unidir\u018F', 'file1.txt'))
os.mkdir(join(TEST_PATH, 'linkdir'))
def setup_symlinks():
join = os.path.join
os.mkdir(join(TEST_PATH, 'linkdir', 'linksubdir'))
create_file(join(TEST_PATH, 'linkdir', 'file1.txt'))
os.symlink(os.path.abspath(join(TEST_PATH, 'linkdir', 'file1.txt')),
join(TEST_PATH, 'linkdir', 'link_to_file'))
dir_name = os.path.abspath(join(TEST_PATH, 'linkdir', 'linksubdir'))
dir_link = join(TEST_PATH, 'linkdir', 'link_to_dir')
if sys.version_info >= (3, 3):
# "target_is_directory" was only added in Python 3.3
os.symlink(dir_name, dir_link, target_is_directory=True)
else:
os.symlink(dir_name, dir_link)
def teardown():
try:
shutil.rmtree(TEST_PATH)
except OSError:
# why does the above fail sometimes?
time.sleep(0.1)
shutil.rmtree(TEST_PATH)
class TestMixin(object):
def setUp(self):
if not os.path.exists(TEST_PATH):
setup_main()
if symlinks_supported and not os.path.exists(
os.path.join(TEST_PATH, 'linkdir', 'linksubdir')):
setup_symlinks()
if not hasattr(unittest.TestCase, 'skipTest'):
def skipTest(self, reason):
sys.stdout.write('skipped {0!r} '.format(reason))
def test_basic(self):
entries = sorted(self.scandir_func(TEST_PATH), key=lambda e: e.name)
self.assertEqual([(e.name, e.is_dir()) for e in entries],
[('file1.txt', False), ('file2.txt', False),
('linkdir', True), ('subdir', True)])
self.assertEqual([e.path for e in entries],
[os.path.join(TEST_PATH, e.name) for e in entries])
def test_dir_entry(self):
entries = dict((e.name, e) for e in self.scandir_func(TEST_PATH))
e = entries['file1.txt']
self.assertEqual([e.is_dir(), e.is_file(), e.is_symlink()], [False, True, False])
e = entries['file2.txt']
self.assertEqual([e.is_dir(), e.is_file(), e.is_symlink()], [False, True, False])
e = entries['subdir']
self.assertEqual([e.is_dir(), e.is_file(), e.is_symlink()], [True, False, False])
self.assertEqual(entries['file1.txt'].stat().st_size, 4)
self.assertEqual(entries['file2.txt'].stat().st_size, 8)
def test_stat(self):
entries = list(self.scandir_func(TEST_PATH))
for entry in entries:
os_stat = os.stat(os.path.join(TEST_PATH, entry.name))
scandir_stat = entry.stat()
self.assertEqual(os_stat.st_mode, scandir_stat.st_mode)
# TODO: be nice to figure out why these aren't identical on Windows and on PyPy
# * Windows: they seem to be a few microseconds to tens of seconds out
# * PyPy: for some reason os_stat's times are nanosecond, scandir's are not
self.assertAlmostEqual(os_stat.st_mtime, scandir_stat.st_mtime, delta=1)
self.assertAlmostEqual(os_stat.st_ctime, scandir_stat.st_ctime, delta=1)
if entry.is_file():
self.assertEqual(os_stat.st_size, scandir_stat.st_size)
def test_returns_iter(self):
it = self.scandir_func(TEST_PATH)
entry = next(it)
assert hasattr(entry, 'name')
def check_file_attributes(self, result):
self.assertTrue(hasattr(result, 'st_file_attributes'))
self.assertTrue(isinstance(result.st_file_attributes, int_types))
self.assertTrue(0 <= result.st_file_attributes <= 0xFFFFFFFF)
def test_file_attributes(self):
if sys.platform != 'win32' or not self.has_file_attributes:
# st_file_attributes is Win32 specific (but can't use
# unittest.skipUnless on Python 2.6)
return self.skipTest('st_file_attributes not supported')
entries = dict((e.name, e) for e in self.scandir_func(TEST_PATH))
# test st_file_attributes on a file (FILE_ATTRIBUTE_DIRECTORY not set)
result = entries['file1.txt'].stat()
self.check_file_attributes(result)
self.assertEqual(result.st_file_attributes & FILE_ATTRIBUTE_DIRECTORY, 0)
# test st_file_attributes on a directory (FILE_ATTRIBUTE_DIRECTORY set)
result = entries['subdir'].stat()
self.check_file_attributes(result)
self.assertEqual(result.st_file_attributes & FILE_ATTRIBUTE_DIRECTORY,
FILE_ATTRIBUTE_DIRECTORY)
def test_path(self):
entries = sorted(self.scandir_func(TEST_PATH), key=lambda e: e.name)
self.assertEqual([os.path.basename(e.name) for e in entries],
['file1.txt', 'file2.txt', 'linkdir', 'subdir'])
self.assertEqual([os.path.normpath(os.path.join(TEST_PATH, e.name)) for e in entries],
[os.path.normpath(e.path) for e in entries])
def test_symlink(self):
if not symlinks_supported:
return self.skipTest('symbolic links not supported')
entries = sorted(self.scandir_func(os.path.join(TEST_PATH, 'linkdir')),
key=lambda e: e.name)
self.assertEqual([(e.name, e.is_symlink()) for e in entries],
[('file1.txt', False),
('link_to_dir', True),
('link_to_file', True),
('linksubdir', False)])
self.assertEqual([(e.name, e.is_file(), e.is_file(follow_symlinks=False))
for e in entries],
[('file1.txt', True, True),
('link_to_dir', False, False),
('link_to_file', True, False),
('linksubdir', False, False)])
self.assertEqual([(e.name, e.is_dir(), e.is_dir(follow_symlinks=False))
for e in entries],
[('file1.txt', False, False),
('link_to_dir', True, False),
('link_to_file', False, False),
('linksubdir', True, True)])
def test_bytes(self):
# Check that unicode filenames are returned correctly as bytes in output
path = os.path.join(TEST_PATH, 'subdir').encode(sys.getfilesystemencoding(), 'replace')
self.assertTrue(isinstance(path, bytes))
# Python 3.6 on Windows fixes the bytes filename thing by using UTF-8
if IS_PY3 and sys.platform == 'win32':
if not (sys.version_info >= (3, 6) and self.scandir_func == os.scandir):
self.assertRaises(TypeError, self.scandir_func, path)
return
entries = [e for e in self.scandir_func(path) if e.name.startswith(b'unicod')]
self.assertEqual(len(entries), 1)
entry = entries[0]
self.assertTrue(isinstance(entry.name, bytes))
self.assertTrue(isinstance(entry.path, bytes))
# b'unicod?.txt' on Windows, b'unicod\xc6\x8f.txt' (UTF-8) or similar on POSIX
entry_name = 'unicod\u018f.txt'.encode(sys.getfilesystemencoding(), 'replace')
self.assertEqual(entry.name, entry_name)
self.assertEqual(entry.path, os.path.join(path, entry_name))
def test_unicode(self):
# Check that unicode filenames are returned correctly as (unicode) str in output
path = os.path.join(TEST_PATH, 'subdir')
if not IS_PY3:
path = path.decode(sys.getfilesystemencoding(), 'replace')
self.assertTrue(isinstance(path, str))
entries = [e for e in self.scandir_func(path) if e.name.startswith('unicod')]
self.assertEqual(len(entries), 1)
entry = entries[0]
self.assertTrue(isinstance(entry.name, str))
self.assertTrue(isinstance(entry.path, str))
entry_name = 'unicod\u018f.txt'
self.assertEqual(entry.name, entry_name)
self.assertEqual(entry.path, os.path.join(path, 'unicod\u018f.txt'))
# Check that it handles unicode input properly
path = os.path.join(TEST_PATH, 'subdir', 'unidir\u018f')
self.assertTrue(isinstance(path, str))
entries = list(self.scandir_func(path))
self.assertEqual(len(entries), 1)
entry = entries[0]
self.assertTrue(isinstance(entry.name, str))
self.assertTrue(isinstance(entry.path, str))
self.assertEqual(entry.name, 'file1.txt')
self.assertEqual(entry.path, os.path.join(path, 'file1.txt'))
def test_walk_unicode_handling(self):
encoding = sys.getfilesystemencoding()
dirname_unicode = u'test_unicode_dir'
dirname_bytes = dirname_unicode.encode(encoding)
dirpath = os.path.join(TEST_PATH.encode(encoding), dirname_bytes)
try:
os.makedirs(dirpath)
if sys.platform != 'win32':
# test bytes
self.assertTrue(isinstance(dirpath, bytes))
for (path, dirs, files) in scandir.walk(dirpath):
self.assertTrue(isinstance(path, bytes))
# test unicode
text_type = str if IS_PY3 else unicode
dirpath_unicode = text_type(dirpath, encoding)
self.assertTrue(isinstance(dirpath_unicode, text_type))
for (path, dirs, files) in scandir.walk(dirpath_unicode):
self.assertTrue(isinstance(path, text_type))
finally:
shutil.rmtree(dirpath)
if has_scandir:
class TestScandirGeneric(TestMixin, unittest.TestCase):
def setUp(self):
self.scandir_func = scandir.scandir_generic
self.has_file_attributes = False
TestMixin.setUp(self)
if getattr(scandir, 'scandir_python', None):
class TestScandirPython(TestMixin, unittest.TestCase):
def setUp(self):
self.scandir_func = scandir.scandir_python
self.has_file_attributes = True
TestMixin.setUp(self)
if getattr(scandir, 'scandir_c', None):
class TestScandirC(TestMixin, unittest.TestCase):
def setUp(self):
self.scandir_func = scandir.scandir_c
self.has_file_attributes = True
TestMixin.setUp(self)
class TestScandirDirEntry(unittest.TestCase):
def setUp(self):
if not os.path.exists(TEST_PATH):
setup_main()
def test_iter_returns_dir_entry(self):
it = scandir.scandir(TEST_PATH)
entry = next(it)
assert isinstance(entry, scandir.DirEntry)
if hasattr(os, 'scandir'):
class TestScandirOS(TestMixin, unittest.TestCase):
def setUp(self):
self.scandir_func = os.scandir
self.has_file_attributes = True
TestMixin.setUp(self)

213
third_party/python/scandir/test/test_walk.py vendored Normal file
View File

@ -0,0 +1,213 @@
"""Tests for scandir.walk(), copied from CPython's tests for os.walk()."""
import os
import shutil
import sys
if sys.version_info[:2] < (2, 7):
import unittest2 as unittest
else:
import unittest
import scandir
walk_func = scandir.walk
class TestWalk(unittest.TestCase):
testfn = os.path.join(os.path.dirname(__file__), 'temp')
def test_traversal(self):
# Build:
# TESTFN/
# TEST1/ a file kid and two directory kids
# tmp1
# SUB1/ a file kid and a directory kid
# tmp2
# SUB11/ no kids
# SUB2/ a file kid and a dirsymlink kid
# tmp3
# link/ a symlink to TESTFN.2
# TEST2/
# tmp4 a lone file
walk_path = os.path.join(self.testfn, "TEST1")
sub1_path = os.path.join(walk_path, "SUB1")
sub11_path = os.path.join(sub1_path, "SUB11")
sub2_path = os.path.join(walk_path, "SUB2")
tmp1_path = os.path.join(walk_path, "tmp1")
tmp2_path = os.path.join(sub1_path, "tmp2")
tmp3_path = os.path.join(sub2_path, "tmp3")
link_path = os.path.join(sub2_path, "link")
t2_path = os.path.join(self.testfn, "TEST2")
tmp4_path = os.path.join(self.testfn, "TEST2", "tmp4")
# Create stuff.
os.makedirs(sub11_path)
os.makedirs(sub2_path)
os.makedirs(t2_path)
for path in tmp1_path, tmp2_path, tmp3_path, tmp4_path:
f = open(path, "w")
f.write("I'm " + path + " and proud of it. Blame test_os.\n")
f.close()
has_symlink = hasattr(os, "symlink")
if has_symlink:
try:
if sys.platform == 'win32' and sys.version_info >= (3, 2):
# "target_is_directory" was only added in Python 3.2 (on Windows)
os.symlink(os.path.abspath(t2_path), link_path, target_is_directory=True)
else:
os.symlink(os.path.abspath(t2_path), link_path)
sub2_tree = (sub2_path, ["link"], ["tmp3"])
except NotImplementedError:
sub2_tree = (sub2_path, [], ["tmp3"])
else:
sub2_tree = (sub2_path, [], ["tmp3"])
# Walk top-down.
all = list(walk_func(walk_path))
self.assertEqual(len(all), 4)
# We can't know which order SUB1 and SUB2 will appear in.
# Not flipped: TESTFN, SUB1, SUB11, SUB2
# flipped: TESTFN, SUB2, SUB1, SUB11
flipped = all[0][1][0] != "SUB1"
all[0][1].sort()
self.assertEqual(all[0], (walk_path, ["SUB1", "SUB2"], ["tmp1"]))
self.assertEqual(all[1 + flipped], (sub1_path, ["SUB11"], ["tmp2"]))
self.assertEqual(all[2 + flipped], (sub11_path, [], []))
self.assertEqual(all[3 - 2 * flipped], sub2_tree)
# Prune the search.
all = []
for root, dirs, files in walk_func(walk_path):
all.append((root, dirs, files))
# Don't descend into SUB1.
if 'SUB1' in dirs:
# Note that this also mutates the dirs we appended to all!
dirs.remove('SUB1')
self.assertEqual(len(all), 2)
self.assertEqual(all[0], (walk_path, ["SUB2"], ["tmp1"]))
self.assertEqual(all[1], sub2_tree)
# Walk bottom-up.
all = list(walk_func(walk_path, topdown=False))
self.assertEqual(len(all), 4)
# We can't know which order SUB1 and SUB2 will appear in.
# Not flipped: SUB11, SUB1, SUB2, TESTFN
# flipped: SUB2, SUB11, SUB1, TESTFN
flipped = all[3][1][0] != "SUB1"
all[3][1].sort()
self.assertEqual(all[3], (walk_path, ["SUB1", "SUB2"], ["tmp1"]))
self.assertEqual(all[flipped], (sub11_path, [], []))
self.assertEqual(all[flipped + 1], (sub1_path, ["SUB11"], ["tmp2"]))
self.assertEqual(all[2 - 2 * flipped], sub2_tree)
if has_symlink:
# Walk, following symlinks.
for root, dirs, files in walk_func(walk_path, followlinks=True):
if root == link_path:
self.assertEqual(dirs, [])
self.assertEqual(files, ["tmp4"])
break
else:
self.fail("Didn't follow symlink with followlinks=True")
# Test creating a directory and adding it to dirnames
sub3_path = os.path.join(walk_path, "SUB3")
all = []
for root, dirs, files in walk_func(walk_path):
all.append((root, dirs, files))
if 'SUB1' in dirs:
os.makedirs(sub3_path)
dirs.append('SUB3')
all.sort()
self.assertEqual(os.path.split(all[-1][0])[1], 'SUB3')
def tearDown(self):
# Tear everything down. This is a decent use for bottom-up on
# Windows, which doesn't have a recursive delete command. The
# (not so) subtlety is that rmdir will fail unless the dir's
# kids are removed first, so bottom up is essential.
for root, dirs, files in os.walk(self.testfn, topdown=False):
for name in files:
os.remove(os.path.join(root, name))
for name in dirs:
dirname = os.path.join(root, name)
if not os.path.islink(dirname):
os.rmdir(dirname)
else:
os.remove(dirname)
os.rmdir(self.testfn)
class TestWalkSymlink(unittest.TestCase):
temp_dir = os.path.join(os.path.dirname(__file__), 'temp')
def setUp(self):
os.mkdir(self.temp_dir)
self.dir_name = os.path.join(self.temp_dir, 'dir')
os.mkdir(self.dir_name)
open(os.path.join(self.dir_name, 'subfile'), 'w').close()
self.file_name = os.path.join(self.temp_dir, 'file')
open(self.file_name, 'w').close()
def tearDown(self):
shutil.rmtree(self.temp_dir)
def test_symlink_to_file(self):
if not hasattr(os, 'symlink'):
return
try:
os.symlink(self.file_name, os.path.join(self.temp_dir,
'link_to_file'))
except NotImplementedError:
# Windows versions before Vista don't support symbolic links
return
output = sorted(walk_func(self.temp_dir))
dirs = sorted(output[0][1])
files = sorted(output[0][2])
self.assertEqual(dirs, ['dir'])
self.assertEqual(files, ['file', 'link_to_file'])
self.assertEqual(len(output), 2)
self.assertEqual(output[1][1], [])
self.assertEqual(output[1][2], ['subfile'])
def test_symlink_to_directory(self):
if not hasattr(os, 'symlink'):
return
link_name = os.path.join(self.temp_dir, 'link_to_dir')
try:
if sys.platform == 'win32' and sys.version_info >= (3, 2):
# "target_is_directory" was only added in Python 3.2 (on Windows)
os.symlink(self.dir_name, link_name, target_is_directory=True)
else:
os.symlink(self.dir_name, link_name)
except NotImplementedError:
# Windows versions before Vista don't support symbolic links
return
output = sorted(walk_func(self.temp_dir))
dirs = sorted(output[0][1])
files = sorted(output[0][2])
self.assertEqual(dirs, ['dir', 'link_to_dir'])
self.assertEqual(files, ['file'])
self.assertEqual(len(output), 2)
self.assertEqual(output[1][1], [])
self.assertEqual(output[1][2], ['subfile'])
output = sorted(walk_func(self.temp_dir, followlinks=True))
dirs = sorted(output[0][1])
files = sorted(output[0][2])
self.assertEqual(dirs, ['dir', 'link_to_dir'])
self.assertEqual(files, ['file'])
self.assertEqual(len(output), 3)
self.assertEqual(output[1][1], [])
self.assertEqual(output[1][2], ['subfile'])
self.assertEqual(os.path.basename(output[2][0]), 'link_to_dir')
self.assertEqual(output[2][1], [])
self.assertEqual(output[2][2], ['subfile'])

53
third_party/python/scandir/winreparse.h vendored Normal file
View File

@ -0,0 +1,53 @@
#ifndef Py_WINREPARSE_H
#define Py_WINREPARSE_H
#ifdef MS_WINDOWS
#include <Windows.h>
#ifdef __cplusplus
extern "C" {
#endif
/* The following structure was copied from
http://msdn.microsoft.com/en-us/library/ff552012.aspx as the required
include doesn't seem to be present in the Windows SDK (at least as included
with Visual Studio Express). */
typedef struct _REPARSE_DATA_BUFFER {
ULONG ReparseTag;
USHORT ReparseDataLength;
USHORT Reserved;
union {
struct {
USHORT SubstituteNameOffset;
USHORT SubstituteNameLength;
USHORT PrintNameOffset;
USHORT PrintNameLength;
ULONG Flags;
WCHAR PathBuffer[1];
} SymbolicLinkReparseBuffer;
struct {
USHORT SubstituteNameOffset;
USHORT SubstituteNameLength;
USHORT PrintNameOffset;
USHORT PrintNameLength;
WCHAR PathBuffer[1];
} MountPointReparseBuffer;
struct {
UCHAR DataBuffer[1];
} GenericReparseBuffer;
};
} REPARSE_DATA_BUFFER, *PREPARSE_DATA_BUFFER;
#define REPARSE_DATA_BUFFER_HEADER_SIZE FIELD_OFFSET(REPARSE_DATA_BUFFER,\
GenericReparseBuffer)
#define MAXIMUM_REPARSE_DATA_BUFFER_SIZE ( 16 * 1024 )
#ifdef __cplusplus
}
#endif
#endif /* MS_WINDOWS */
#endif /* !Py_WINREPARSE_H */