This patch was taken from
https://github.com/yaml/pyyaml/issues/369#issuecomment-571596545,
authored by Pekka Klärck <peke@iki.fi>.
In short, Jython doesn't support lone surrogates, so importing yaml (and
in particular, loading `reader.py`) caused a UnicodeDecodeError. This
patch works around this through a clever use of `eval` to defer
evaluation of the string containing the lone surrogates, only doing it
on non-Jython platforms.
This is only done in `lib/yaml/reader.py` and not `lib3/yaml/reader.py`
because Jython does not support Python 3.
With this patch, Jython's behavior with respect to Unicode code points
over 0xFFFF becomes as it was before
0716ae21a1. It still does not pass all the
unit tests on Jython (passes 1275, fails 3, errors on 1); all the
failing tests are related to unicode. Still, this is better than simply
crashing upon `import yaml`.
With this patch, all tests continue to pass on Python 2 / Python 3.
Repeated calls to `resolve` can experience performance degredation, if
`add_implicit_resolver` has been called with `first=None` (to add an
implicit resolver with an unspecified first character).
For example, every time `foo` is encountered, the "wildcard implicit
resolvers" (with `first=None`) will be appended to the list of implicit
resolvers for strings starting with `f`, which will normally be the
resolver for booleans. The list `yaml_implicit_resolvers['f']` will keep
getting longer. The same behavior applies for any first-letter matches
with existing implicit resolvers.
This change avoids unintentionally mutating the lists in the class-level
dict `yaml_implicit_resolvers` by looping through a temporary copy.
Fixes: #439
* Move most CI to GitHub Actions
* Build sdist
* Build manylinux1 wheels with libyaml ext (also tested with 2010 and 2014)
* Build MacOS x86_64 wheels with libyaml ext
* Windows wheel builds remain on AppVeyor until we drop 2.7 support in 6.0
* Smoke tests of all post-build artifacts
* Add PEP517/518 build declaration (pyproject.toml with setuptools backend)
* Fully move build to setuptools
* Drop Python 3.5 support
* Declare Python 3.9 support
* Update PyPI metadata now that setuptools lets it flow through
Co-authored-by: Matt Davis <mrd@redhat.com>
* Prevents arbitrary code execution during python/object/new constructor
In FullLoader python/object/new constructor, implemented by
construct_python_object_apply, has support for setting the state of a
deserialized instance through the set_python_instance_state method.
After setting the state, some operations are performed on the instance
to complete its initialization, however it is possible for an attacker
to set the instance' state in such a way that arbitrary code is executed
by the FullLoader.
This patch tries to block such attacks in FullLoader by preventing
set_python_instance_state from setting arbitrary properties. It
implements a blacklist that includes `extend` method (called by
construct_python_object_apply) and all special methods (e.g. __set__,
__setitem__, etc.).
Users who need special attributes being set in the state of a
deserialized object can still do it through the UnsafeLoader, which
however should not be used on untrusted input. Additionally, they can
subclass FullLoader and redefine `get_state_keys_blacklist()` to
extend/replace the list of blacklisted keys, passing the subclassed
loader to yaml.load.
* Make sure python/object/new constructor does not set some properties
* Add test to show how to subclass FullLoader with new blacklist
* On load, now use aware datetimes if possible
On loading data, if timestamps have an ISO "+HH:MM" UTC offset then the resultant datetime is converted to UTC. This change adds that timezone information to the datetime objects.
Importantly, this addresses a Django warning (and potential error) that appears when using both YAML fixtures in a timezone-aware project. It was raised as a Django issue (https://code.djangoproject.com/ticket/18867), but subsequently closed because the Django devs felt that this is a PyYAML problem.
* Create timezone-aware datetime in timezone from data
* Create timezone-aware datetime in timezone from data for python2
* Define better timezone implementation for python2
* Handle timezone "Z" for python 3
* Handle timezone "Z" for python 2
* Fix code structure for Python 3
Call datetime.datetime constructor once at return.
* Fix code structure for Python 2
Call datetime.datetime constructor once at return.
* remove some unused imports
as suggested by lgtm
https://lgtm.com/projects/g/yaml/pyyaml/
* add back import * from nodes
* remove also sys import
* remove mkpath import
This patch fixes the handling of inputs with Unicode code points over
0xFFFF when running on a Python 2 that does not have UCS-4 support
(which certain distributions still ship, e.g. macOS).
When Python is compiled without UCS-4 support, it uses UCS-2. In this
situation, non-BMP Unicode characters, which have code points over
0xFFFF, are represented as surrogate pairs. For example, if we take
u'\U0001f3d4', it will be represented as the surrogate pair
u'\ud83c\udfd4'. This can be seen by running, for example:
[i for i in u'\U0001f3d4']
In PyYAML, the reader uses a function `check_printable` to validate
inputs, making sure that they only contain printable characters. Prior
to this patch, on UCS-2 builds, it incorrectly identified surrogate
pairs as non-printable.
It would be fairly natural to write a regular expression that captures
strings that contain only *printable* characters, as opposed to
*non-printable* characters (as identified by the old code, so not
excluding surrogate pairs):
PRINTABLE = re.compile(u'^[\x09\x0A\x0D\x20-\x7E\x85\xA0-\uD7FF\uE000-\uFFFD]*$')
Adding support for surrogate pairs to this would be straightforward,
adding the option of having a surrogate high followed by a surrogate low
(`[\uD800-\uDBFF][\uDC00-\uDFFF]`):
PRINTABLE = re.compile(u'^(?:[\x09\x0A\x0D\x20-\x7E\x85\xA0-\uD7FF\uE000-\uFFFD]|[\uD800-\uDBFF][\uDC00-\uDFFF])*$')
Then, this regex could be used as follows:
def check_printable(self, data):
if not self.PRINTABLE.match(data):
raise ReaderError(...)
However, matching printable strings, rather than searching for
non-printable characters as the code currently does, would have the
disadvantage of not identifying the culprit character (we wouldn't get
the position and the actual non-printable character from a lack of a
regex match).
Instead, we can modify the NON_PRINTABLE regex to allow legal surrogate
pairs. We do this by removing surrogate pairs from the existing
character set and adding the following options for illegal uses of
surrogate code points:
- Surrogate low that doesn't follow a surrogate high (either a surrogate
low at the start of a string, or a surrogate low that follows a
character that's not a surrogate high):
(?:^|[^\uD800-\uDBFF])[\uDC00-\uDFFF]
- Surrogate high that isn't followed by a surrogate low (either a
surrogate high at the end of a string, or a surrogate high that is
followed by a character that's not a surrogate low):
[\uD800-\uDBFF](?:[^\uDC00-\uDFFF]|$)
The behavior of this modified regex should match the one that is used
when Python is built with UCS-4 support.
When someone writes a subclass of the YAMLObject class, the constructors
will now be added to all 3 (non-safe) loaders.
Furthermore, we support the class variable `yaml_loader` being a list,
offering more control of which loaders are affected.
To support safe_load in your custom class you could add this:
yaml_loader = yaml.SafeLoader
yaml_loader = yaml.YAMLObject.yaml_loader
yaml_loader.append(yaml.SafeLoader)
* Fix logic for quoting special characters
* Remove has_ucs4 from condition
on systems with `sys.maxunicode <= 0xffff` the comparison
(u'\U00010000' <= ch < u'\U0010ffff') can't be true anyway I think
The `load` and `load_all` methods will issue a warning when they are
called without the 'Loader=' parameter. The warning will point to a URL
that is always up to date with the latest information on the usage of
`load`.
There are several ways to stop the warning:
* Use `full_load(input)` - sugar for `yaml.load(input, FullLoader)`
* FullLoader is the new safe but complete loader class
* Use `safe_load(input)` - sugar for `yaml.load(input, SafeLoader)`
* Make sure your input YAML consists of the 'safe' subset
* Use `unsafe_load(input)` - sugar for `yaml.load(input, UnsafeLoader)`
* Make sure your input YAML consists of the 'safe' subset
* Use `yaml.load(input, Loader=yaml.<loader>)`
* Or shorter `yaml.load(input, yaml.<loader>)`
* Where '<loader>' can be:
* FullLoader - safe, complete Python YAML loading
* SafeLoader - safe, partial Python YAML loading
* UnsafeLoader - more explicit name for the old, unsafe 'Loader' class
* yaml.warnings({'YAMLLoadWarning': False})
* Use this when you use third party modules that use `yaml.load(input)`
* Only do this if input is trusted
The above `load()` expressions all have `load_all()` counterparts.
You can get the original unsafe behavior with:
* `yaml.unsafe_load(input)`
* `yaml.load(input, Loader=yaml.UnsafeLoader)`
In a future release, `yaml.load(input)` will raise an exception.
The new loader called FullLoader is almost entirely complete as
Loader/UnsafeLoader but it does it avoids all known code execution
paths. It is the preferred YAML loader, and the current default for
`yaml.load(input)` when you get the warning.
Here are some of the exploits that can be triggered with UnsafeLoader
but not with FullLoader:
```
python -c 'import os, yaml; yaml.full_load("!!python/object/new:os.system [echo EXPLOIT!]")'`
python -c 'import yaml; print yaml.full_load("!!python/object/new:abs [-5]")'
python -c 'import yaml; yaml.full_load("!!python/object/new:eval [exit(5)]")' ; echo $?
python -c 'import yaml; yaml.full_load("!!python/object/new:exit [5]")' ; echo $?
This is the first release under new maintainership. A bunch of things
involving resource URLs and copyright details needed updating; in
addition to the normal version and changelog updates.
From the Psyco website:
> 12 March 2012
>
> Psyco is unmaintained and dead. Please look at PyPy for the
> state-of-the-art in JIT compilers for Python.
http://psyco.sourceforge.net/
