This patch was taken from
https://github.com/yaml/pyyaml/issues/369#issuecomment-571596545,
authored by Pekka Klärck <peke@iki.fi>.
In short, Jython doesn't support lone surrogates, so importing yaml (and
in particular, loading `reader.py`) caused a UnicodeDecodeError. This
patch works around this through a clever use of `eval` to defer
evaluation of the string containing the lone surrogates, only doing it
on non-Jython platforms.
This is only done in `lib/yaml/reader.py` and not `lib3/yaml/reader.py`
because Jython does not support Python 3.
With this patch, Jython's behavior with respect to Unicode code points
over 0xFFFF becomes as it was before
0716ae21a1. It still does not pass all the
unit tests on Jython (passes 1275, fails 3, errors on 1); all the
failing tests are related to unicode. Still, this is better than simply
crashing upon `import yaml`.
With this patch, all tests continue to pass on Python 2 / Python 3.
Repeated calls to `resolve` can experience performance degredation, if
`add_implicit_resolver` has been called with `first=None` (to add an
implicit resolver with an unspecified first character).
For example, every time `foo` is encountered, the "wildcard implicit
resolvers" (with `first=None`) will be appended to the list of implicit
resolvers for strings starting with `f`, which will normally be the
resolver for booleans. The list `yaml_implicit_resolvers['f']` will keep
getting longer. The same behavior applies for any first-letter matches
with existing implicit resolvers.
This change avoids unintentionally mutating the lists in the class-level
dict `yaml_implicit_resolvers` by looping through a temporary copy.
Fixes: #439
* Move most CI to GitHub Actions
* Build sdist
* Build manylinux1 wheels with libyaml ext (also tested with 2010 and 2014)
* Build MacOS x86_64 wheels with libyaml ext
* Windows wheel builds remain on AppVeyor until we drop 2.7 support in 6.0
* Smoke tests of all post-build artifacts
* Add PEP517/518 build declaration (pyproject.toml with setuptools backend)
* Fully move build to setuptools
* Drop Python 3.5 support
* Declare Python 3.9 support
* Update PyPI metadata now that setuptools lets it flow through
Co-authored-by: Matt Davis <mrd@redhat.com>
* Prevents arbitrary code execution during python/object/new constructor
In FullLoader python/object/new constructor, implemented by
construct_python_object_apply, has support for setting the state of a
deserialized instance through the set_python_instance_state method.
After setting the state, some operations are performed on the instance
to complete its initialization, however it is possible for an attacker
to set the instance' state in such a way that arbitrary code is executed
by the FullLoader.
This patch tries to block such attacks in FullLoader by preventing
set_python_instance_state from setting arbitrary properties. It
implements a blacklist that includes `extend` method (called by
construct_python_object_apply) and all special methods (e.g. __set__,
__setitem__, etc.).
Users who need special attributes being set in the state of a
deserialized object can still do it through the UnsafeLoader, which
however should not be used on untrusted input. Additionally, they can
subclass FullLoader and redefine `get_state_keys_blacklist()` to
extend/replace the list of blacklisted keys, passing the subclassed
loader to yaml.load.
* Make sure python/object/new constructor does not set some properties
* Add test to show how to subclass FullLoader with new blacklist
* On load, now use aware datetimes if possible
On loading data, if timestamps have an ISO "+HH:MM" UTC offset then the resultant datetime is converted to UTC. This change adds that timezone information to the datetime objects.
Importantly, this addresses a Django warning (and potential error) that appears when using both YAML fixtures in a timezone-aware project. It was raised as a Django issue (https://code.djangoproject.com/ticket/18867), but subsequently closed because the Django devs felt that this is a PyYAML problem.
* Create timezone-aware datetime in timezone from data
* Create timezone-aware datetime in timezone from data for python2
* Define better timezone implementation for python2
* Handle timezone "Z" for python 3
* Handle timezone "Z" for python 2
* Fix code structure for Python 3
Call datetime.datetime constructor once at return.
* Fix code structure for Python 2
Call datetime.datetime constructor once at return.
* remove some unused imports
as suggested by lgtm
https://lgtm.com/projects/g/yaml/pyyaml/
* add back import * from nodes
* remove also sys import
* remove mkpath import
* increase size of index, line, and column fields
* use size_t instead of unsigned long long
* better test infrastructure for test for large file
* only run large file test when env var is set
* fix review comments regarding env vars
* fix missing import on python 3
* force all tests in CI
This patch fixes the handling of inputs with Unicode code points over
0xFFFF when running on a Python 2 that does not have UCS-4 support
(which certain distributions still ship, e.g. macOS).
When Python is compiled without UCS-4 support, it uses UCS-2. In this
situation, non-BMP Unicode characters, which have code points over
0xFFFF, are represented as surrogate pairs. For example, if we take
u'\U0001f3d4', it will be represented as the surrogate pair
u'\ud83c\udfd4'. This can be seen by running, for example:
[i for i in u'\U0001f3d4']
In PyYAML, the reader uses a function `check_printable` to validate
inputs, making sure that they only contain printable characters. Prior
to this patch, on UCS-2 builds, it incorrectly identified surrogate
pairs as non-printable.
It would be fairly natural to write a regular expression that captures
strings that contain only *printable* characters, as opposed to
*non-printable* characters (as identified by the old code, so not
excluding surrogate pairs):
PRINTABLE = re.compile(u'^[\x09\x0A\x0D\x20-\x7E\x85\xA0-\uD7FF\uE000-\uFFFD]*$')
Adding support for surrogate pairs to this would be straightforward,
adding the option of having a surrogate high followed by a surrogate low
(`[\uD800-\uDBFF][\uDC00-\uDFFF]`):
PRINTABLE = re.compile(u'^(?:[\x09\x0A\x0D\x20-\x7E\x85\xA0-\uD7FF\uE000-\uFFFD]|[\uD800-\uDBFF][\uDC00-\uDFFF])*$')
Then, this regex could be used as follows:
def check_printable(self, data):
if not self.PRINTABLE.match(data):
raise ReaderError(...)
However, matching printable strings, rather than searching for
non-printable characters as the code currently does, would have the
disadvantage of not identifying the culprit character (we wouldn't get
the position and the actual non-printable character from a lack of a
regex match).
Instead, we can modify the NON_PRINTABLE regex to allow legal surrogate
pairs. We do this by removing surrogate pairs from the existing
character set and adding the following options for illegal uses of
surrogate code points:
- Surrogate low that doesn't follow a surrogate high (either a surrogate
low at the start of a string, or a surrogate low that follows a
character that's not a surrogate high):
(?:^|[^\uD800-\uDBFF])[\uDC00-\uDFFF]
- Surrogate high that isn't followed by a surrogate low (either a
surrogate high at the end of a string, or a surrogate high that is
followed by a character that's not a surrogate low):
[\uD800-\uDBFF](?:[^\uDC00-\uDFFF]|$)
The behavior of this modified regex should match the one that is used
when Python is built with UCS-4 support.
It helps people to use `safe_load` if they discover the library.
It's more secure if `safe_load()` is used by default, and `load()` is used if it's necessary (and the developer knows what is does).
