This implements an API in `nsIOSKeyStore.idl` and `OSKeyStore.cpp` to encrypt and decrypt bytes with a key that is stored in the OS key store.
There are two OS adapters in this patch.
Libsecret is used on Linux if available.
The NSS key store is used as fallback if no OS specific key store is implemented.
Differential Revision: https://phabricator.services.mozilla.com/D1858
--HG--
extra : rebase_source : 99d7d646968a46a13ffa61885bb246f6d3e443e4
Now that we've generated an order file of the first N functions invoked
during startup, let's tell the linker about said functions so it can
cluster them appropriately.
When I originally implemented bug 1458161, this is how it was done, but
it was suggested to use a configure-time check. This turned out to not
be great, because the rust compiler changes regularly, and we don't run
the configure tests when the version changes. When people upgraded their
rust compiler to 1.27, the code subsequently failed to build because the
features were still set for the previous version they had installed.
--HG--
extra : rebase_source : 1b5f7a02ad8495d68cd29289f7beea59b8912183
Overall, this makes the whole setup less fragile, and make it work with
LTO in more situations.
--HG--
extra : rebase_source : de968c61dc4ef337fdc28745c202334ac41763cd
We perform, on the binaries we build, a series of check, that are
implemented as half-baked make commands, invoked after linking them.
- check libstdc++ symbol versions to ensure binary compatibility with
a baseline.
- check glibc symbol versions to ensure binary compatibility with a
baseline.
- check that target binaries don't contain text relocations.
- check that libmozglue is linked before libc on android.
- on libxul, check that NSModules are laid out correctly.
- on libxul, check that there is more than one PT_LOAD segment.
Those checks happen to work where they matter, but their setup is
unreliable. For example, the checks for symbol versions are supposed to
work for libclang-plugin on cross osx builds, but in fact, don't,
because the readelf path doesn't exist, and the command doesn't fail in
that case.
So move them all to a standalone script, performing the checks more
thoroughly (especially the NSModules one, where we now also check that
they are all adjacent), and more verbosely.
--HG--
extra : rebase_source : 7072e622e95f363d4a6c3a8e272d3445d998b592
Bug 1458161 added a rust OOM handler based on an unstable API that was
removed in 1.27, replaced with something that didn't allow to get the
failed allocation size.
Latest 1.28 nightly (2018-06-13) has
https://github.com/rust-lang/rust/pull/50880,
https://github.com/rust-lang/rust/pull/51264 and
https://github.com/rust-lang/rust/pull/51241 merged, which allow to
hook the OOM handler and get the failed allocation size again.
Because this is still an unstable API, we explicitly depend on strict
versions of rustc. We also explicitly error out if automation builds
end up using a rustc version that doesn't allow us to get the allocation
size for rust OOM, because we don't want that to happen without knowing.
--HG--
extra : rebase_source : 6c097151046d088cf51f4755dd69bde97bb8bd8b
The crash reporter symbol files are the easiest cross-platform way to
find static initializers. While some types of static initializers (e.g.
__attribute__(constructor) functions) don't appear there in a notable
way, the static initializers we do care the most about for tracking do
(static initializers from C++ globals). As a matter of fact, there is
only a difference of 2 compared to the currently reported count of 125
on a linux64 build, so this is a good enough approximation. And allows
us to easily track the count on Android, OSX and Windows builds, which
we currently don't do.
The tricky part is that the symbol files are in
dist/crashreporter-symbols/$lib/$fileid/$lib.sym, and $fileid is hard to
figure out. There is a `fileid` tool in testing/tools, but it is a
target binary, meaning it's not available on cross builds (OSX,
Android).
So the simplest is just to gather the data while creating the symbol
files, which unfortunately requires to go through some hoops to make it
happen for just the files we care about.
--HG--
extra : rebase_source : 458fed1ffd6f9294eefef61f10ff7a284af0d986
Our bundled Hunspell now significantly differs from upstream Hunspell. Most
importantly, it supports loading dictionaries from jar: URIs, which is now a
requirement for loading bundled and extension dictionaries. This means that
system Hunspell libraries are no longer compatible with our spell checker
code. We should remove the option to use them so that users don't fall into
the trap of trying to use them.
MozReview-Commit-ID: 2ihJe6YOnGf
--HG--
extra : rebase_source : ceb091b9475a2b101156405a02a60015fc36da17
Update mp4parse-rust to 0c8e1d91464aaa63b82ebf076b63cda1df4230d1, which adds
uuid parsing support and exports the mp4parse_fallible feature from
mp4parse_capi.
Update gkrust to pass MOZ_MEMORY as a feature, and use that to conditionally
enable mp4parse_fallible/FallibleVec.
MozReview-Commit-ID: 2HDYbL2CGgJ
--HG--
extra : rebase_source : 6e8cf15241b0282406322cce29220a677edd1585
Update mp4parse-rust to 0c8e1d91464aaa63b82ebf076b63cda1df4230d1, which adds
uuid parsing support and exports the mp4parse_fallible feature from
mp4parse_capi.
Update gkrust to pass MOZ_MEMORY as a feature, and use that to conditionally
enable mp4parse_fallible/FallibleVec.
MozReview-Commit-ID: 2HDYbL2CGgJ
--HG--
extra : rebase_source : 299d9f8347d2f0ef0d66b9ea52a4ee7a31af0cd2
OOM rust crashes are currently not identified as such in crash reports
because rust libstd handles the OOMs and panics itself.
There are unstable ways to hook into this, which unfortunately are under
active changes in rust 1.27, but we're currently on 1.24 and 1.27 is not
released yet. The APIs didn't change between 1.24 and 1.26, so it's
fine-ish to use them as long as we limit their use to those versions.
As long as the Firefox versions we ship (as opposed to downstream) use
the "right" version of rust, we're good to go.
The APIs are in their phase of stabilization, so there shouldn't be too
many variants of the code to support.
--HG--
extra : rebase_source : 08a85aa102b24380b1f6764effffcc909ef3191b
This patch rewrites the rust-url-capi crate as the mozurl crate, which
provides a threadsafe MozURL object which is compatible with the
previous MozURL class.
Creating a MozURL this way performs a single allocation, which contains
only a rust-url Url object and an atomic refcnt, however it is fully
compatible with the C++ RefPtr type.
This patch also exposes methods for accessing dependent substrings of
the serialized spec, meaning that string copies can be avoided in many
situations when inspecting attributes of the MozURL.
--HG--
rename : netwerk/base/rust-url-capi/.gitignore => netwerk/base/mozurl/.gitignore
Remove the syn dependency from gkrust-shared, which was added in bug 1373878
as a workaround for this Cargo bug:
https://github.com/rust-lang/cargo/issues/3923
MozReview-Commit-ID: L34J0davEYd
--HG--
extra : rebase_source : 79e5684cba143f6215e7dcf9367f82227bca48c5
Starting with Rust 1.24, the default codegen-units limit is 16,
with jobserver control to avoid overprovisioning. Remove our
previous fixed limit of 4 threads for debug builds.
For release, retain codegen-units=1 to make sure we get the
most complete optimization results.
Thanks to Simon Sapin for the suggestion.
MozReview-Commit-ID: FmYF4DcmBvt
--HG--
extra : rebase_source : 307ad8fad2874636adb3ce95a5cd47339e83f40c
Import v0.10.0 of the mp4parse and mp4parse_capi crates
and update dependencies.
Reduces library size by removing debug tracing in release builds.
Also adds recognition of the ALAC codec, although we don't plan
to support it.
MozReview-Commit-ID: F1bnotCmbDf
--HG--
extra : rebase_source : 55bc014378d7f65fca8af82a9222edd36870b351
The prefs parser has two significant problems.
- It doesn't separate tokenizing from parsing.
- It is implemented as a loop around a big switch on a "current state"
variable.
As a result, it is hard to understand and modify, slower than it could be, and
in obscure cases (involving comments and whitespace) it fails to parse what
should be valid input.
This patch replaces it with a recursive descent parser (albeit one without any
recursion!) that has separate tokenization. The new parser is easier to
understand and modify, more correct, and has better error messages. It doesn't
do error recovery, but that would be much easier to add than in the old parser.
The new parser also runs about 1.9x faster than the existing parser. (As
measured by parsing greprefs.js's contents from memory 1000 times in
succession, omitting the prefs hash table construction. If the table
construction is included, it's about 1.6x faster.)
The new parser is slightly stricter than the old parser in a few ways.
- Disconcertingly, the old parser allowed arbitrary junk between prefs
(including at the start and end of the prefs file) so long as that junk
didn't include any of the following chars: '/', '#', 'u', 's', 'p'. I.e.
lines like these:
!foo@bar&pref("prefname", true);
ticky_pref("prefname", true); // missing 's' at start
User_pref("prefname", true); // should be 'u' at start
would all be treated the same as this:
pref("prefname", true);
The new parser disallows such junk because it isn't necessary and seems like
an unintentional botch by the old parser.
- The old parser allowed character 0x1a (SUB) between tokens and treated it
like '\n'.
The new parser does not allow this character. SUB was used to indicate
end-of-file (*not* end-of-line) in some old operating systems such as MS-DOS,
but this doesn't seem necessary today.
- The old parser tolerated (with a warning) invalid escape sequences within
string literals -- such as "\q" (not a valid escape) and "\x1" and "\u12"
(both of which have insufficient hex digits) -- accepting them literally.
The new parser does not tolerate invalid escape sequences because it doesn't
seem necessary and would complicate things.
- The old parser tolerated character 0x00 (NUL) within string literals; this is
dangerous because C++ code that manipulates string values with embedded NULs
will almost certainly consider those chars as end-of-string markers.
The new parser treats NUL chars as end-of-file, to avoid this danger and
because it facilitates a significant optimization (described within the
code).
- The old parser allowed integer literals to overflow, silently wrapping them.
The new parser treats integer overflow as a parse error. This seems better,
and it caught existing overflows of places.database.lastMaintenance, in
testing/profiles/prefs_general.js (bug 1424030) and
testing/talos/talos/config.py (bug 1434813).
The first of these changes meant that a couple of existing prefs with ";;" at
the end had to be changed (done in the preceding patch).
The minor increase in strictness shouldn't be a problem for default pref files
such as greprefs.js within the application (which we can modify), nor for
app-written prefs files such as prefs.js. It could affect user-written prefs
files such as user.js; the experience above suggests that integer overflow and
";;" are the most likely problems in practice. In my opinion, the risk here is
acceptable.
The new parser also does a better job of tracking line numbers because it (a)
treats "\r\n" sequences as a single end-of-line marker, and (a) pays attention
to end-of-line sequences within string literals.
Finally, the patch adds thorough tests of both valid and invalid syntax.
MozReview-Commit-ID: JD3beOQl4AJ
The prefs parser has two significant problems.
- It doesn't separate tokenizing from parsing.
- It is implemented as a loop around a big switch on a "current state"
variable.
As a result, it is hard to understand and modify, slower than it could be, and
in obscure cases (involving comments and whitespace) it fails to parse what
should be valid input.
This patch replaces it with a recursive descent parser (albeit one without any
recursion!) that has separate tokenization. The new parser is easier to
understand and modify, more correct, and has better error messages. It doesn't
do error recovery, but that would be much easier to add than in the old parser.
The new parser also runs about 1.9x faster than the existing parser. (As
measured by parsing greprefs.js's contents from memory 1000 times in
succession, omitting the prefs hash table construction. If the table
construction is included, it's about 1.6x faster.)
The new parser is slightly stricter than the old parser in a few ways.
- Disconcertingly, the old parser allowed arbitrary junk between prefs
(including at the start and end of the prefs file) so long as that junk
didn't include any of the following chars: '/', '#', 'u', 's', 'p'. I.e.
lines like these:
!foo@bar&pref("prefname", true);
ticky_pref("prefname", true); // missing 's' at start
User_pref("prefname", true); // should be 'u' at start
would all be treated the same as this:
pref("prefname", true);
The new parser disallows such junk because it isn't necessary and seems like
an unintentional botch by the old parser.
- The old parser allowed character 0x1a (SUB) between tokens and treated it
like '\n'.
The new parser does not allow this character. SUB was used to indicate
end-of-file (*not* end-of-line) in some old operating systems such as MS-DOS,
but this doesn't seem necessary today.
- The old parser tolerated (with a warning) invalid escape sequences within
string literals -- such as "\q" (not a valid escape) and "\x1" and "\u12"
(both of which have insufficient hex digits) -- accepting them literally.
The new parser does not tolerate invalid escape sequences because it doesn't
seem necessary and would complicate things.
- The old parser tolerated character 0x00 (NUL) within string literals; this is
dangerous because C++ code that manipulates string values with embedded NULs
will almost certainly consider those chars as end-of-string markers.
The new parser treats NUL chars as end-of-file, to avoid this danger and
because it facilitates a significant optimization (described within the
code).
- The old parser allowed integer literals to overflow, silently wrapping them.
The new parser treats integer overflow as a parse error. This seems better,
and it caught an existing overflow in testing/profiles/prefs_general.js, for
places.database.lastMaintenance (see bug 1424030).
The first of these changes meant that a couple of existing prefs with ";;" at
the end had to be changed (done in the preceding patch).
The minor increase in strictness shouldn't be a problem for default pref files
such as greprefs.js within the application (which we can modify), nor for
app-written prefs files such as prefs.js. It could affect user-written prefs
files such as user.js; the experience above suggests that ";;" is the most
likely problem in practice. In my opinion, the risk here is acceptable.
The new parser also does a better job of tracking line numbers because it (a)
treats "\r\n" sequences as a single end-of-line marker, and (a) pays attention
to end-of-line sequences within string literals.
Finally, the patch adds thorough tests of both valid and invalid syntax.
MozReview-Commit-ID: 8EYWH7KxGG
* * *
[mq]: win-fix
MozReview-Commit-ID: 91Bxjfghqfw
--HG--
extra : rebase_source : a8773413e5d68c33e4329df6819b6e1f82c22b85
We don't add build-time dependency on Wayland libraries; that allows Wayland enabled Firefox
builds to run on systems without Wayland support.
We also can't dlsym() Wayland symbols directly from libwayland-client.so library as we miss
global data entries referenced by code at wayland-client.h then.
As a partial solution (for glibc systems only) we create dummy libwayland-client.so library
implementation and place it *after* Gtk+ 3.0 libraries at linking time.
It fixes build-time dependencies (we link against our libwayland-client.so library).
Run-time dependency is resolved by ld.so - when Gtk+ 3.0 libraries are linked with
system libwayland-client.so library, wayland symbols are pulled from there instead
from our dummy libwayland-client.so library.
When Gtk+ 3.0 is not linked with system libwayland-client.so it means we're running
on system without Wayland support. Our dummy libwayland-client.so implementation
is used (symbols are pulled run-time from there) and Firefox Wayland support is
disabled then.
MozReview-Commit-ID: IyaePwp4MxV
--HG--
extra : rebase_source : 852955d001657176e0bf69c099580be862d0b448
