I think they're remnants from the past that we don't really need anymore.
And they're making things more complicated for some pending work of mine.
Differential Revision: https://phabricator.services.mozilla.com/D89687
resolver_64.cc should have been on the list all along, because the InternalThunk
constructor runs before ASan init. It was probably just accident (maybe inlining?)
that we got away with not including it in the past.
Differential Revision: https://phabricator.services.mozilla.com/D89669
In addition to the existing build telemetry, also gather the stats and
report with Glean. This new telemetry is reported in tandem with the existing
telemetry to allow testing and confidence before a full roll-out.
Additionally, Glean isn't compatible with Python 2, so the new telemetry only runs
on Python 3 mach commands.
Differential Revision: https://phabricator.services.mozilla.com/D83572
The files are copied verbatim from upstream autoconf 2.13 (but only the
files we need) and old.configure is adapted to use the vendored version.
Differential Revision: https://phabricator.services.mozilla.com/D89554
This was originally meant to allow `virtualenv`s to use packages from a parent Python environment without having to re-install them. This turned out to not pan out as we would have liked, so we're going another way to solve the same problem. Bug 1660351 walked back a bunch of this logic; this patch deletes the rest of it.
Differential Revision: https://phabricator.services.mozilla.com/D89492
This commit does the following.
- Renames `slashslash` as `dumbComments`. As a result, it now comes before
`emptyLines` in alphabetical ordering, which means that if you apply both
`dumbComments` and `emptyLines`, lines that contain only comments will be
fully removed.
(I contemplated changing the filter ordering to match the order specified,
rather than using alphabetical ordering, but that was more invasive and not
obviously better.)
- Changes `dumbComments` so it only applies if the comment is at the start of
the line (with optional leading whitespace). This is so it can be used with
prefs files, which contain lines like `pref("foo", "https://mozilla.org");`
where the `//` must not be treated as a comment.
Note that `slashslash` wasn't being used anywhere.
Depends on D88240
Differential Revision: https://phabricator.services.mozilla.com/D88242
It's not used, probably because it's pretty strange and hard to imagine using
safely. (Stripping leading and trailing space could be useful, but collapsing
sequences of spaces? Hmm.)
Differential Revision: https://phabricator.services.mozilla.com/D88240
When configuring without system NSPR the configuration variable
PKGCONF_REQUIRES_PRIVATE isn't ever set, leading to a .pc file that still
contains the @PKGCONF_REQUIRES_PRIVATE@ stub.
Ensure that we always define PKGCONF_REQUIRES_PRIVATE, by setting it to an
empty string in case no system-nsrp is enabled.
In this way, the pkg-config file stub will be always replaced.
Differential Revision: https://phabricator.services.mozilla.com/D88179
widl output for dlldata has #defines immediately followed by #includes,
so looking for empty lines when we observer a #define doesn't work. We
instead look for #defines.
Differential Revision: https://phabricator.services.mozilla.com/D88938
The work in bug 1620133 ended up moving the execution of some of the
midl commands to the top-level, which was alleviated by adding the -out
midl command line option to make midl place its output in the expected
directory. Unfortunately, that option is not handled by widl, which is
the alternative command used in mingw builds.
So instead of using -out, we set the cwd for the midl command, and
readjust the command line arguments to be relative to that.
Differential Revision: https://phabricator.services.mozilla.com/D88937
For not-well-understood reasons, ld's `--gc-sections` discards a large number of the PGO bookkeeping structures that enable us to keep track of function counters, and the effect gets worse in object files generated by clang-10.
As much as I'd like to understand this better, the investigations take way too much time. As a path of least resistance, we can disable `--gc-sections` for the instrumentation phase of PGO builds. It won't harm anything since users never see those builds, and it will improve the performance of the optimized phase greatly.
Differential Revision: https://phabricator.services.mozilla.com/D78112
We don't anticipate end users will actually care to do this, but it's useful especially for unit tests. For example, after bug 1659539, Python `configure` tests will run in a new, non-`init_py3` `virtualenv`, and we'll want to target that `virtualenv` for `configure` rather than having it create a new `virtualenv` for no reason.
Differential Revision: https://phabricator.services.mozilla.com/D88661
2020-08-21 Kevin Jacobs <kjacobs@mozilla.com>
* automation/abi-check/previous-nss-release, lib/nss/nss.h,
lib/softoken/softkver.h, lib/util/nssutil.h:
Set version numbers to 3.57 Beta
[783f49ae6126]
2020-08-24 Kevin Jacobs <kjacobs@mozilla.com>
* gtests/ssl_gtest/ssl_auth_unittest.cc, lib/ssl/dtls13con.c,
lib/ssl/dtlscon.c, lib/ssl/ssl3con.c, lib/ssl/sslimpl.h,
lib/ssl/sslnonce.c:
Bug 1653641 - Cleanup inaccurate DTLS comments, code review fixes.
r=mt
[0e1b5c711cb9]
2020-08-24 Robert Relyea <rrelyea@redhat.com>
* lib/freebl/fipsfreebl.c, lib/softoken/fipstest.c,
lib/softoken/kbkdf.c, lib/softoken/lowpbe.c, lib/softoken/lowpbe.h,
lib/softoken/pkcs11c.c, lib/softoken/pkcs11i.h,
lib/softoken/sftkhmac.c, lib/softoken/sftkike.c:
Bug 1660304 New FIPS IG requires self-tests for approved kdfs.
r=ueno comments=kjacobs
FIPS guidance now requires self-tests for our kdfs. It also requires
self-tests for cmac which we didn't have in the cmac patch.
Currently only one test per kdf is necessary. Specifially for
SP-800-108, only one of the three flavors are needed (counter,
feedback, or pipeline). This patch includes more complete testing
but it has been turned off the currently extraneous tests under the
assumption that NIST guidance may require them in the future. HKDF
is currently not included in FIPS, but is on track to be included,
so hkdf have been included in this patch.
Because the test vectors are const strings, the patch pushes some
const definitions that were missing in existing private interfaces.
There are three flavors of self-tests: Function implemented in
freebl are added to the freebl/fipsfreebl.c Functions implemented in
pkcs11c.c have selftests completely implemented in
softoken/fipstest.c Functions implemented in their own .c file have
their selftest function implemented in that .c file and called by
fipstests.c These are consistant with the previous choices for
selftests.
Some private interfaces that took in keys from pkcs #11 structures
or outputted keys to pkcs #11 structures were modified to optionally
take keys in by bytes and output keys as bytes so the self-tests can
work in just bytes.
[5dca54fe61c2]
2020-08-25 Daiki Ueno <dueno@redhat.com>
* lib/softoken/manifest.mn:
Bug 1659252, disable building libnssdbm3.so if NSS_DISABLE_DBM=1,
r=rrelyea
Reviewers: rrelyea
Reviewed By: rrelyea
Bug #: 1659252
[4d55d36ca6ef]
2020-08-24 Kevin Jacobs <kjacobs@mozilla.com>
* lib/pk11wrap/pk11cxt.c, lib/softoken/pkcs11c.c, lib/softoken/sdb.c,
lib/softoken/sftkpwd.c:
Bug 1651834 - Fix various static analyzer warnings. r=rrelyea
[ab04fd73fd6d]
2020-08-28 Mike Hommey <mh@glandium.org>
* lib/freebl/blapii.h:
Bug 1661810 - Define pre_align/post_align based on the compiler.
r=jcj
Things worked fine before we upgraded to clang 11 presumably because
the stack was always 16-bytes aligned in the first place, or
something akin to that, and the lack of pre_align/post_align doing
anything didn't matter. The runtime misalignment of the stack may
well be a clang > 9 bug, but keeping pre_align/post_align tied to
the x86/x64 is a footgun anyways.
[c100e11991f6] [tip]
Differential Revision: https://phabricator.services.mozilla.com/D88876
config.guess infers information about the compiler using environment
variables, such as CC. However, we use such environment variables to
configure the tooling for the target.
Differential Revision: https://phabricator.services.mozilla.com/D88085
Previously, we would optimistically attempt to use a Rust toolchain that
matches the current C toolchain, and would throw an error if an
attempted compile with that Rust toolchain failed.
Instead, if we fail to detect a usable Rust toolchain, we now helpfully
inform users of their two options: change C toolchain, or install
matching Rust toolchain.
Differential Revision: https://phabricator.services.mozilla.com/D88084
config.guess infers information about the compiler using environment
variables, such as CC. However, we use such environment variables to
configure the tooling for the target.
Differential Revision: https://phabricator.services.mozilla.com/D88085
Previously, we would optimistically attempt to use a Rust toolchain that
matches the current C toolchain, and would throw an error if an
attempted compile with that Rust toolchain failed.
Instead, if we fail to detect a usable Rust toolchain, we now helpfully
inform users of their two options: change C toolchain, or install
matching Rust toolchain.
Differential Revision: https://phabricator.services.mozilla.com/D88084
For not-well-understood reasons, ld's `--gc-sections` discards a large number of the PGO bookkeeping structures that enable us to keep track of function counters, and the effect gets worse in object files generated by clang-10.
As much as I'd like to understand this better, the investigations take way too much time. As a path of least resistance, we can disable `--gc-sections` for the instrumentation phase of PGO builds. It won't harm anything since users never see those builds, and it will improve the performance of the optimized phase greatly.
Differential Revision: https://phabricator.services.mozilla.com/D78112
For not-well-understood reasons, ld's `--gc-sections` discards a large number of the PGO bookkeeping structures that enable us to keep track of function counters, and the effect gets worse in object files generated by clang-10.
As much as I'd like to understand this better, the investigations take way too much time. As a path of least resistance, we can disable `--gc-sections` for the instrumentation phase of PGO builds. It won't harm anything since users never see those builds, and it will improve the performance of the optimized phase greatly.
Differential Revision: https://phabricator.services.mozilla.com/D78112
This adds toolchain definitions for clang 11.0.0 rc2, so that developers can get a sneak peek, but nothing in automation uses these tasks yet. We'll make the switch in a later patch.
NB: most of `clang.yml` is rote copy-paste, except for `macosx64-clang-11` which makes a deliberate departure, described in a comment.
Differential Revision: https://phabricator.services.mozilla.com/D88189
This avoids a set of intermittent issues related to `zstd` decompression failures, which in the absence of these changes break the entire build.
This also requires [updating an environment variable](https://github.com/mozilla/sccache/pull/822), which we do in `client.mk` as well as documentation.
Differential Revision: https://phabricator.services.mozilla.com/D88184
Also define a scheme for storing the index of Glean definitions files in a file
separate from the build system for consumption by
* mach build
* mach doc
* (future) mozilla/probe-scraper
Differential Revision: https://phabricator.services.mozilla.com/D87600
We have a minimum rust version required for compilations. For both stable and beta rust compilers, we can trust that they will have all the stabilized features we're expecting.
However, for nightlies, they may "match" our minimum version, but may have been released in the version window before a certain feature we need has been stabilized.
So, when validating rustc version in configure, ensure that the nightly is at least one version newer than our expected version.
Differential Revision: https://phabricator.services.mozilla.com/D86889
`mach create-mach-environment` is what installs `glean_sdk` to the `mach` `virtualenv`. `create-mach-environment` runs on the system Python and we can't assume the system Python has `glean_sdk` installed.
Differential Revision: https://phabricator.services.mozilla.com/D87507
There are zero uses of this `mach` command over the past 90 days according to our telemetry. There are no external references to `mach python-safety` in-tree, and indeed if you track the history of the originating bug 1468394, it appears that once the `mach` command was created, none of the follow-up work that was discussed (i.e. running this in CI and triaging failures to appropriate owners) was done over the following 2 years.
If this ever does appear to be useful in the future, we can just resurrect this code from source control.
Differential Revision: https://phabricator.services.mozilla.com/D87351
In two different places we've been encountering issues regarding 1) how we configure the system Python environment and 2) how the system Python environment relates to the `virtualenv`s that we use for building, testing, and other dev tasks. Specifically:
1. With the push to use `glean` for telemetry in `mach`, we are requiring (or rather, strongly encouraging) the `glean_sdk` Python package to be installed with bug 1651424. `mach bootstrap` upgrades the library using your system Python 3 in bug 1654607. We can't vendor it due to the package containing native code. Since we generally vendor all code required for `mach` to function, requiring that the system Python be configured with a certain version of `glean` is an unfortunate change.
2. The build uses the vendored `glean_parser` for a number of build tasks. Since the vendored `glean_parser` conflicts with the globally-installed `glean_sdk` package, we had to add special ad-hoc handling to allow us to circumvent this conflict in bug 1655781.
3. We begin to rely more and more on the `zstandard` package during build tasks, this package again being one that we can't vendor due to containing native code. Bug 1654994 contained more ad-hoc code which subprocesses out from the build system's `virtualenv` to the SYSTEM `python3` binary, assuming that the system `python3` has `zstandard` installed.
As we rely more on `glean_sdk`, `zstandard`, and other packages that are not vendorable, we need to settle on a standard model for how `mach`, the build process, and other `mach` commands that may make their own `virtualenv`s work in the presence of unvendorable packages.
With that in mind, this patch does all the following:
1. Separate out the `mach` `virtualenv_packages` from the in-build `virtualenv_packages`. Refactor the common stuff into `common_virtualenv_packages.txt`. Add functionality to the `virtualenv_packages` manifest parsing to allow the build `virtualenv` to "inherit" from the parent by pointing to the parent's `site-packages`. The `in-virtualenv` feature from bug 1655781 is no longer necessary, so delete it.
2. Add code to `bootstrap`, as well as a new `mach` command `create-mach-environment` to create `virtualenv`s in `~/.mozbuild`.
3. Add code to `mach` to dispatch either to the in-`~/.mozbuild` `virtualenv`s (or to the system Python 3 for commands which cannot run in the `virtualenv`s, namely `bootstrap` and `create-mach-environment`).
4. Remove the "add global argument" feature from `mach`. It isn't used and conflicts with (3).
5. Remove the `--print-command` feature from `mach` which is obsoleted by these changes.
This has the effect of allowing us to install packages that cannot be vendored into a "common" place (namely the global `~/.mozbuild` `virtualenv`s) and use those from the build without requiring us to hit the network. Miscellaneous implementation notes:
1. We allow users to force running `mach` with the system Python if they like. For now it doesn't make any sense to require 100% of people to create these `virtualenv`s when they're allowed to continue on with the old behavior if they like. We also skip this in CI.
2. We needed to duplicate the global-argument logic into the `mach` script to allow for the dispatch behavior. This is something we avoided with the Python 2 -> Python 3 migration with the `--print-command` feature, justifying its use by saying it was only temporarily required until all `mach` commands were running with Python 3. With this change, we'll need to be able to determine the `mach` command from the shell script for the forseeable future, and committing to this forever with the cost that `--print-command` incurs (namely `mach` startup time, an additional .4s on my machine) didn't seem worth it to me. It's not a ton of duplicated code.
Differential Revision: https://phabricator.services.mozilla.com/D85916
This is a partial revert of "Bug 1650996 - P3. Have RemoteDecoderManagerChild use a TaskQueue over a media threadpool."
The RemoteDecoderManagerChild dispatch tasks synchronously, right now it is doing so on the media controller's thread pool ; however in the following patch it will change the creation to the decoder's thread pool.
If we attempt to instantiate too many decoders at once, we run out of available threads and dead-lock in the sync dispatch.
This issue has bitten us in various places already and the solution was always assuming that the decoder will always be created on the controller's thread and used on the decoder's thread.
This assumption won't hold any longer and was difficult to keep anyway.
So we have the RemoteDecoderManagerChild uses a dedicated thread so that we can guarantee there will always be an available thread to create the decoder.
Depends on D86543
Differential Revision: https://phabricator.services.mozilla.com/D86895
This makes us use one less tool from MSVC, and removes one more use of wine
in cross builds.
We replace the call to either rc/llvm-rc or windres with a wrapper script.
While the script is not strictly needed for the latter, we use a wrapper
in that case anyway because it's one step towards fixing bug 1498414.
For llvm-rc, however, we need a wrapper because llvm-rc doesn't preprocess
on its own, so the wrapper does that too.
The wrapper script also allows to deal with the default flags passed to
llvm-rc or windres, rather than inherit them from old-configure.
We also need to explicitly pass the codepage to llvm-rc, which was not
necessary with rc (presumably, llvm-rc has a different default).
While here, remove the unused WINDRES subst from js/src/old-configure.in.
Also, while here, we remove --use-temp-file, because as described in the
linked bug and in the windres manual page, it was used to work around bugs
on Windows 98 and earlier.
Differential Revision: https://phabricator.services.mozilla.com/D86312
Today we don't require that `mach` `CommandProvider`s subclass from any particular parent class and we're very lax about the requirements they must meet. While that's convenient in certain circumstances, it has some unfortunate implications for feature development.
Today the only requirements that we have for `CommandProvider`s are that they have an `__init__()` method that takes either 1 or 2 arguments, the second of which must be called `context` and is populated with the `mach` `CommandContext`. Again, while this flexibility is occasionally convenient, it is limiting. As we add features to `mach`, having a better idea what the shape of our `CommandProvider`s are and how we can instantiate them and use them is increasingly important, and this gives us additional control when having `mach` configure `CommandProvider`s based on data that is only available at the `mach` level. In particular, we plan to leverage this in bugs 985141 and 1654074.
Here we add validation to the `CommandProvider` decorator to ensure all classes inherit from `MachCommandBase`, update all `CommandProvider`s in-tree to inherit from `MachCommandBase`, and update source and test code accordingly.
Follow-up work: we now require (de facto) that the `context` be populated with a `topdir` attribute by the `populate_context_handler` function, since instantiating the `MachCommandBase` requires a `topdir` be provided. This is fine for now in the interest of keeping this patch reasonably sized, but some additional refactoring could make this cleaner.
Differential Revision: https://phabricator.services.mozilla.com/D86255
I noticed that the `objdir:build` entry in `build/virtualenv_packages.txt` entry was apparently unused. This originates from bug 841713, seven years ago, when the `objdir` handling was introduced. Today, this doesn't appear to be serving a purpose. There is no Python library in my `$objdir/build` directory; nor can I see anything in `build/moz.build` or any related files suggesting one could ever appear. I can only assume this feature has outlived its usefulness, so delete it and the relevant in-tree support.
This necessitates slightly changing the signature and implementation of the `activate_pipenv()` method; also update all callers.
Differential Revision: https://phabricator.services.mozilla.com/D85635
I noticed that the `objdir:build` entry in `build/virtualenv_packages.txt` entry was apparently unused. This originates from bug 841713, seven years ago, when the `objdir` handling was introduced. Today, this doesn't appear to be serving a purpose. There is no Python library in my `$objdir/build` directory; nor can I see anything in `build/moz.build` or any related files suggesting one could ever appear. I can only assume this feature has outlived its usefulness, so delete it and the relevant in-tree support.
This necessitates slightly changing the signature and implementation of the `activate_pipenv()` method; also update all callers.
Differential Revision: https://phabricator.services.mozilla.com/D85635
We are currently relying on two different ways to insert side-by-side
manifests in binaries on Windows: through resource files, or through
the use of MT. The latter is not supported on mingw builds, which is
not great.
Link.exe has options to add a manifest at link time without relying on
either method above, but that's not supported on mingw either.
So the best we can do is to move everything to using resource files.
This also avoids using MT, which, on cross builds, requires using wine.
Ideally, the manifests would be declared in moz.build, but that
complicates things for cases like TestDllInterceptor, where there are
multiple binaries in the same directory, but only one of them needs the
manifest. This keeps the status quo of getting the manifest
automatically from the source directory.
Differential Revision: https://phabricator.services.mozilla.com/D85382
We are currently relying on two different ways to insert side-by-side
manifests in binaries on Windows: through resource files, or through
the use of MT. The latter is not supported on mingw builds, which is
not great.
Link.exe has options to add a manifest at link time without relying on
either method above, but that's not supported on mingw either.
So the best we can do is to move everything to using resource files.
This also avoids using MT, which, on cross builds, requires using wine.
Ideally, the manifests would be declared in moz.build, but that
complicates things for cases like TestDllInterceptor, where there are
multiple binaries in the same directory, but only one of them needs the
manifest. This keeps the status quo of getting the manifest
automatically from the source directory.
Differential Revision: https://phabricator.services.mozilla.com/D85382
clang-11 has a new warning that fires when you pass a 256-bit vector type as a parameter or return value, and your compilation doesn't enable AVX. The aim is to warn you that the ABI is different depending on whether you enable AVX, which could become a problem if AVX and non-AVX files communicate through such parameters.
While the intent is good, it's not really worth 1800 lines of log spam for us. We have only a tiny number of AVX compilations in media codecs, and the only things they interchange with other code are buffer pointers.
Differential Revision: https://phabricator.services.mozilla.com/D86107
Add a modular approach for the integration of `static-analysis` module in order
to be able to share components of it with other modules, like the integration of
`clangd` in `vscode` where we need to have access to the configuration of `clang-tidy`
in order to have `in-ide` `static-analysis` messages.
In this initial step we make a separate module for the clang-tidy configuration.
Differential Revision: https://phabricator.services.mozilla.com/D85979
2020-07-27 Jan-Marek Glogowski <glogow@fbihome.de>
* lib/freebl/Makefile:
Bug 1652032 Disable all freebl assembler code for MSVC arm64
r=rrelyea,bbeurdouche
There are two places, where NSS tries to compile either x86_64 MSVC
assembler or GCC aarch64 code, which will fail the build. And also
drop the non-MSVC arch build flags for them.
AFAI could identify, there isn't any armasm64 compatible asm code in
the whole NSS library, so I don't even adapt AS for the build. The
cross-build finishes this way.
[d98bbb6168f4]
2020-07-24 Benjamin Beurdouche <bbeurdouche@mozilla.com>
* cmd/bltest/blapitest.c, coreconf/config.gypi, coreconf/config.mk,
lib/freebl/alg2268.c, lib/freebl/deprecated/alg2268.c,
lib/freebl/freebl_base.gypi, lib/freebl/ldvector.c,
lib/freebl/loader.c, lib/freebl/loader.h, lib/freebl/manifest.mn,
lib/softoken/lowpbe.c, lib/softoken/pkcs11c.c:
Bug 1652729 - Add build flag to disable RC2 and relocate to
lib/freebl/deprecated. r=kjacobs
[e6c6f1d2d544]
2020-07-27 Robert Relyea <rrelyea@redhat.com>
* gtests/softoken_gtest/manifest.mn,
gtests/softoken_gtest/softoken_dh_vectors.h,
gtests/softoken_gtest/softoken_gtest.cc,
gtests/softoken_gtest/softoken_gtest.gyp, lib/freebl/blapi.h,
lib/freebl/dh.c, lib/freebl/ldvector.c, lib/freebl/loader.c,
lib/freebl/loader.h, lib/softoken/manifest.mn,
lib/softoken/pkcs11.c, lib/softoken/pkcs11c.c,
lib/softoken/pkcs11i.h, lib/softoken/pkcs11u.c,
lib/softoken/sftkdhverify.c, lib/softoken/softoken.gyp:
Bug 1648822 Add stricter validation of DH keys when in FIPS mode.
Update: FIPS now also requires us to do y^q mod p testing on key
generation (always). We now do that in FIPS mode only, but in all
modes we do full DH verification for DH and ECDH. Because of this,
the path has now separated out the prime checks, which are now only
done for the DH operation if we aren't using a known prime and the
subprime value has been provided. I've also learned we can accept
keys that we do full validation on in FIPS mode, so I've added that
to this patch, though we still can't generate those kinds of keys
without adding the subprime at keygen time.
The new FIPS standard is dh operations must use approved primes.
Approved primes are those selected in the tls and ike RFCs.
Currently tls and ike have modes with checks whether the primes are
approved, but the check may not always happen. The safest thing to
do in FIPS mode is only allow those primes. In addition, FIPS
requires 1< y < p-1 (or technically 2<=y<=p-2, since y is an integer
those two tests are identical).
While making changes I realized we would want a mode where we can do
more strict checks on the prime while not requiring that the prime
be an approved prime. We already allow for strict checking if q is
supplied with the private key, but there were a couple of issues
with that check:
1. there was no way of actually setting q in the current NSS
pk11wrap interfaces. 2. If the prime was a safe prime, but g was an
actual generator, then we would fail the y^q mod p = 1 tests for 50%
of the keys, even though those keys are safe. 3. We weren't checking
primality of p and q.
So the old code:
if (q) { check y^q mod p = 1 if not fail }
check 1 <y < p-1 (done in DH_Derive).
New code:
if (! p is approved prime) { if (FIPS) fail; if (q) { y_test = y if
(p,q-> p is a safe prime) { y_test = 1 } check prime is prime Fail
if not check subprime is subprime fail if not y_test^q mod p = 1 } }
check 1 < y < p-1 (done in DH_Derive)
This means:
Existing code non-fips without setting the subprime continues to run
as before. Non-fips code which sets the subprime now runs slower,
but p and q are checked if p or q where not prime, the derive fails
(which it should). In FIPS mode only approved primes will succeed
now. Non-fips code can now set the subprime to q=(p-1)/2 if it
doesn't have an explicit q value (like in tls). If the derive
succeeds, we know that p is a safe prime. If p is approved, the
checks are skipped because we already know that p is a safe prime.
Code can optionally do a test derive on a new p and remember it's
safe so that we know longer need to check ever call (though if q is
not (p-1)/2, you will need to continue to do the checks each call
because y could still be a small subgroup).
This patch:
gtests/softoken_gtest
1. Added New dh tests to softoken_gtests. The tests were added to
softoken_gtests because we need to test both non-FIPS and FIPS mode.
Test vectors include a category, so the same test vectors can be
used in FIPS and non-FIPS even though each class may have different
results. Most of the test vectors where created either by dhparams
command in openssl, dsaparams in openssl, and the nss makepqg
command. Each vector includes a label, prime, base, optional
subprime, optional public key, test type, and key class (basically
size). 2. If public key is not supplied, we use a generated public
key. 3. If subPrime is supplied to wet it on the private key after
generation.
lib/freebl/dh.c
add primality tests to KEA_VerifyKey().
lib/softokn/
1. Allow CKA_SUBPRIME to be set after key generation or import.
This affects how we test for it's existance, since it is now always
there on the key, we check it's length to make sure it's non-zero.
2. We implement the psuedocode above as real code. 3. We create two
new functions: sftl_VerifyDH_Prime which return SECSuccess if Prime
is an approved prime. sftk_IsSafePrime which returns SECSuess of
both prime and subprime look reasonable, and sets a Bool to PR_TRUE
is subprime -> prime is safe (subprime = (prime-1)/2. These
functions are implemented in sftkdhverify.c 4.Cleanup incorrect
nominclature on primes (safe primes are not strong primes).
[0be91fa2217a]
* gtests/softoken_gtest/softoken_dh_vectors.h,
gtests/softoken_gtest/softoken_gtest.cc:
Fix more of the timeout issues on tests. (Drop expensive 4098 dh
tests ).
[4014c075a31b]
2020-07-29 Makoto Kato <m_kato@ga2.so-net.ne.jp>
* coreconf/config.gypi, lib/freebl/Makefile, lib/freebl/blinit.c,
lib/freebl/freebl.gyp, lib/freebl/sha1-armv8.c,
lib/freebl/sha_fast.c, lib/freebl/sha_fast.h:
Bug 1650702 - Use ARM's crypt extension for SHA1. r=kjacobs
ARM Crypto extension has SHA1 acceleration. Using this, SHA1 is 3
times faster on ARMv8 CPU. The following data is AWS's a1 instance
(Cortex-A72).
Before ====== ``` # mode in opreps cxreps context op time(sec)
thrgput sha1_e 954Mb 31M 0 0.000 10000.000 10.000 95Mb ```
After ===== ``` # mode in opreps cxreps context op time(sec) thrgput
sha1_e 2Gb 94M 0 0.000 10000.000 10.000 288Mb ```
[68b6eb737689]
2020-07-29 Jan-Marek Glogowski <glogow@fbihome.de>
* manifest.mn:
Bug 1653975 - Set "all" as the default Makefile target r=jcj,rrelyea
Just reorder the rules in manifest.mn, so all is again the first
rule. This restores pre-3.53 Makefile defaults.
[eb52747b7000]
2020-07-31 Makoto Kato <m_kato@ga2.so-net.ne.jp>
* lib/freebl/blapii.h, lib/freebl/blinit.c, nss-tool/hw-support.c:
Bug 1654142 - Add CPU feature detection for Intel SHA extension.
r=kjacobs
[e6b77a9c417a]
2020-08-03 Nathan Froyd <froydnj@mozilla.com>
* coreconf/detect_host_arch.py:
Bug 1656986 - special-case arm64 in detect_host_arch.py; r=jcj
This case comes up when attempting to build NSS on ARM64 Mac. If we
don't do this, we wind up detecting arm64 as "arm", with predictably
bad consequences.
[afa38fb2f0b5] [tip]
Differential Revision: https://phabricator.services.mozilla.com/D85888
The order in which toolchain binaries are resolved change based on environmental factors,
such as whether Firefox is being built in release mode or not.
An informative log was added in either case.
Differential Revision: https://phabricator.services.mozilla.com/D84439
This has the side effect of not initializing fontconfig before the
valgrind test itself runs, which changes the code path leading to
`FcConfigAddDirList`, which eventually leads to suppressed leaks.
Those leaks are then not discarded because the caller doesn't match what
is in the suppression file anymore, so we remove the caller.
Differential Revision: https://phabricator.services.mozilla.com/D85353
This solves the same problem we attempted to solve in bug 1654663. That was a low-cost, sensible solution when there was only one in-build reference to `glean_parser`, but with project FOG we're about to drastically increase the in-build reliance on the library, so the ad-hoc `sys.path` manipulation is an increasingly insensible solution. Here we address this in a first-class way by specifying that `glean_parser` should be imported in `virtualenv`s, but NOT by top-level `mach` commands that run outside of an in-`objdir` `virtualenv`.
Differential Revision: https://phabricator.services.mozilla.com/D85182
It turns out setting CARGO_PROFILE_RELEASE_LTO has unwanted side
effects.
First it's not actually strictly equivalent to using `cargo rustc --
-Clto`. For instance, it apparently also enables cross-language LTO in
newer versions of cargo.
Second, it changes the rust computed hash for all the dependencies of
the crate being built with the variable set, which makes them diverge
from when the same dependencies are built through another crate in the
tree that is not LTOed. This effectively makes us build a _lot_ of
crates twice, many of which are not cacheable.
Since the original problem is that cargo >= 1.45 passes extra flags (`-C
embed-bitcode=no`) to rustc that are incompatible with `-Clto`, and while
it knows to adjust based on the `lto` setting in the build profile
(which CARGO_PROFILE_RELEASE_LTO overrides the default of), cargo
ignores flags passed via `cargo rustc -- ...` when making those
adjustments.
So, we need to override with `-C embed-bitcode=yes` on our own at the
same time we pass `-Clto`. But doing that through `cargo rustc -- ...`
is not enough because all the dependencies of the crate built with
`-Clto` need to be built with `-C embed-bitcode=yes`. So we need to
override with `RUSTFLAGS`, which will affect all the dependencies.
But we also need to do this consistently across all crates, not only the
dependencies of crates built with `-Clto`, otherwise we'd still end up
building crates twice (once with and once without the override).
Unfortunately, the `-C embed-bitcode=*` flag is also not supported in
versions older than 1.45, so we have to avoid adding it on older
versions.
We unfortunately support a large range of versions of rustc (albeit only
for tools/crashreporter), but we actually need to upgrade the smaller
supported version because rustc < 1.38 doesn't support our top-level
Cargo.lock. This makes the version check slightly less awful.
Differential Revision: https://phabricator.services.mozilla.com/D84652
In preparation for Glean telemetry, we scope the availability of the out-of-date vendored
"glean_parser" library to its one usage: "run_glean_parser.py".
This allows Glean telemetry to load its modern "glean_parser" dependency from the
"--user" package environment.
Differential Revision: https://phabricator.services.mozilla.com/D84610
afa1afd410 changed a line at the edge of the context of this patch. I'm really not keen to fork this patch into a separate clang-12 version, so I'd prefer to just shrink the context a little.
Differential Revision: https://phabricator.services.mozilla.com/D84609
