A few notes:
* This doesn't accommodate general OMNIJAR_NAME definitions. The
current name (assets/omni.ja) is baked into the product in a few
places, and is very unlikely to change, so we just error out if this
isn't true.
* This makes the package-manifest.in file authoritative for what goes
into assets/, libs/, and the APK root. Previously,
package-manifest.in wrote into assets/ and libs/ but
upload-files-APK.mk also had a convoluted DIST_FILES filtering
process to work through before a file actually made it into the APK.
* This is intentional about repackaging. It simplifies the repackage
step rather than trying to make unpackage-then-repackage the same as
just package. I pretty much never get repackaging correct the first
time; this should help. (I've manually tested it.)
* The ALREADY_SZIPPED during repackaging is subsumed by the previous
check if UNPACKAGE is set. The custom linker expects stored, not
deflated, libraries, so there's some small legwork to accommodate
that in mozjar.
MozReview-Commit-ID: JvVtIUSX685
--HG--
extra : rebase_source : fd8a9cfe3dc364d23b1065995db599f99e676e38
The initial goal of templates was to provide a way to write shorter
constructs for some generic tasks during configure. But the limitations
of the sandbox and the properties of templates made them used for more
general functions.
Consequently, this led to templates having to be available from
anywhere, which, in turn, led to difficult to introspect constructs.
With bug 1257823, we've made almost everything use set_config and
similar functions from the global scope, but we don't enforce that
those primitives are only used at the global scope.
This change does that: it enforces that primitives are only used at
the global scope. Or in templates.
Now, since templates were used for other purposes than generic uses
of other primitives, we now allow non-template functions to be declared.
Those can be used everywhere, but don't have access to the sandbox
primitives.
Currently, we have @advanced, that gives the decorated functions access
to all the builtins and consequently, to the import statement.
That is a quite broad approach and doesn't allow to easily introspect
what functions are importing which modules.
This change introduces a new decorator that allows to import modules one
by one into the decorated functions.
Note: by the end of the change series, @advanced will be gone.
So far, we've been using the lowercase of the variable name, but it's
not enough for some special cases. Those special cases could do their
own business, but then, they'd have to duplicate 90% of check_prog,
which is less desirable.
While DummyFunction is descriptive of what the instances are (and they
can't even be called), the various uses of isintance(obj, DummyFunction)
are kind of confusing, especially when they are in moz.configure land
(and this bug is about to add another one).
subprocess functions doesn't directly take file-like objects, so add a
minimalistic wrapper to do the right thing instead of subprocess.call
when given a file-like object.
And since the file is also used for old-configure, close our handle on
the file before spawning old-configure, and make old-configure append
there instead of truncating the file.
DMD currently uses a very hacky form of "sampling" by default to avoid
recording stack traces for all blocks. This makes DMD run faster than when it
records all stack traces.
This patch changes the sampling method used; in fact, it avoids "sampling" at
all. The existence of all heap blocks is now recorded exactly, but by default
we only record an allocation stack for each heap block if a Bernoulli trial
succeeds. This choice works well because getting the stack trace is ~100x
slower than recording the block's existence.
Overall, this approach is simpler and it also gives better output -- the choice
of which blocks to record allocation stacks for is mathematically sound, no
stack trace gets blamed for allocations it didn't do, and block counts and
sizes are now always exact.
Other specific things changed.
- All notion of sampling is removed from the various data structures.
- The --sample-below option is removed in favour of --stacks={partial,full}.
- The format of the JSON output file has changed.
- The names of various test files have changed to reflect concept changes.
--HG--
rename : memory/replace/dmd/test/full-empty-cumulative-expected.txt => memory/replace/dmd/test/complete-empty-cumulative-expected.txt
rename : memory/replace/dmd/test/full-empty-dark-matter-expected.txt => memory/replace/dmd/test/complete-empty-dark-matter-expected.txt
rename : memory/replace/dmd/test/full-empty-live-expected.txt => memory/replace/dmd/test/complete-empty-live-expected.txt
rename : memory/replace/dmd/test/full-unsampled1-dark-matter-expected.txt => memory/replace/dmd/test/complete-full1-dark-matter-expected.txt
rename : memory/replace/dmd/test/full-unsampled1-live-expected.txt => memory/replace/dmd/test/complete-full1-live-expected.txt
rename : memory/replace/dmd/test/full-unsampled2-cumulative-expected.txt => memory/replace/dmd/test/complete-full2-cumulative-expected.txt
rename : memory/replace/dmd/test/full-unsampled2-dark-matter-expected.txt => memory/replace/dmd/test/complete-full2-dark-matter-expected.txt
rename : memory/replace/dmd/test/full-sampled-live-expected.txt => memory/replace/dmd/test/complete-partial-live-expected.txt
extra : rebase_source : 47d287405dc5e9075f08addaba49e879c2c6e23f
Before, virtualenv.py may have attempted to use 3rd party
(untrusted) pip indices when installing wheels for pip,
setuptools, and wheel. These dependencies are vendored in
the tree for a reason. So don't let virtualenv contact the
outside world.
MozReview-Commit-ID: 6BCU0WegJO1
--HG--
extra : rebase_source : 6231235486e8cfa784e67e8b18004dd63d8aaf36
While we're addressing virtualenv foo, let's ensure we are running
the latest version. This also pulls in newer versions of pip (8.1.1),
setuptools (20.3), and wheel (0.29.0).
MozReview-Commit-ID: G5uSy66Kd6u
--HG--
extra : rebase_source : 804f230adcf77335c79a93537d9623ac3836d9bf
@depends functions are declared like the following:
@depends('--option', other_function, '--other-option', ...)
def ...
To simplify some of the processing related to those arguments it's
passed, keep a tuple of Option and DummyFunction objects corresponding
to those arguments.
For the same reasons as set_config is being moved to the global scope,
we're moving set_define to the global scope here. An additional change
is that set_define is now part of the sandbox itself instead of being
defined within the sandbox, which makes it share the implementation
details with set_config.
The way set_config is set currently makes it difficult to introspect
moz.configure files to know what configuration items are being set,
because they're hidden in the control flow of functions.
This makes some of the moz.configure more convoluted, but this is why
there are templates, and we can improve the recurring cases afterwards.
Currently, ConfigureSandbox._db stores two different kind of
information. This split those in two different instance variables
instead, making things clearer.
The way functions are being sandboxed in moz.configure land is that
their global namespace is being replaced with a limited and identifiable
dict. And we avoid re-wrapping a function that already received this
treatment.
The problem is that template functions have their global namespace
replaced, and any function that is defined within the template inherits
that global namespace. So when it comes time to wrap those functions
defined in templates with e.g. depends, we detect that they're already
wrapped although they are not, because we look if their global namespace
is of the recognizable type we use when replacing it.
So instead of looking at the global namespace type, keep track of all
functions that are wrapped.
Currently, if a @depends function doesn't have a return statement or
return None, a result is automatically set from all the set_config()
called from the function.
As we're going to move set_config to the global namespace, and as this
feature is only used once, and it's only used for something that was
written before ReadOnlyNamespace was exposed to the sandbox, we can
"safely" get rid of it.
This will be needed to teach artifact builds to extract files from
omni.ja files whose content is loaded into memory (from a tar
archive).
MozReview-Commit-ID: LH2HkKx5Zj3
--HG--
extra : rebase_source : 7f6b176f0ef9fc87889151d1d02da62de8a455d8
extra : source : 266928b5a7615fa054c70adf0f649cbb3f085e8d
This change adds a `Version` type to moz.configure which is a small
wrapper around `distutils.version.Version`. It's suitable for wrapping
version numbers in configure checks and doing equality or greater-than
less-than comparisons in a sensible way.
MozReview-Commit-ID: BOL6yvemulG
--HG--
extra : rebase_source : 3b463eac0499086f8acffda0d01418b6ab17f3d6
extra : amend_source : aebd6e40c408d9f868623b2f53fcdf7455e2fff5
In Python parens around an expression without a trailing comma is not a tuple,
so ('foo') == 'foo'. This is really easy to screw up with check_progs, which
coerces progs to a list and would give you ['f','o','o'] in this case. This
patch enforces that the progs argument is a tuple or list and errors if it
is not.
MozReview-Commit-ID: 7BJZuF9B8D5
--HG--
extra : rebase_source : 5db9a6b4bb9ef7195c2513407810093bff5e9174
extra : amend_source : f67ab46c2ac00a2a95cfc67e9763ac12b690ac14
