gecko-dev/python/mozlint
Andrew Halberstadt 83df36524d Bug 1494069 - [mozlint] Collapse exclude paths into their smallest possible set, r=egao
Often we specify globs in our exclude patterns, e.g:

    exclude:
        - **/node_modules
        - obj*

However, these globs get expanded out to *every* file that matches them. This
can sometimes be thousands or even tens of thousands of files.

We then pass these paths on to the underlying linters and tell them to
exclude them all. This causes a lot of overhead and slows down performance.

This commit implements a "collapse" function. Given a set of paths, it'll
collapse them into the smallest set of parent directories that contain the
original set, and that don't contain any extra files.

For example, given a directory structure like this:

    a
    -- foo.txt
    -- b
       -- bar.txt
       -- baz.txt
    -- c
       -- ham.txt
       -- d
          -- spam.txt

Then the following will happen:

     >>> collapse(['a/foo.txt', 'a/b/bar.txt', 'a/c/ham.txt', 'a/c/d/spam.txt'])
     ['a/foo.txt', 'b/bar.txt', 'c']

Since all files under directory 'c' are specified by the original set (both
'c/ham.txt' and 'c/d/spam.txt'), we can collapse it down to just 'c'. However
not all files under 'b' are specified (we're missing 'a/b/baz.txt'), so we
can't collapse 'b' (and by extension also can't collapse 'a').

If we had included 'a/b/baz.txt':

     >>> collapse(['a/foo.txt', 'a/b/bar.txt', 'a/b/baz.txt', 'a/c/ham.txt', 'a/c/d/spam.txt'])
     ['a']

In both cases, the smallest set of paths that contains the original set (and
only the original set) is computed.

The collapse function has a little bit of overhead but it's not too bad.
For example collapsing all files matched by '**/node_modules' takes ~0.015s.
Collapsing two full objdirs, takes ~0.6s. But a follow up commit is planned to
make sure we stop using 'obj*' to reduce that overhead.

Depends on D7738

Differential Revision: https://phabricator.services.mozilla.com/D7739

--HG--
extra : moz-landing-system : lando
2018-10-09 19:26:23 +00:00
..
mozlint Bug 1494069 - [mozlint] Collapse exclude paths into their smallest possible set, r=egao 2018-10-09 19:26:23 +00:00
test Bug 1494069 - [mozlint] Collapse exclude paths into their smallest possible set, r=egao 2018-10-09 19:26:23 +00:00
setup.py Bug 1397423 - Enable py2 linter on python/mozlint. r=ahal 2017-09-06 22:52:46 -04:00