This is a Swift feature. The output stream for the index page and the source
HTML page is utf-8 now.
The next patch will add the HTML magic to properly render these characters in
the browser.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@320725 91177308-0d34-0410-b5e6-96231b3b80d8
Most likely, this is not how we want to handle this in the long term. This
code should probably be in the Swift repo and somehow plugged into the
opt-viewer. This is still however very experimental at this point so I don't
want to over-engineer it at this point.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@319902 91177308-0d34-0410-b5e6-96231b3b80d8
Summary:
D36624 added some python3 compatibility. But that fix has a problem..
With python2 (which is specified by `#!/usr/bin/env python2.7`), if the env variables do not specify the UTF8,
and the source file is UTF8 (contains non-ASCII symbols), then the `.decode('utf-8')` causes the following exception:
```
Reading YAML files...
Rendering HTML files...
8 of 41Traceback (most recent call last):
File "/build/llvm/tools/opt-viewer/opt-viewer.py", line 277, in <module>
print_progress)
File "/build/llvm/tools/opt-viewer/opt-viewer.py", line 213, in generate_report
should_print_progress)
File "/build/llvm/tools/opt-viewer/optpmap.py", line 45, in pmap
result = map(_wrapped_func, func_and_args, *args, **kwargs)
File "/build/llvm/tools/opt-viewer/optpmap.py", line 25, in _wrapped_func
return func(argument)
File "/build/llvm/tools/opt-viewer/opt-viewer.py", line 174, in _render_file
SourceFileRenderer(source_dir, output_dir, filename).render(remarks)
File "/build/llvm/tools/opt-viewer/opt-viewer.py", line 125, in render
self.render_source_lines(self.source_stream, line_remarks)
File "/build/llvm/tools/opt-viewer/opt-viewer.py", line 79, in render_source_lines
</tr>'''.format(**locals()), file=self.stream)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xf4' in position 47: ordinal not in range(128)
```
This is similar to https://bugs.llvm.org/show_bug.cgi?id=33548, which was fixed by https://reviews.llvm.org/D37661
Unlike that fix, here, *removing* `.decode('utf-8')` actually fixes it.
Since i assume that the original fix is needed, i simply made
that fix conditional, since for python2 it actually breaks things.
Reviewers: modocache, anemet
Reviewed By: anemet
Subscribers: fhahn, llvm-commits
Differential Revision: https://reviews.llvm.org/D38289
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@315350 91177308-0d34-0410-b5e6-96231b3b80d8
I've seen cases where tiny inlined functions have such a high execution count
that most everything would show up with a relative of hotness of 0%. Since
the inlined functions effectively disappear you need to tune in the lower
range, thus we need more precision.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@314537 91177308-0d34-0410-b5e6-96231b3b80d8
This is slightly less verbose for the common case of a single build directory
and more intuitive when using this API directly from the interpreter.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@314491 91177308-0d34-0410-b5e6-96231b3b80d8
Summary:
In Python 2, calling `dict.items()` returns an indexable `list`, whereas
on Python 3 it returns a set-like `dict_items` object, which cannot be
indexed. Explicitly onvert the `dict_items` object so that it can be
indexed when using Python 3.
In combination with D36622, D36623, and D36624, this change allows
`opt-viewer.py` to exit successfully when run with Python 3.4.
Test Plan:
Run `opt-viewer.py` using Python 3.4 and confirm it does not encounter a
runtime error when when indexing into `dict.items()`.
Reviewers: anemet
Reviewed By: anemet
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D36630
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@310810 91177308-0d34-0410-b5e6-96231b3b80d8
Summary:
When using Python 3, `pygments.highlight()` returns a `bytes` object, not
a `str`, causing the call to `str.replace` on the following line to fail
with a runtime exception:
`TypeError: 'str' does not support the buffer interface`. Decode the
bytes into a string in order to fix the exception.
Test Plan:
Run `opt-viewer.py` with Python 3.4, and confirm no runtime error occurs
when calling `str.replace`.
Reviewers: anemet
Reviewed By: anemet
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D36624
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@310741 91177308-0d34-0410-b5e6-96231b3b80d8
Summary:
Replace a usage of a Python 2-specific `dict.iteritems()` with the
Python 3-compatible definition provided at the top of the same file.
Test Plan:
Run `opt-viewer.py` using Python 3 and confirm it no longer encounters a
runtime error when calling `dict.iteritems()`.
Reviewers: anemet
Reviewed By: anemet
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D36623
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@310740 91177308-0d34-0410-b5e6-96231b3b80d8
Summary:
In Python 2, `intern()` is a builtin function available to all programs.
In Python 3, it was moved into the `sys` module, available as
`sys.intern`. Import it such that, within `optrecord.py`, `intern()` is
available whether run using Python 2 or 3.
Test Plan:
Run `opt-viewer.py` using Python 3, confirm it no longer
encounters a runtime error when `intern()` is called.
Reviewers: anemet
Reviewed By: anemet
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D36622
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@310739 91177308-0d34-0410-b5e6-96231b3b80d8
The Args field of the remark which consists of a list of mappings in YAML is
translated into a list of (small) dicts on Python. An empty dict is 280 bytes
on my system so we can save memory by using a tuple of tuples instead.
Making a tuple of tuples rather than a list of tuples allows Args to be shared
with the key of the remark. This is actually an even greater saving. (Keys
are alive throughout the entire run in all_remarks.)
Here are a few opt-stats runs with different input sizes while measuring heap
usage with heapy. Avg remark size is simply estimated as
heap-size / # of remarks:
| # of files | 60 | 114 | 308 | 605 | 1370 |
| # of remarks | 20K | 37K | 146K | 180K | 640K |
| total file size (MB) | 22 | 51 | 219 | 202 | 1034 |
|------------------------+------+------+------+------+------|
| Avg remark size before | 4339 | 4792 | 4761 | 4096 | 4607 |
| Avg remark size after | 3446 | 3641 | 3567 | 3146 | 3347 |
| Rate | 0.79 | 0.76 | 0.75 | 0.77 | 0.73 |
Differential Revision: https://reviews.llvm.org/D35611
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@308538 91177308-0d34-0410-b5e6-96231b3b80d8
If heapy is installed print the "average" in-memory remark size. This is
estimated by dividing the total heap size by the number of unique remarks.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@308537 91177308-0d34-0410-b5e6-96231b3b80d8
The observation is that we have a lot of similar remarks with lots of
identical strings (e.g. file paths, text from the remark). Storing a copy of
each of those strings in memory is wasteful. This makes all the strings in
the remark interned which maintains a single immutable instance that is
referenced everywhere.
I get an average 20% heap size reduction with this but it's possible that this
varies with the typical length of the file paths used. (I used heapy to
report the heap size.) Runtime is same or a tiny bit better.
| # of files | 60 | 114 | 308 | 605 | 1370 |
| # of remarks | 20K | 37K | 146K | 180K | 640K |
| total file size (MB) | 22 | 51 | 219 | 202 | 1034 |
|-----------------------+------+------+------+------+------|
| Heap size before (MB) | 106 | 226 | 894 | 934 | 3573 |
| Heap size after | 86 | 179 | 694 | 739 | 2798 |
| Rate | 0.81 | 0.79 | 0.78 | 0.79 | 0.78 |
|-----------------------+------+------+------+------+------|
| Average remark size | 4.30 | 4.84 | 4.75 | 4.11 | 4.37 |
| Mem2disk ratio | 3.91 | 3.51 | 3.17 | 3.66 | 2.71 |
Differential Revision: https://reviews.llvm.org/D35534
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@308536 91177308-0d34-0410-b5e6-96231b3b80d8
Summary:
When using opt-viewer.py with files with '#' in their name, such as
'foo#bar.cpp', opt-viewer.py would generate links such as
'/path/to/foo#bar.cpp.opt.yaml#L42'. In this case, the link is
interpreted by browsers as a link to the file '/path/to/foo', and to the
section within that file with ID 'bar.cpp.opt.yaml#L42'.
To work around this issue, replace '#' with '_' in file names and links
in opt-viewer.py.
Reviewers: anemet, davidxl
Reviewed By: davidxl
Subscribers: llvm-commits, fhahn
Differential Revision: https://reviews.llvm.org/D34646
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@308346 91177308-0d34-0410-b5e6-96231b3b80d8
This allows to pass the build directory where all the opt.yaml files are
rather than find | xargs which may invoke opt-viewer multiple times producing
incomplete html output.
The patch generalizes the same functionality from opt-diff.
Differential Revision: https://reviews.llvm.org/D35491
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@308200 91177308-0d34-0410-b5e6-96231b3b80d8
For records without DebugLoc we simply skip the caller location in
map_remarks.
Fixes PR33764.
Patch by Simon Whittaker!
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@307880 91177308-0d34-0410-b5e6-96231b3b80d8
We weren't installing opt-viewer and co before, this fixes the omission. I am
also moving the tools from utils/ to tools/. I believe that this is more
appropriate since these tools have matured greatly in the past year through
contributions by multiple people (thanks!) so they are ready to become
external tools.
The tools are installed under <install>/share/opt-viewer/.
I am *not* adding the llvm- prefix. If people feel strongly about adding
that, this is probably a good time since the new location will require some
mental adjustment anyway.
Fixes PR33521
Differential Revision: https://reviews.llvm.org/D35048
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@307285 91177308-0d34-0410-b5e6-96231b3b80d8
