Commit Graph

124 Commits

Author SHA1 Message Date
Duncan P. N. Exon Smith
660531e5f9 LTO: Export local context symbols
Export symbols in libLTO.dylib for the local context-related functions
added in r221733 (`LTO_API_VERSION=11`)... and add the missing
definition for `lto_codegen_create_in_local_context()`.

llvm-svn: 224567
2014-12-19 07:19:50 +00:00
Duncan P. N. Exon Smith
7c5c7fb60f libLTO: Allow linker to choose context of modules and codegen
Add API for specifying which `LLVMContext` each `lto_module_t` and
`lto_code_gen_t` is in.

In particular, this enables the following flow:

    for (auto &File : Files) {
      lto_module_t M = lto_module_create_in_local_context(File...);
      querySymbols(M);
      lto_module_dispose(M);
    }

    lto_code_gen_t CG = lto_codegen_create_in_local_context();
    for (auto &File : FilesToLink) {
      lto_module_t M = lto_module_create_in_codegen_context(File..., CG);
      lto_codegen_add_module(CG, M);
      lto_module_dispose(M);
    }
    lto_codegen_compile(CG);
    lto_codegen_write_merged_modules(CG, ...);
    lto_codegen_dispose(CG);

This flow has a few benefits.

  - Only one module (two if you count the combined module in the code
    generator) is in memory at a time.

  - Metadata (and constants) from files that are parsed to query symbols
    but not linked into the code generator don't pollute the global
    context.

  - The first for loop can be parallelized, since each module is in its
    own context.

  - When the code generator is disposed, the memory from LTO gets freed.

rdar://problem/18767512

llvm-svn: 221733
2014-11-11 23:19:23 +00:00
Arnold Schwaighofer
574ccf8f46 Add an option to the LTO code generator to disable vectorization during LTO
We used to always vectorize (slp and loop vectorize) in the LTO pass pipeline.

r220345 changed it so that we used the PassManager's fields 'LoopVectorize' and
'SLPVectorize' out of the desire to be able to disable vectorization using the
cl::opt flags 'vectorize-loops'/'slp-vectorize' which the before mentioned
fields default to.
Unfortunately, this turns off vectorization because those fields
default to false.
This commit adds flags to the LTO library to disable lto vectorization which
reconciles the desire to optionally disable vectorization during LTO and
the desired behavior of defaulting to enabled vectorization.

We really want tools to set PassManager flags directly to enable/disable
vectorization and not go the route via cl::opt flags *in*
PassManagerBuilder.cpp.

llvm-svn: 220652
2014-10-26 21:50:58 +00:00
Rafael Espindola
67b4b608c2 Update the error handling of lib/Linker.
Instead of passing a std::string&, use the new diagnostic infrastructure.

llvm-svn: 220608
2014-10-25 04:06:10 +00:00
Rafael Espindola
1d42bd3b29 Remove lto_codegen_set_attr.
It was never exported, so no functionality change.

llvm-svn: 214519
2014-08-01 14:57:05 +00:00
Rafael Espindola
858b9e1423 Update the MemoryBuffer API to use ErrorOr.
llvm-svn: 212405
2014-07-06 17:43:13 +00:00
Rafael Espindola
0f593fe8ae Change LTOModule`s getTargetTriple and setTargetTriple to use c++ types.
llvm-svn: 212343
2014-07-04 14:19:41 +00:00
Alp Toker
a27c4dc68d Sink undesirable LTO functions into the old C API
We want to encourage users of the C++ LTO API to reuse memory buffers instead
of repeatedly opening and reading the same file contents.

This reverts commit r212305 and implements a tidier scheme.

llvm-svn: 212308
2014-07-04 00:58:41 +00:00
Peter Collingbourne
83a67bf953 LTO: rename the various makeLTOModule overloads.
This rename makes it easier to identify the specific overload being called
in each particular case and makes future refactorings easier.

Differential Revision: http://reviews.llvm.org/D4370

llvm-svn: 212302
2014-07-03 23:28:00 +00:00
Alp Toker
65d86fdf47 Remove unused includes following r211294
llvm-svn: 211297
2014-06-19 19:25:49 +00:00
Rafael Espindola
effced1056 Use the c++ APIs.
No functionality change.

llvm-svn: 211294
2014-06-19 19:11:22 +00:00
Patrik Hagglund
e8843303a9 Fix gcc -pedantic warning in lto.cpp.
llvm-svn: 207959
2014-05-05 12:24:08 +00:00
Rafael Espindola
6884239aa9 Move LTOModule and LTOCodeGenerator to the llvm namespace.
llvm-svn: 207911
2014-05-03 14:59:52 +00:00
Rafael Espindola
c154780200 Style update: don't duplicate comments, they were getting out of sync.
llvm-svn: 207909
2014-05-03 14:34:48 +00:00
Tom Roeder
a8ef44c3ed Add an -mattr option to the gold plugin to support subtarget features in LTO
This adds support for an -mattr option to the gold plugin and to llvm-lto. This
allows the caller to specify details of the subtarget architecture, like +aes,
or +ssse3 on x86.  Note that this requires a change to the include/llvm-c/lto.h
interface: it adds a function lto_codegen_set_attr and it increments the
version of the interface.

llvm-svn: 207279
2014-04-25 21:46:51 +00:00
Duncan P. N. Exon Smith
7f2af7c18b Revert "Reapply "LTO: add API to set strategy for -internalize""
This reverts commit r199244.

Conflicts:
	include/llvm-c/lto.h
	include/llvm/LTO/LTOCodeGenerator.h
	lib/LTO/LTOCodeGenerator.cpp

llvm-svn: 205471
2014-04-02 22:05:57 +00:00
Eli Bendersky
76b2f4f843 Refactor TargetOptions initialization into a single place.
The same code (~20 lines) for initializing a TargetOptions object from CodeGen
cmdline flags is duplicated 4 times in 4 different tools. This patch moves it
into a utility function.

Since the CodeGen/CommandFlags.h file defines cl::opt flags in a header, it's
a bit of a touchy situation because we should only link them into tools. So this
patch puts the init function in the header.

llvm-svn: 201699
2014-02-19 17:09:35 +00:00
Manman Ren
5a1cf3b255 LTO API: add lto_module_create_from_memory_with_path.
This function adds an extra path argument to lto_module_create_from_memory.
The path argument will be passed to makeBuffer to make sure the MemoryBuffer
has a name and the created module has a module identifier.

This is mainly for emitting warning messages from the linker. When we emit
warning message on a module, we can use the module identifier.

rdar://15985737

llvm-svn: 201114
2014-02-10 23:26:14 +00:00
Yunzhong Gao
12926dc854 Adding new LTO APIs to parse metadata nodes and extract linker options and
dependent libraries from a bitcode module.

Differential Revision: http://llvm-reviews.chandlerc.com/D2343

llvm-svn: 199759
2014-01-21 18:31:27 +00:00
Quentin Colombet
fa288f1121 [LTO] Add a hook to map LLVM diagnostics into the clients of LTO.
Add a hook in the C API of LTO so that clients of the code generator can set
their own handler for the LLVM diagnostics.
The handler is defined like this:
typedef void (*lto_diagnostic_handler_t)(lto_codegen_diagnostic_severity_t
severity, const char *diag, void *ctxt)
- severity says how bad this is.
- diag is a string that contains the diagnostic message.
- ctxt is the registered context for this handler.

This hook is more general than the lto_get_error_message, since this function
keeps only the latest message and can only be queried when something went wrong
(no warning for instance).

<rdar://problem/15517596>

llvm-svn: 199338
2014-01-15 22:04:35 +00:00
Duncan P. N. Exon Smith
bb847bd59e Reapply "LTO: add API to set strategy for -internalize"
Reapply r199191, reverted in r199197 because it carelessly broke
Other/link-opts.ll.  The problem was that calling
createInternalizePass("main") would select
createInternalizePass(bool("main")) instead of
createInternalizePass(ArrayRef<const char *>("main")).  This commit
fixes the bug.

The original commit message follows.

Add API to LTOCodeGenerator to specify a strategy for the -internalize
pass.

This is a new attempt at Bill's change in r185882, which he reverted in
r188029 due to problems with the gold linker.  This puts the onus on the
linker to decide whether (and what) to internalize.

In particular, running internalize before outputting an object file may
change a 'weak' symbol into an internal one, even though that symbol
could be needed by an external object file --- e.g., with arclite.

This patch enables three strategies:

- LTO_INTERNALIZE_FULL: the default (and the old behaviour).
- LTO_INTERNALIZE_NONE: skip -internalize.
- LTO_INTERNALIZE_HIDDEN: only -internalize symbols with hidden
  visibility.

LTO_INTERNALIZE_FULL should be used when linking an executable.

Outputting an object file (e.g., via ld -r) is more complicated, and
depends on whether hidden symbols should be internalized.  E.g., for
ld -r, LTO_INTERNALIZE_NONE can be used when -keep_private_externs, and
LTO_INTERNALIZE_HIDDEN can be used otherwise.  However,
LTO_INTERNALIZE_FULL is inappropriate, since the output object file will
eventually need to link with others.

lto_codegen_set_internalize_strategy() sets the strategy for subsequent
calls to lto_codegen_write_merged_modules() and lto_codegen_compile*().

<rdar://problem/14334895>

llvm-svn: 199244
2014-01-14 18:52:17 +00:00
NAKAMURA Takumi
068c8352f7 Revert r199191, "LTO: add API to set strategy for -internalize"
Please update also Other/link-opts.ll, in next time.

llvm-svn: 199197
2014-01-14 09:40:18 +00:00
Duncan P. N. Exon Smith
95dadb39e4 LTO: add API to set strategy for -internalize
Add API to LTOCodeGenerator to specify a strategy for the -internalize
pass.

This is a new attempt at Bill's change in r185882, which he reverted in
r188029 due to problems with the gold linker.  This puts the onus on the
linker to decide whether (and what) to internalize.

In particular, running internalize before outputting an object file may
change a 'weak' symbol into an internal one, even though that symbol
could be needed by an external object file --- e.g., with arclite.

This patch enables three strategies:

- LTO_INTERNALIZE_FULL: the default (and the old behaviour).
- LTO_INTERNALIZE_NONE: skip -internalize.
- LTO_INTERNALIZE_HIDDEN: only -internalize symbols with hidden
  visibility.

LTO_INTERNALIZE_FULL should be used when linking an executable.

Outputting an object file (e.g., via ld -r) is more complicated, and
depends on whether hidden symbols should be internalized.  E.g., for
ld -r, LTO_INTERNALIZE_NONE can be used when -keep_private_externs, and
LTO_INTERNALIZE_HIDDEN can be used otherwise.  However,
LTO_INTERNALIZE_FULL is inappropriate, since the output object file will
eventually need to link with others.

lto_codegen_set_internalize_strategy() sets the strategy for subsequent
calls to lto_codegen_write_merged_modules() and lto_codegen_compile*().

<rdar://problem/14334895>

llvm-svn: 199191
2014-01-14 06:37:26 +00:00
Chandler Carruth
03b6c941a3 Re-sort #include lines again, prior to moving headers around.
llvm-svn: 199080
2014-01-13 08:04:33 +00:00
Rafael Espindola
24353f2de2 Use LTO_SYMBOL_SCOPE_DEFAULT_CAN_BE_HIDDEN instead of the "dso list".
There are two ways one could implement hiding of linkonce_odr symbols in LTO:
* LLVM tells the linker which symbols can be hidden if not used from native
  files.
* The linker tells LLVM which symbols are not used from other object files,
  but will be put in the dso symbol table if present.

GOLD's API is the second option. It was implemented almost 1:1 in llvm by
passing the list down to internalize.

LLVM already had partial support for the first option. It is also very similar
to how ld64 handles hiding these symbols when *not* doing LTO.

This patch then
* removes the APIs for the DSO list.
* marks LTO_SYMBOL_SCOPE_DEFAULT_CAN_BE_HIDDEN all linkonce_odr unnamed_addr
  global values and other linkonce_odr whose address is not used.
* makes the gold plugin responsible for handling the API mismatch.

llvm-svn: 193800
2013-10-31 20:51:58 +00:00
Reid Kleckner
69aecb4269 lto.h: Use lto_bool_t instead of int to restore the ABI
This reverts commit r193255 and instead creates an lto_bool_t typedef
that points to bool, _Bool, or unsigned char depending on what is
available.  Only recent versions of MSVC provide a stdbool.h header.

Reviewers: rafael.espindola

Differential Revision: http://llvm-reviews.chandlerc.com/D2019

llvm-svn: 193377
2013-10-24 22:26:04 +00:00
NAKAMURA Takumi
a198c22185 llvm-c/lto.h: Avoid use of bool.
llvm-svn: 193255
2013-10-23 17:56:46 +00:00
Rafael Espindola
a88c415234 Optimize linkonce_odr unnamed_addr functions during LTO.
Generalize the API so we can distinguish symbols that are needed just for a DSO
symbol table from those that are used from some native .o.

The symbols that are only wanted for the dso symbol table can be dropped if
llvm can prove every other dso has a copy (linkonce_odr) and the address is not
important (unnamed_addr).

llvm-svn: 191922
2013-10-03 18:29:09 +00:00
Rafael Espindola
83e2fc9be0 Fix option parsing in the gold plugin.
This was broken when options were moved up in r191680. No test because this is
specific LLVMgold.so/libLTO.so.

Patch by Tom Roeder!

llvm-svn: 191829
2013-10-02 14:36:23 +00:00
Rafael Espindola
c7fe26d6e4 Move command line options to the users of libLTO. Fixes --enable-shared build.
Patch by Richard Sandiford.

llvm-svn: 191680
2013-09-30 16:39:19 +00:00
Peter Collingbourne
8969240f9e Move LTO support library to a component, allowing it to be tested
more reliably across platforms.  Patch by Tom Roeder!

llvm-svn: 191343
2013-09-24 23:52:22 +00:00
Shuxin Yang
07a72e63c6 Revert 189297, the original commit message is following.
----
  Add new API lto_codegen_compile_parallel().

  This API is proposed by Nick Kledzik. The semantic is:

  --------------------------------------------------------------------------
   Generate code for merged module into an array of native object files. On 
   success returns a pointer to an array of NativeObjectFile.  The count 
   parameter returns the number of elements in the array.  Each element is 
   a pointer/length for a generated mach-o/ELF buffer.  The buffer is owned
   by the lto_code_gen_t and will be freed when lto_codegen_dispose() is called, 
   or lto_codegen_compile() is called again. On failure, returns NULL 
   (check lto_get_error_message() for details).

   extern const struct  NativeObjectFile*
   lto_codegen_compile_parallel(lto_code_gen_t cg, size_t *count);
  ---------------------------------------------------------------------------

  This API is currently only called on OSX platform. Linux or other Unixes
using GNU gold are not supposed to call this function, because on these systems, 
object files are fed back to linker via disk file instead of memory buffer.

  In this commit, lto_codegen_compile_parallel() simply calls
lto_codegen_compile() to return a single object file. In the near future,
this function is the entry point for compilation with partition. Linker can
blindly call this function even if partition is turned off; in this case,
compiler will return only one object file.

llvm-svn: 189386
2013-08-27 17:15:54 +00:00
Shuxin Yang
da579e3471 Add new API lto_codegen_compile_parallel().
This API is proposed by Nick Kledzik. The semantic is:

  --------------------------------------------------------------------------
   Generate code for merged module into an array of native object files. On 
   success returns a pointer to an array of NativeObjectFile.  The count 
   parameter returns the number of elements in the array.  Each element is 
   a pointer/length for a generated mach-o/ELF buffer.  The buffer is owned
   by the lto_code_gen_t and will be freed when lto_codegen_dispose() is called, 
   or lto_codegen_compile() is called again. On failure, returns NULL 
   (check lto_get_error_message() for details).

   extern const struct  NativeObjectFile*
   lto_codegen_compile_parallel(lto_code_gen_t cg, size_t *count);
  ---------------------------------------------------------------------------

  This API is currently only called on OSX platform. Linux or other Unixes
using GNU gold are not supposed to call this function, because on these systems, 
object files are fed back to linker via disk file instead of memory buffer.

  In this commit, lto_codegen_compile_parallel() simply calls
lto_codegen_compile() to return a single object file. In the near future,
this function is the entry point for compilation with partition. Linker can
blindly call this function even if partition is turned off; in this case,
compiler will return only one object file.

llvm-svn: 189297
2013-08-27 00:03:23 +00:00
Shuxin Yang
b3af4b7f36 Revert r188188 and r188200.
In order to appease people (in Apple) who accuse me for committing "huge change" (?) without proper review. 

Thank Eric for fixing a compile-warning. 

llvm-svn: 188204
2013-08-12 21:07:31 +00:00
Shuxin Yang
4b4ff9ec1f Misc enhancements to LTO:
1. Add some helper classes for partitions. They are designed in a
     way such that the top-level LTO driver will not see much difference 
     with or without partitioning.

  2. Introduce work-dir. Now all intermediate files generated during 
     LTO phases will be saved under work-dir. User can specify the workdir
     via -lto-workdir=/path/to/dir. By default the work-dir will be 
     erased before linker exit. To keep the workdir, do -lto-keep, or -lto-keep=1.

    TODO: Erase the workdir, if the linker exit prematurely.  
      We are currently not able to remove directory on signal. The support 
      routines simply ignore directory.

  3. Add one new API lto_codegen_get_files_need_remove().
     Linker and LTO plugin will communicate via this API about which files
    (including directories) need to removed before linker exit.

llvm-svn: 188188
2013-08-12 18:29:43 +00:00
Shuxin Yang
a9213839b0 Change public functions of LTOCodeGenerator from ret-false-on-succ to ret-true-on-succ.
As of this revision, all functions of LTOCodeGenerator are consistent in
ret-true-on-succ.

Tested on multiple OSes.

llvm-svn: 187864
2013-08-07 05:19:23 +00:00
Rafael Espindola
f3a9f67eee Split getOpenFile into getOpenFile and getOpenFileSlice.
The main observation is that we never need both the filesize and the map size.
When mapping a slice of a file, it doesn't make sense to request a null
terminator and that would be the only case where the filesize would be used.

There are other cleanups that should be done in this area:

* A client should not have to pass the size (even an explicit -1) to say if
  it wants a null terminator or not, so we should probably swap the argument
  order.
* The default should be to not require a null terminator. Very few clients
  require this, but many end up asking for it just because it is the default.

llvm-svn: 186984
2013-07-23 20:25:01 +00:00
Bill Wendling
0af6f08453 Revert r169656.
The linker will call `lto_codegen_add_must_preserve_symbol' on all globals that
should be kept around. The linker will pretend that a dylib is being created.
<rdar://problem/12528059>

llvm-svn: 169770
2012-12-10 21:33:45 +00:00
Bill Wendling
6026bd1239 Add the `lto_codegen_set_export_dynamic' function.
This function sets the `_exportDynamic' ivar. When that's set, we export all
symbols (e.g. we don't run the internalize pass). This is equivalent to the
`--export-dynamic' linker flag in GNU land:

--export-dynamic
  When creating a dynamically linked executable, add all symbols to the dynamic
  symbol table. The dynamic symbol table is the set of symbols which are visible
  from dynamic objects at run time. If you do not use this option, the dynamic
  symbol table will normally contain only those symbols which are referenced by
  some dynamic object mentioned in the link. If you use dlopen to load a dynamic
  object which needs to refer back to the symbols defined by the program, rather
  than some other dynamic object, then you will probably need to use this option
  when linking the program itself.

The Darwin linker will support this via the `-export_dynamic' flag. We should
modify clang to support this via the `-rdynamic' flag.

llvm-svn: 169656
2012-12-08 00:18:16 +00:00
Chandler Carruth
800daa7d3d Sort the #include lines for tools/...
Again, tools are trickier to pick the main module header for than
library source files. I've started to follow the pattern of using
LLVMContext.h when it is included as a stub for program source files.

llvm-svn: 169252
2012-12-04 10:44:52 +00:00
Rafael Espindola
3c1722e285 Remove lto_codegen_set_whole_program_optimization. It is a work in progress,
so we don't want it to show up in the stable 3.1 interface.

While at it, add a comment about why LTOCodeGenerator manually creates the
internalize pass.

llvm-svn: 154807
2012-04-16 10:58:38 +00:00
Bill Wendling
756a33b1e6 s/lto_codegen_whole_program_optimization/lto_codegen_set_whole_program_optimization/
llvm-svn: 154312
2012-04-09 08:32:21 +00:00
Bill Wendling
adcb1f1d29 Add a hook to turn on the internalize pass through the LTO interface.
llvm-svn: 154306
2012-04-09 05:26:48 +00:00
Bill Wendling
cbcec066af Cleanup whitespace and remove unneeded 'extern' keyword on function definitions.
llvm-svn: 153802
2012-03-31 10:44:20 +00:00
Bill Wendling
619da9f4da Cleanup whitespace. Doxygenize comments. And indent to llvm coding standards.
llvm-svn: 153740
2012-03-30 10:29:38 +00:00
Chris Lattner
0304b82f80 Fix a ton of comment typos found by codespell. Patch by
Luis Felipe Strano Moraes!

llvm-svn: 129558
2011-04-15 05:18:47 +00:00
Rafael Espindola
1592f3ae24 Add a lto_codegen_compile_to_file to avoid producing a file, reading it to
memory and writing it back to disk.

llvm-svn: 128108
2011-03-22 20:57:13 +00:00
Rafael Espindola
8dab9a762b Add support in the LTO library for loading an object from the middle
of an file.

llvm-svn: 127781
2011-03-17 00:36:11 +00:00
Rafael Espindola
58423e8aad Switch LTO to use MC. This takes the linking of libxul.so from about 7m to
6m30.

llvm-svn: 126426
2011-02-24 21:04:06 +00:00
Rafael Espindola
43f9672afa Don't open the file again in the gold plugin. To be able to do this, update
MemoryBuffer::getOpenFile to not close the file descriptor.

llvm-svn: 125128
2011-02-08 22:40:47 +00:00