This extends the shell of llvm-rc tool with the ability of tokenization
of the input files. Currently, ASCII and ASCII-compatible UTF-8 files
are supported.
Thanks to Nico Weber (thakis) for his original work in this area.
Differential Revision: https://reviews.llvm.org/D35957
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@310621 91177308-0d34-0410-b5e6-96231b3b80d8
