The code did not account for unicode escape sequences when inserting
scrambling characters which distorted the text and fequently caused
invalid UTF-8 sequences to appear in the text. This fixes that by
using the text iterators already present in the game code. I needed to
rewrite the logic of this block a bit since the previous logic could not
be converted to the way we need to handle unicode sequences at the
moment.
This fixes#1720.
This adds proper Unicode support by allowing the parse system to read
UTF-8 encoded files. The new mode is controlled by a mod option which
allows older mods to continue using the legacy text encoding.
That option changes the behavior of several systems in FSO. It disallows
the usage of Volition fonts since they don't support Unicode encoding
and changes a few usages of special characters to their unicode
counterparts of generic rendering solutions.
This fixes#1415.
asarium