Consolidate all the functions which parse lists of data. They're almost the same except for the data itself, so use templates.
As for the templated function which handles the list, boost the robustness by adding additional checking and error messages.
This refactors the tolower/toupper code to use standards-conformant logic, and also adds some helpful utility functions. Regular tolower/toupper must be supplied a value that can be represented as `unsigned char`, but the std:: functions are a bit more robust.
One small detail that was overlooked in PR #3021... KeldorKatarn's old code used `strtod` while the new code used `stuff_float`. This wouldn't be a problem except that a couple MVP models use annoyingly formatted floats like 8.9e-08. So I decided to convert `stuff_float` to use `strtof` and do some more upgrades while I was at it.
For compatibility purposes some mods want alt names to truncate at the hash symbol. But we can't actually do that at mission load,
since we have to save them again in FRED. So this commit processes the names just before the mission starts.
To further complicate things, some mods actually want the hash to be displayed. So this commit introduces the double-hash as an escape sequence for a single hash.
If a message string contained a word that was too long, on its own, for the available screen real estate, the function wouldn't split the string at all. The new behavior is to split the string even if the split location is suboptimal, since it's better to have as much of the message visible as possible.
Quoting from the code:
```
// only update the line break if:
// a) we don't have a line break yet;
// b) we're still within the required real estate
// (basically we want the latest line break that doesn't go off the edge of the screen,
// but if the *first* line break is off the end of the screen, we want that)
```
When I rewrote `strip_comments()` to take quotes into account, I didn't correspondingly update `get_line_num()`. This fixes that. I discovered that a Lua file with a lot of semicolons inside quotes was erroneously getting the past-EOF warning.
Also, in the event that the warning *is* tripped, this exits the loop instead of repeatedly displaying the warning.
1) Write all non-UTF-8 strings to JSON, either by converting or truncating
2) Add `coerce_to_utf8` function
3) Tweak `read_raw_file_text` because the function returns length on success, not 0
This is a follow-up to PR #2373. Tomimaki noticed that the first character was truncated in the cutscene description, and it turned out that the algorithm was slightly wrong.
This adds the capability to have a Lua script that does not need to use
our table syntax at all but instead defines the hooks by calling API
functions. This also makes it possible to add hook functions at runtime
if the script requires that.
Also clean up some unused debug stuff.
error_display shows the line where the error was detected. Warning
itself is pretty useless for parsing related errors so I replaced all
Warnings (where appropriate) with an equivalent error_display.
This bug has been around since the original public source code release!
The error message character calculation did not agree with the actual error check. This PR fixes that. Thanks to Novachen on Discord for discovering this.
This can be used for mark parts of the code which should be unreachable
in normal execution. In debug mode, if such a statement is reached an
Error dialog is shown which aborts the execution. In release mode the
macro is replaced with a compiler specific expression that lets the
compiler know that this part is unreachable which allows it to optimize
the code better.
The code did not account for unicode escape sequences when inserting
scrambling characters which distorted the text and fequently caused
invalid UTF-8 sequences to appear in the text. This fixes that by
using the text iterators already present in the game code. I needed to
rewrite the logic of this block a bit since the previous logic could not
be converted to the way we need to handle unicode sequences at the
moment.
This fixes#1720.
This is pretty much an exact copy of the mission save code of FRED2. I
removed the campaign save code for now since it requires some
interactions with the campaign editor which does not exist yet.
This was an issue since UTF-8 encoded text can contain -128 as part of
an encoding sequence which causes a lot of issues for the parsing code.
I could not find any indication why the original code didn't use null
bytes from the start and I also couldn't find any code that would break
because of this. If anyone knows why this was done like this please let
me know.
This adds proper Unicode support by allowing the parse system to read
UTF-8 encoded files. The new mode is controlled by a mod option which
allows older mods to continue using the legacy text encoding.
That option changes the behavior of several systems in FSO. It disallows
the usage of Volition fonts since they don't support Unicode encoding
and changes a few usages of special characters to their unicode
counterparts of generic rendering solutions.
This fixes#1415.
This was reported by @AxemP who wanted to have leading tabs in a text
line. However, split_str removed those characters by default since it
ignores leading whitespace. To fix that unexpected behavior of
gr.drawString I added a new paramter to split_str which disabled
whitespace removal and enabled that flag for the scripting function.
That should not be a problem for existing scripts since the expected
behavior would be that leading whitespace remains in place and since we
have not received any bug reports about missing white space I'm going to
assume that no one has used the function like this before removed those
characters by default since it ignores leading whitespace. To fix that
unexpected behavior of gr.drawString I added a new paramter to split_str
which disabled whitespace removal and enabled that flag for the
scripting function.
That should not be a problem for existing scripts since the expected
behavior would be that leading whitespace remains in place and since we
have not received any bug reports about missing white space I'm going to
assume that no one has used the function like this before.
As far as I can tell this issue has been present for a very long time,
maybe even since retail. It was only uncovered since the recent changes
initialized the mission text by default which corrupted the table text
that was currently being parsed.
This also adds a unit test that validated that pausing works correctly.
Fabian Woltermann