code documentation, part 2

CompatBot/Commands/Attributes/readme.md

@@ -0,0 +1,28 @@
+Command Attributes
+==================
+
+This folder contains various custom attributes that could be used with command groups (classes) and commands themselves (functions).
+
+They're intended for easier management of said commands, such as implementing access rights management and spam reduction policies.
+
+`CheckBaseAttributeWithReactions`
+---------------------------------
+
+This is a base class that implements an ability to create Discord reactions in case the check has passed or failed, and also logs the check itself for audit purposes.
+
+`LimitedToXyzChannel`, `RequiresDm`, `RequiresNotMedia`
+-------------------------------------------------------
+
+These attributes are intended to limit potential impact of command usage outside of specific channels. There are some allowances for users with "trusted" roles sometimes, but in general it's an easy way to make sure bot won't reply when it is not needed.
+
+`RequiresXyzRole`
+-----------------
+
+Similarly implement command access rights management to prevent their use by regular users. As a bonus, every user will only get the list of commands that they can actually use when using `!help`.
+
+`TriggersTyping`
+---------------
+
+This is a legacy attribute that will trigger `Typing...` message at the bottom of the chat window. It was used to indicate that the bot is working on the command, but it has several issues that prevents it from being very useful (you can't control how long it will be shown, and it eats up an API call).
+
+Generally speaking bot replies too fast to have any special indicator for most commands. Whenever it is needed, it is better to create a reaction, or post and then update an explicit message.

CompatBot/Commands/ContentFilters.cs

@@ -22,7 +22,7 @@ namespace CompatBot.Commands
 {
     [Group("filters"), Aliases("piracy", "filter"), RequiresBotSudoerRole, RequiresDm]
     [Description("Used to manage content filters. **Works only in DM**")]
-    internal sealed class Antipiracy: BaseCommandModuleCustom
+    internal sealed class ContentFilters: BaseCommandModuleCustom
     {
         private static readonly TimeSpan InteractTimeout = TimeSpan.FromMinutes(5);
         private static readonly char[] Separators = {' ', ',', ';', '|'};

CompatBot/Commands/Converters/readme.md

@@ -0,0 +1,9 @@
+Converters
+==========
+
+Here we have custom converter classes that could be used with DSharpPlus library.
+
+`TextOnlyDiscordChannelConverter`
+---------------------------------
+
+This converter is used to more easily parse channel names to their respective library object, that prevents most common issues (such as trying to send messages in a voice or a category channel).

CompatBot/Commands/readme.md

@@ -0,0 +1,107 @@
+Commands
+========
+
+The potatoes and gravy of the bot. All of them _should_ be inherited from the `BaseCommandModuleCustom` class for uniform command management and permission model.
+
+`BotMath`
+---------
+This command is intended to parse and evaluate various math expressions.
+The implementation is based on [mXparser](https://mathparser.org/).
+
+`BotStats`
+----------
+This command is intended to get general bot status information with some fun statistics thrown in.
+
+`CommandsManagement`
+--------------------
+This command group is intended to manage other command policies. The purpose of which is to be able disable and enable specific commands or groups of commands at runtime to prevent bot abuse and exploitation by regular users until the issue could be properly fixed.
+
+`CompatList`
+------------
+Command group dedicated for [game compatibility API](Clients/CompatApiClient).
+Game listing is heavily relied on [fuzzy string matching](CompatBot/Utils/Extensions#StringUtils) to sort by similarity score to the original request string.
+
+One fun feature is to prevent bot abuse by one specific user, that's just a home grown meme.
+
+`ContentFilters` aka `Antipiracy`
+---------------------------------
+One of the more useful moderation command groups that allows content filtering in Discord text channels, as well as for screening uploaded user log files for support purposes.
+
+For the most part it is straightforward, except for the filter editor logic in `EditFilterPropertiesAsync()`.
+It is structured as a simple state machine with labels denoting the nodes.
+
+The gist of it is to have a wizard-like experience with multiple stages to edit the object model in a manner that would help moderators to correctly fill out all the necessary data. ~~It's also a fun abuse of Discord embeds.~~
+
+`EventsBaseCommand`, `Events`, `E3`, `Cyberpunk2077`
+----------------------------------------------------
+`EventsBaseCommand` implements simple calendar/scheduling system to track various events. It is mostly intended to show various gaming-related events and link the associated livestreams.
+It also implements the event object model editor in a similar fashion as `ContentFilters`.
+
+`Events` command group then implements general event management commands, and `E3` and `Cyberpunk2077` are simple aliases for easier use.
+
+`Explain`
+---------
+One of the more useful command groups, mainly used for tech support.
+It is providing the means to define and show short explanation, a sort of FAQ or a wiki.
+
+Each explanation has a `term` that defines the entry, and is how you can recall the explanation body. When searching for an explanation, if no direct match to term is found, some [fuzzy text match logic](CompatBot/Utils/Extensions#StringUtils) is applied.
+
+There's also an ability to attach a file to each explain. Media files are automatically recognized by the clients, and shown inline. This includes spoiler marks (which is implemented by simply prepending `SPOILER_` to the filename).
+
+`Invites`
+---------
+This command group is dedicated to management of Discord invite whitelist. By default any invite link will be removed, but moderators can add target servers to the whitelist to allow invite sharing by regular users.
+
+`Ird`
+-----
+Simple command to look up available items from the [IRD Library](Clients/IrdLibraryClient).
+
+`Misc`
+------
+A fun pile of random commands, including:
+* `!about` to show people contributed to the bot in some way
+* `!roll` to generate random number from 1 to specified value, or to cast a bunch of [dice](https://en.wikipedia.org/wiki/Dice#Polyhedral_dice)
+* `!8ball` to randomly pick one of the predefined answers for a yes/no question
+* `!rate` to randomly pick one of the predefined quality answers, using data-driven seed to produce consistent reply for the same question
+* `!download` is a meme/bait command that is aliased to `!psn search`
+
+`Moderation`
+------------
+Command group for various moderation-related commands, including:
+* `!report` to link any message to the moderation queue
+* `!analyze` to force log analysis in case it is needed again
+* `!badupdate` to mark update announcement as not good for general consumption
+
+Audit commands are used to check various suspicious stuff like users trying to impersonate staff, or users abusing Unicode combining characters to produce [zalgo](https://www.urbandictionary.com/define.php?term=Zalgo).
+
+`Pr`
+----
+This command is used to query GitHub for any open pull request that meets the specified text filter.
+First word in the query is additionally treated as a PR author.
+In case only one items is returned, it is show as an embed with the link to the latest available artifact download from AppVeyor.
+
+`Psn`
+-----
+Various (PSN-related)[Clients/PsnClient] commands.
+Most commonly used is the `!psn check updates` that is using PSN game update API to show links for the game updates.
+
+Same kind of anti-abuse measures is implemented here, as for the [`CompatList`](#CompatList).
+
+`Sudo`
+------
+Commands for the bot management ~~and abuse by mods~~.
+`say` and `react` will make the bot to post a specified message or reaction, `log` will attach current log file for easier access.
+
+`mod` subgroup is for managing bot roles, which are currently separated from Discord roles.
+
+`fix` subgroup is a technical group to help fix various data in bot state.
+
+`bot` subgroup is for managing the bot instance itself: `stop`, `restart`, `update` are self-explanatory; `status` can set Discord status that will show up on bot profile.
+
+`Syscall`
+---------
+Command to query information about what games used what functions from the PS3 kernel and firmware modules.
+
+`Warnings`
+----------
+Warning system management commands. You can look at various warning lists, as well ass issue and retract warnings for specific users.

CompatBot/Database/Providers/readme.md

@@ -0,0 +1,65 @@
+Data Providers
+==============
+
+These classes should ease the data access and manipulation. If no special consideration is required, you should use database access directly.
+
+`AmdDriverVersionProvider`
+--------------------------
+This class builds and keeps the AMD driver version mappings for Vulkan and OpenGL using [GPUOpen project](https://github.com/GPUOpen-Drivers) as a source.
+
+AMD has many gotchas with their driver versioning, included, but not limited to:
+* existing versioning information goes only as far back as 18.1.1 release
+* OpenGL driver version may differ from actual driver version
+  * `major` part (**m**.x.y.z) seems to be following the [Windows Driver versioning](https://docs.microsoft.com/en-us/windows-hardware/drivers/display/version-numbers-for-display-drivers) scheme
+  * `build` part (x.y.z.**b**) may vary depending on different driver package (desktop vs mobile vs minor re-releases)
+* Vulkan driver versioning is completely separate and may span several driver releases
+
+`ContentFilter`
+---------------
+This class implements actual content filter using the pre-defined filter descriptions.
+
+As you can probably guess, simply iterating over each filter trigger is not gonna cut it if you have more than 3 filters and a lot of text to moderate.
+This is where [Aho-Corasick](https://en.wikipedia.org/wiki/Aho%E2%80%93Corasick_algorithm) algorithm comes into play. It allows for linear complexity over the _filtered text length_.
+
+On bot start up and every change to the filter list, this provider will reconstruct the Aho-Corasick state machine that is used for content filtering.
+As an implementation detail, we construct one state machine per filter context (Discord messages or log file).
+
+The idea here is to feed the state machine with the text we want to check. It will invoke the callback on any trigger match, where we can decide if the filter should be applied or not.
+
+As a convenience we have a static methods that will wrap all the details inside, and will perform all the required actions when necessary.
+
+`DisabledCommandsProvider`
+--------------------------
+This provider wraps the complexity of bot command framework enumeration in case of a wildcard command matching, and also for synchronization of in-memory hash map and persistent list in the database.
+
+`InviteWhitelistProvider`
+-------------------------
+This is mostly needed for maintenance of expired invite codes.
+
+`ModProvider`
+-------------
+This is mostly a legacy provider for managing the bot roles.
+
+`ScrapeStateProvider`
+---------------------
+This provider wraps up the logic of refreshing the crawler state for different PSN categories, thumbnail caches, etc.
+
+`StatsStorage`
+--------------
+Contains logic for serialization and deserialization of in-memory stat tracking.
+One fun part about serialization is that it is using reflection to access non-public member of the cache class to get the actual data.
+
+`SyscallInfoProvider`
+---------------------
+This is mostly needed to wrap up the complexity of persisting the function call data gathered from the logs, as it must be aware of already existing data to insertion of non-unique keys.
+
+`ThumbnailProvider`
+-------------------
+This one will:
+* look up thumbnail image from multiple sources
+* check if it's current
+* check if it can be embedded by Discord (it requires a proper file extension)
+  * re-upload it to a private channel for persistent caching if needed
+* update the associated tables in the database
+
+And at the end it will return the readily usable URL.

CompatBot/Database/readme.md

@@ -0,0 +1,69 @@
+Databases
+=========
+Currently we use [Entity Framework Core](https://docs.microsoft.com/en-us/ef/core/) as an [O/RM](https://en.wikipedia.org/wiki/Object-relational_mapping) and SQLite as a storage.
+Among easier code generation, ef core allows for automated [database schema migrations](https://docs.microsoft.com/en-us/ef/core/managing-schemas/migrations/) that are persisted in the `Migrations` folder.
+
+There are some [data provider classes](Providers) for easier access and management, but most of the time you'll see direct access an manipulation of the data throughout the code base.
+
+After changing the classes you need to generate a new migration using dotnet ef tools
+```
+$ dotnet ef migrations add -c <DbClass> <MigrationName>
+```
+Note that ef tools come bundled with dotnet sdk 2.x, but require [manual installation](https://docs.microsoft.com/en-us/ef/core/miscellaneous/cli/dotnet) starting dotnet sdk 3.0.
+
+`BotDb`, `ThumbnailDb`
+-----------------------
+Two main database description classes used to store settings and PSN metadata respectively.
+
+Note that due to legacy requirements, every table has a numeric primary key with auto increment value, even if it is never used.
+
+`BotDb`
+-------
+Contains the following tables that hold bot runtime configuration:
+
+### `bot-state`
+A simple key-value table for saving any miscellaneous data that is needed.
+
+### `moderator`
+Stores bot roles (mods and sudoers).
+
+### `piracystring`
+Stores content moderation filter descriptions, including filter trigger string, trigger context, validating regex, custom message, and required action flags.
+
+### `warning`
+Stores user warning history, including user IDs, publicly visible comment on the reason, and privately visible context in case of automated warnings issued by the bot.
+
+### `explanation`
+Stores explanation descriptions and data: term as a key, text for an explanation, and optional file attachment as a binary blob.
+
+### `disabled-commands`, `whitelisted-invites`
+Simple lists of the associated data.
+
+### `event-schedule`
+Contains information about the events: date and time intervals, titles, etc.
+
+### `stats`
+This table is used for persisting stat caches in case of process restart.
+
+`ThumbnailDb`
+--------------
+Contains mostly PSN-related metadata in the following tables:
+
+### `state`
+A key-value storage of the PSN crawling state, and some other named time stamps.
+
+### `thumbnail`
+Contains mapping for the [product code](http://www.psdevwiki.com/ps3/Productcode), [content ID](https://www.psdevwiki.com/ps3/Content_ID), source thumbnail URL, and re-uploaded cached thumbnail URL.
+
+### `title-info`
+??? Seems to duplicate `thumbnail` table, with added optional column for the embed color. TBH I don't remember how this happened. I need to remove this later.
+
+### `syscall-info`
+Contains information about what game used what function from what firmware or kernel module.
+
+### `syscall-to-product-map`
+This is a utility table that contains foreign keys that tie `thumbnail` and `syscall-info` tables together to allow required `JOIN`s for queries.
+
+`DbImporter`
+------------
+This class is handling all the import logic from the legacy python version of the bot, and also applying all the available migrations on program start up.

CompatBot/Program.cs

@@ -101,7 +101,7 @@ namespace CompatBot
                     commands.RegisterCommands<CompatList>();
                     commands.RegisterCommands<Sudo>();
                     commands.RegisterCommands<CommandsManagement>();
-                    commands.RegisterCommands<Antipiracy>();
+                    commands.RegisterCommands<ContentFilters>();
                     commands.RegisterCommands<Warnings>();
                     commands.RegisterCommands<Explain>();
                     commands.RegisterCommands<Psn>();