fix(core): strip a leading BOM before converting a TOML command (#13975)

* fix(core): strip a leading BOM before converting a TOML command

`@iarna/toml` rejects U+FEFF, so a `.toml` custom command saved with a
UTF-8 BOM — what Notepad and several editor encodings write on Windows —
threw `Unknown character "65279" at row 1, col 2` out of
`convertTomlToMarkdown`. Both callers swallow that into a skip, so the
command simply never appeared, while the same content saved without a BOM
worked. The two read paths that reach the converter are the Gemini CLI
command migration and the extension TOML command loader, and both hand it
text read straight off disk.

Strip one leading BOM before parsing, the way the config readers and
`normalizeContent` already do. Only the leading character is touched, so a
U+FEFF anywhere else in the file is preserved, and line endings are left
alone.

This is the conversion path only. The custom-command loader reads its TOML
in a different file and is fixed separately, so the issue is not closed
here.

Part of #13944

Co-authored-by: chatgpt-codex-connector[bot] <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>

* fix(core): make isTomlFormat accept BOM-prefixed TOML too

* fix(core): make isTomlFormat accept BOM-prefixed TOML too

---------

Co-authored-by: F415643 <[email protected]>
Co-authored-by: chatgpt-codex-connector[bot] <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
Co-authored-by: Shaojin Wen <[email protected]>

packages/core/src/utils/toml-to-markdown-converter.test.ts

@@ -54,6 +54,25 @@ description = "Multi-line test"`;
     expect(result).toContain('description: Multi-line test');
   });
 
+  it('should convert TOML saved with a UTF-8 BOM', () => {
+    // Windows editors write the BOM, and `@iarna/toml` rejects U+FEFF with
+    // `Unknown character "65279" at row 1, col 2`, so the command was dropped
+    // before it ever reached Markdown.
+    const tomlContent = `\uFEFFprompt = "BOM prompt"
+description = "BOM command"`;
+
+    const result = convertTomlToMarkdown(tomlContent);
+
+    expect(readDescriptionFromFrontmatter(result)).toBe('BOM command');
+    expect(result).toContain('BOM prompt');
+  });
+
+  it('should keep a BOM-like character that is not the leading one', () => {
+    const tomlContent = `prompt = "a\uFEFFb"`;
+
+    expect(convertTomlToMarkdown(tomlContent)).toBe('a\uFEFFb\n');
+  });
+
   it('should throw error for invalid TOML', () => {
     const invalidToml = 'this is not valid toml {[}]';
 
@@ -120,4 +139,14 @@ Prompt content`;
 
     expect(isTomlFormat(markdown)).toBe(false);
   });
+
+  it('agrees with convertTomlToMarkdown on BOM-prefixed input', () => {
+    // The two exported functions must not disagree: a consumer composing them
+    // as `if (isTomlFormat(c)) convertTomlToMarkdown(c)` would otherwise skip a
+    // command file a Windows editor saved with a BOM.
+    const withBom = '\uFEFFprompt = "x"';
+
+    expect(isTomlFormat(withBom)).toBe(true);
+    expect(convertTomlToMarkdown(withBom)).toBe('x\n');
+  });
 });

packages/core/src/utils/toml-to-markdown-converter.ts

@@ -25,7 +25,11 @@ export interface TomlCommandFormat {
 export function convertTomlToMarkdown(tomlContent: string): string {
   let parsed: unknown;
   try {
-    parsed = toml.parse(tomlContent);
+    // `@iarna/toml` rejects U+FEFF, so a BOM that a Windows editor wrote makes
+    // an otherwise valid command file unparseable and the caller drops it.
+    // Strip one leading BOM, as the config readers do, and leave line endings
+    // and every other character alone.
+    parsed = toml.parse(tomlContent.replace(/^\uFEFF/, ''));
   } catch (error) {
     throw new Error(
       `Failed to parse TOML: ${error instanceof Error ? error.message : String(error)}`,
@@ -72,7 +76,11 @@ ${prompt}
  */
 export function isTomlFormat(content: string): boolean {
   try {
-    toml.parse(content);
+    // Strip one leading BOM so this detector agrees with
+    // `convertTomlToMarkdown`, which accepts BOM-prefixed files. Otherwise the
+    // natural `if (isTomlFormat(c)) convertTomlToMarkdown(c)` pair would skip a
+    // command a Windows editor saved with a BOM.
+    toml.parse(content.replace(/^\uFEFF/, ''));
     return true;
   } catch {
     return false;