diff --git a/content/1.features/knowledge-stack/1.basics.md b/content/1.features/knowledge-stack/1.basics.md new file mode 100644 index 0000000..4a233fd --- /dev/null +++ b/content/1.features/knowledge-stack/1.basics.md @@ -0,0 +1,152 @@ +--- +title: Knowledge Stack Basics +description: Get started with Knowledge Stacks in Msty +navTruncate: false +--- + +Knowledge Stacks let you chat with your content - from simple documents to entire knowledge bases. Think of it as having a smart assistant that has read and understood all your materials. + +::alert{type="info" icon="tabler:info-circle"} +All processing happens privately on your device. Your content stays under your control! +:: + +## What Can You Add? + +::tabs + ::div{label="Files" icon="tabler:file"} + ### Documents & Files + Supported formats include: + - Documents: PDF, Word (.docx), RTF + - Text: Markdown, TXT, CSV + - Data: JSON, JSONL + - Books: EPUB + - And many more! + + ::image-plain{src="/how-to/create-knowledge-stack/file-import.webp" alt="File import interface"} + :: + :: + + ::div{label="Obsidian" icon="tabler:notebook"} + ### Obsidian Vaults + Import entire vaults while preserving: + - Folder structure + - Internal links + - Metadata + - Attachments + + ::image-plain{src="/how-to/create-knowledge-stack/obsidian-import.webp" alt="Obsidian import interface"} + :: + :: + + ::div{label="Folders" icon="tabler:folder"} + ### Folder Import + Add entire folders of content: + - Maintains structure + - Processes all supported files + - Bulk import capability + + ::image-plain{src="/how-to/create-knowledge-stack/folder-import.webp" alt="Folder import interface"} + :: + :: + + ::div{label="Notes" icon="tabler:pencil"} + ### Quick Notes + Add custom notes directly: + - Type or paste text + - Supports markdown + - Perfect for quick additions + + ::image-plain{src="/how-to/create-knowledge-stack/custom-notes-import.webp" alt="Custom notes interface"} + :: + :: + + ::div{label="YouTube" icon="tabler:brand-youtube"} + ### YouTube Content + Add video knowledge: + - Paste single or multiple URLs (space-separated) + - Auto-fetches transcripts + - Includes video metadata + + ::image-plain{src="/how-to/create-knowledge-stack/youtube-import.webp" alt="YouTube import interface"} + :: + :: +:: + +## Getting Started + +::steps + ### 1. Create a Stack + Click the Knowledge Stack button in the sidebar + + ::image-plain{src="/how-to/create-knowledge-stack/new-stack-button.webp" alt="New Knowledge Stack button"} + :: + + ### 2. Add Content + - Drag & drop files or folders + - Browse for content + - Paste YouTube links + - Type quick notes + + ### 3. Configure Settings + Click the gear icon to adjust: + - Embedding model (local/remote) + - Chunk settings + - Processing options + + ::alert{type="tip" icon="tabler:bulb"} + Start with default settings - they work great for most content! See [Advanced Features](/features/knowledge-stack/advanced-features) for customization. + :: + + ### 4. Save & Activate + Choose your workflow: + - **Save Draft**: Store for later + - **Compose**: Build and activate + - **Three-dot Menu**: + - "Compose new changes": Update with new content + - "Recompose Stack": Apply setting changes +:: + +## Using Your Stack + +1. **Start a Chat** + - Click the chat icon + - Select your Knowledge Stack(s) + - Multiple stacks can be active at once + +2. **Adjust Search** + - Set similarity (Low to Highest) + - Choose number of chunks (default: 15) + - Enable Jina AI reranking (optional) + +3. **Ask Questions** + - Use natural language + - Get sourced answers + - Reference multiple stacks + +::alert{type="info" icon="tabler:info-circle"} +Want to understand how it works? Check out our guides on: +- [Embeddings](/features/knowledge-stack/embeddings) +- [RAG Technology](/features/knowledge-stack/rag-explained) +- [Advanced Settings](/features/knowledge-stack/advanced-features) +:: + +## Best Practices + +1. **Organize Content** + - Group related materials + - Use clear names + - Keep stacks focused + +2. **Optimize Processing** + - Start with default settings + - Test with sample questions + - Adjust based on results + +3. **Manage Updates** + - Use "Compose new changes" for content updates + - "Recompose Stack" for setting changes + - Review results after major changes + +::alert{type="tip" icon="tabler:bulb"} +New to Knowledge Stacks? Start with a small set of related content to get familiar with the features! +:: \ No newline at end of file diff --git a/content/1.features/knowledge-stack/2.embeddings.md b/content/1.features/knowledge-stack/2.embeddings.md new file mode 100644 index 0000000..60f6e75 --- /dev/null +++ b/content/1.features/knowledge-stack/2.embeddings.md @@ -0,0 +1,83 @@ +--- +title: Understanding Embeddings +description: Learn how Msty makes your documents searchable +navTruncate: false +--- + +Embeddings are what make Knowledge Stacks "smarter" than a regular search. They help Msty understand the meaning behind your words, not just match exact phrases. + +## What Makes it Special? + +Think about how you'd search for a recipe: +- Regular search looks for exact words like "chocolate cake" +- Msty understands that "dessert with cocoa" means the same thing +- It can even find recipes that never use the exact words you typed + +::image-plain{src="/how-to/create-knowledge-stack/search-settings-interface.webp" alt="Search settings showing similarity controls"} +:: + +## How Embeddings Work + +Imagine translating colors into numbers: +- Red = [255, 0, 0] +- Blue = [0, 0, 255] +- Purple = [128, 0, 128] + +Embeddings do the same thing with words and ideas: +- "happy" and "joyful" get similar numbers +- "bank" (money) and "bank" (river) get different numbers +- "I'm freezing" and "it's cold" get similar numbers + +## Embedding Models + +::image-plain{src="/how-to/create-knowledge-stack/chunk-settings-interface.webp" alt="Embedding model selection"} +:: + +### 1. Local Embeddings +- ✅ Completely private - nothing leaves your computer +- ✅ Free to use forever +- ✅ Works offline +- ⚠️ Slightly less accurate +- ⚠️ Uses more CPU power + +### 2. Remote Embeddings (via OpenAI) +- ✅ More accurate understanding +- ✅ Less CPU usage +- ⚠️ Requires API key +- ⚠️ Small cost per use +- ⚠️ Sends text to external service + +::alert{type="tip" icon="tabler:bulb"} +Start with local embeddings - they work great for most uses! Switch to remote only if you need extra accuracy. +:: + +## Similarity Settings + +Control how strictly Msty matches content: + +1. **Similarity Threshold** + - Low: Broader matches, more results + - Medium: Balanced relevance + - High: Stricter matching + - Highest: Only very close matches + +2. **Number of Chunks** + - Default: 15 chunks + - Adjust based on needs: + - More chunks → broader context + - Fewer chunks → focused answers + +::alert{type="info" icon="tabler:info-circle"} +These settings can be adjusted per chat session in the Knowledge Stack selector! +:: + +## Getting Started + +1. Create a new Knowledge Stack +2. Click the gear icon +3. Choose your embedding model +4. Adjust similarity settings as needed + +::alert{type="tip" icon="tabler:bulb"} +Want to fine-tune your results? Check out our guide on [Advanced Knowledge Stack Settings](/features/knowledge-stack/advanced-features). +:: \ No newline at end of file diff --git a/content/1.features/knowledge-stack/3.rag-explained.md b/content/1.features/knowledge-stack/3.rag-explained.md new file mode 100644 index 0000000..de5218c --- /dev/null +++ b/content/1.features/knowledge-stack/3.rag-explained.md @@ -0,0 +1,80 @@ +--- +title: Understanding RAG in Knowledge Stacks +description: Learn how Retrieval Augmented Generation works in Msty +navTruncate: false +--- + +## What is RAG? + +RAG (Retrieval Augmented Generation) is the technology that powers Knowledge Stacks in Msty. It's important to understand that RAG doesn't "train" or "teach" the AI new information - instead, it's more like giving the AI a temporary reference book to consult while answering your questions. + +## How RAG Works + +Think of it like this: + +1. **You**: Ask a question about your documents +2. **Msty**: + - Searches your documents for relevant information + - Uses embeddings to find matches + - Selects the best chunks (default: 15) +3. **AI**: + - Receives your question and selected chunks + - Uses this default prompt: + ``` + The following text has been extracted from a data source due to its probable relevance to the question. + Please use the given information if it is relevant to come up with an answer and don't use anything else. + The answer should be as concise and succinct as possible to answer the question. + ``` + - Generates a focused answer + +::image-plain{src="/how-to/create-knowledge-stack/search-settings-interface.webp" alt="Search settings showing RAG controls"} +:: + +::alert{type="info" icon="tabler:info-circle"} +The AI model itself never learns or remembers your documents. Each time you ask a question, Msty finds the relevant information fresh - like looking up answers in a book each time. +:: + +## Fine-tuning RAG + +Control how RAG works in the chat interface: + +1. **Similarity Threshold**: + - Low: Broader context, more results + - Medium: Balanced matching + - High: Strict matching + - Highest: Only exact matches + +2. **Number of Chunks**: + - Default: 15 chunks + - More chunks = broader context + - Fewer chunks = focused answers + +3. **Custom Prompt**: + - Modify the default prompt + - Guide AI response style + - Maintain answer focus + +::alert{type="tip" icon="tabler:bulb"} +Pro Tip: You can select multiple Knowledge Stacks at once for cross-referencing information! +:: + +## Why Use RAG? + +1. **Accuracy**: + - References specific facts + - Reduces "hallucinations" + - Provides sourced answers + +2. **Privacy**: + - Documents stay local + - Only relevant snippets sent + - Full control over data + +3. **Cost Effective**: + - Sends minimal context + - No training needed + - Works with any AI model + +::alert{type="info" icon="tabler:info-circle"} +Want to optimize your results? Learn about [chunk settings](/features/knowledge-stack/advanced-features) and [embedding options](/features/knowledge-stack/embeddings). +:: \ No newline at end of file diff --git a/content/1.features/knowledge-stack/4.advanced-features.md b/content/1.features/knowledge-stack/4.advanced-features.md new file mode 100644 index 0000000..c1bf50c --- /dev/null +++ b/content/1.features/knowledge-stack/4.advanced-features.md @@ -0,0 +1,120 @@ +--- +title: Advanced Knowledge Stack Settings +description: Fine-tune document processing and search quality +navTruncate: false +--- + +Learn how to optimize your Knowledge Stack's search accuracy and processing with these advanced settings. + +## Processing Settings + +::image-plain{src="/how-to/create-knowledge-stack/chunk-settings-interface.webp" alt="Chunk settings interface"} +:: + +Access these settings by clicking the gear icon at the bottom of your Knowledge Stack: + +### Embedding Model +Choose how Msty understands your content: +- **Local**: Private, offline processing +- **Remote**: Enhanced accuracy via OpenAI +- Click to see all configured models + +### Splitter Type +Choose how to divide your content: +- **Recursive Character**: Like chapters in a book + - Best for normal documents + - Keeps paragraphs together + - Works well with mixed content +- **Sentence**: Like individual sentences + - Perfect for technical docs + - Keeps code blocks intact + - Better for precise answers + +### Chunk Settings +Control how Msty processes your text: + +1. **Chunk Size** + - Low: Quick, specific answers + - Medium: Balanced (recommended) + - High: More context + - Highest: Full context + +2. **Chunk Overlap** + - Low: Minimal connection + - Medium: Good balance + - High: Strong connection + - Highest: Maximum context + +3. **Ignore Small Chunks** + - Default: 50 characters + - Filters out headers, footers + - Skips single-line comments + - Removes short metadata + +::alert{type="tip" icon="tabler:bulb"} +Start with Medium settings - they work great for most documents! Adjust only if you notice issues with answer quality. +:: + +## Search Settings + +::image-plain{src="/how-to/create-knowledge-stack/search-settings-interface.webp" alt="Search settings interface"} +:: + +Access these by clicking the sliders icon in the top right: + +### Results Control +1. **Number of Chunks** + - Default: 15 chunks + - Adjust based on needs + - More chunks = broader context + - Fewer = focused answers + +2. **Similarity Threshold** + - Low to Highest options + - Controls match strictness + - Higher = more precise matches + - Lower = broader results + +### Jina AI Reranking +Enhance search accuracy: +- Free API key from [jina.ai/reranker](https://jina.ai/reranker) +- Improves result ranking +- Better handles complex queries + +::alert{type="warning" icon="tabler:alert-triangle"} +Privacy Note: Jina AI processes search queries and metadata to improve results. Check their [privacy policy](https://jina.ai/legal/#privacy-policy) for details. +:: + +## Managing Changes + +::image-plain{src="/how-to/create-knowledge-stack/compose-save-as-draft-buttons.webp" alt="Compose and save buttons"} +:: + +### Save Options +- **Save Draft**: Store unfinished work +- **Compose**: Activate your stack +- **Three-dot Menu**: + - "Compose new changes": Update content + - "Recompose Stack": Apply setting changes + +## When to Adjust Settings + +1. **Content Issues** + - Long documents → Increase chunk size + - Missing context → Increase overlap + - Too much noise → Raise similarity threshold + - Short answers → Add more chunks + +2. **Performance Tips** + - Test with real questions + - Change one setting at a time + - Note what works best + - Save successful configurations + +::alert{type="info" icon="tabler:info-circle"} +Remember: You can adjust search settings per chat session in the Knowledge Stack selector! +:: + +::alert{type="success" icon="tabler:hand-love-you"} +Thanks to our wonderful Discord user @Tdwag fo contributing almost all the content for this guide! +:: diff --git a/content/2.how-to guides/create-knowledge-stacks.md b/content/2.how-to guides/create-knowledge-stacks.md new file mode 100644 index 0000000..9fdebcc --- /dev/null +++ b/content/2.how-to guides/create-knowledge-stacks.md @@ -0,0 +1,131 @@ +--- +title: Creating a Knowledge Stack +description: Step-by-step guide to creating and populating Knowledge Stacks +navTruncate: false +--- + +::steps + ### 1. Start a New Stack + Click the Knowledge Stack button in the sidebar to begin + + ::image-plain{src="/how-to/create-knowledge-stack/new-stack-button.webp" alt="New Knowledge Stack button in sidebar"} + :: + + ### 2. Add Your Content + ::tabs + ::div{label="Files" icon="tabler:file"} + #### Documents & Files + Drag and drop or browse for: + - PDFs, Word docs, text files + - Code files and spreadsheets + - EPUBs and RTF documents + + ::image-plain{src="/how-to/create-knowledge-stack/file-import.webp" alt="File import interface"} + :: + :: + + ::div{label="Obsidian" icon="tabler:notebook"} + #### Vault Integration + Connect entire Obsidian vaults while preserving: + - Folder structure + - Internal links + - Metadata + + ::image-plain{src="/how-to/create-knowledge-stack/obsidian-import.webp" alt="Obsidian vault import"} + :: + :: + + ::div{label="Folders" icon="tabler:folder"} + #### Bulk Import + Add entire folders with mixed content: + - Drag & drop folders + - Maintains original structure + - Processes all supported file types + + ::image-plain{src="/how-to/create-knowledge-stack/folder-import.webp" alt="Folder import interface"} + :: + :: + + ::div{label="Notes" icon="tabler:pencil"} + #### Quick Add Notes + Type or paste custom text directly: + - Perfect for last-minute additions + - Supports markdown formatting + - Appears as "Custom Notes" in stack + + ::image-plain{src="/how-to/create-knowledge-stack/custom-notes-import.webp" alt="Custom notes field"} + :: + :: + + ::div{label="YouTube" icon="tabler:brand-youtube"} + #### Video Content + Paste YouTube URLs to add: + - Automatic transcript processing + - Video metadata inclusion + - Multiple URLs supported + + ::image-plain{src="/how-to/create-knowledge-stack/youtube-import.webp" alt="YouTube URL field"} + :: + :: + :: + + ### 3. Configure Processing + Click the gear icon to adjust chunk settings: + + ::image-plain{src="/how-to/create-knowledge-stack/chunk-settings-interface.webp" alt="Chunk processing settings"} + :: + + - **Embedding Model**: Choose local or cloud-based + - **Splitter Type**: Recursive vs sentence-based + - **Chunk Size**: Balance context vs specificity + - **Overlap**: Control context connections + + ::alert{type="tip" icon="tabler:bulb"} + Need help choosing settings? See our [Advanced Configuration Guide](/features/knowledge-stack/advanced-features) + :: + + ### 4. Fine-tune Search + Click the sliders icon for precision controls: + + ::image-plain{src="/how-to/create-knowledge-stack/search-settings-interface.webp" alt="Search settings interface"} + :: + + - Results quantity and quality thresholds + - Custom prompt prefixes + - Jina AI reranking integration + + ### 5. Save & Activate + Choose your workflow: + + ::image-plain{src="/how-to/create-knowledge-stack/compose-save-as-draft-buttons.webp" alt="Save and compose buttons"} + :: + + - **Save Draft**: Store unfinished stack + - **Compose**: Build ready-to-use stack + - **Three-dot Menu**: Update existing stacks + +::alert{type="info" icon="tabler:info-circle"} +Remember: You can always [recompose your stack](/features/knowledge-stack/advanced-features#recomposing-stacks) later if you add new content or change settings! +:: + + ### 6. Chat with Your Stack + Access your knowledge in any chat: + + ::image-plain{src="/how-to/create-knowledge-stack/choose-knowledge-stack-in-chat-interface.webp" alt="Knowledge Stack selection in chat interface"} + :: + + 1. Start a new chat + 2. Click the Knowledge Stack icon + 3. Select stacks to reference + 4. Adjust settings: + - **Similarity**: Match strictness (Low=Broad, High=Exact) + - **Chunks**: Number of references to use + 5. Ask natural language questions + + ::alert{type="tip" icon="tabler:bulb"} + Pro Tip: Combine multiple stacks for cross-reference queries! The selected stacks will show a checkmark badge when active. + :: + + ::alert{type="info" icon="tabler:info-circle"} + Want deeper control? Learn about [optimizing search results](/features/knowledge-stack/embeddings) and [how RAG works](/features/knowledge-stack/rag-explained). + :: \ No newline at end of file diff --git a/public/how-to/create-knowledge-stack/choose-knowledge-stack-in-chat-interface.webp b/public/how-to/create-knowledge-stack/choose-knowledge-stack-in-chat-interface.webp new file mode 100644 index 0000000..ded61ef Binary files /dev/null and b/public/how-to/create-knowledge-stack/choose-knowledge-stack-in-chat-interface.webp differ diff --git a/public/how-to/create-knowledge-stack/chunk-settings-interface.webp b/public/how-to/create-knowledge-stack/chunk-settings-interface.webp new file mode 100644 index 0000000..ba9fd50 Binary files /dev/null and b/public/how-to/create-knowledge-stack/chunk-settings-interface.webp differ diff --git a/public/how-to/create-knowledge-stack/compose-save-as-draft-buttons.webp b/public/how-to/create-knowledge-stack/compose-save-as-draft-buttons.webp new file mode 100644 index 0000000..08612b2 Binary files /dev/null and b/public/how-to/create-knowledge-stack/compose-save-as-draft-buttons.webp differ diff --git a/public/how-to/create-knowledge-stack/custom-notes-import.webp b/public/how-to/create-knowledge-stack/custom-notes-import.webp new file mode 100644 index 0000000..bc44f12 Binary files /dev/null and b/public/how-to/create-knowledge-stack/custom-notes-import.webp differ diff --git a/public/how-to/create-knowledge-stack/file-import.webp b/public/how-to/create-knowledge-stack/file-import.webp new file mode 100644 index 0000000..e89b369 Binary files /dev/null and b/public/how-to/create-knowledge-stack/file-import.webp differ diff --git a/public/how-to/create-knowledge-stack/folder-import.webp b/public/how-to/create-knowledge-stack/folder-import.webp new file mode 100644 index 0000000..aebbe47 Binary files /dev/null and b/public/how-to/create-knowledge-stack/folder-import.webp differ diff --git a/public/how-to/create-knowledge-stack/new-stack-button.webp b/public/how-to/create-knowledge-stack/new-stack-button.webp new file mode 100644 index 0000000..51f6e30 Binary files /dev/null and b/public/how-to/create-knowledge-stack/new-stack-button.webp differ diff --git a/public/how-to/create-knowledge-stack/obsidian-import.webp b/public/how-to/create-knowledge-stack/obsidian-import.webp new file mode 100644 index 0000000..a5b94f9 Binary files /dev/null and b/public/how-to/create-knowledge-stack/obsidian-import.webp differ diff --git a/public/how-to/create-knowledge-stack/search-settings-interface.webp b/public/how-to/create-knowledge-stack/search-settings-interface.webp new file mode 100644 index 0000000..2395290 Binary files /dev/null and b/public/how-to/create-knowledge-stack/search-settings-interface.webp differ diff --git a/public/how-to/create-knowledge-stack/youtube-import.webp b/public/how-to/create-knowledge-stack/youtube-import.webp new file mode 100644 index 0000000..a7f0eb8 Binary files /dev/null and b/public/how-to/create-knowledge-stack/youtube-import.webp differ