docs: Knowledge Stack

content/1.features/knowledge-stack/1.basics.md

@@ -0,0 +1,152 @@
+---
+title: Knowledge Stack Basics
+description: Get started with Knowledge Stacks in Msty
+navTruncate: false
+---
+
+Knowledge Stacks let you chat with your content - from simple documents to entire knowledge bases. Think of it as having a smart assistant that has read and understood all your materials.
+
+::alert{type="info" icon="tabler:info-circle"}
+All processing happens privately on your device. Your content stays under your control!
+::
+
+## What Can You Add?
+
+::tabs
+  ::div{label="Files" icon="tabler:file"}
+    ### Documents & Files
+    Supported formats include:
+    - Documents: PDF, Word (.docx), RTF
+    - Text: Markdown, TXT, CSV
+    - Data: JSON, JSONL
+    - Books: EPUB
+    - And many more!
+
+    ::image-plain{src="/how-to/create-knowledge-stack/file-import.webp" alt="File import interface"}
+    ::
+  ::
+
+  ::div{label="Obsidian" icon="tabler:notebook"}
+    ### Obsidian Vaults
+    Import entire vaults while preserving:
+    - Folder structure
+    - Internal links
+    - Metadata
+    - Attachments
+
+    ::image-plain{src="/how-to/create-knowledge-stack/obsidian-import.webp" alt="Obsidian import interface"}
+    ::
+  ::
+
+  ::div{label="Folders" icon="tabler:folder"}
+    ### Folder Import
+    Add entire folders of content:
+    - Maintains structure
+    - Processes all supported files
+    - Bulk import capability
+
+    ::image-plain{src="/how-to/create-knowledge-stack/folder-import.webp" alt="Folder import interface"}
+    ::
+  ::
+
+  ::div{label="Notes" icon="tabler:pencil"}
+    ### Quick Notes
+    Add custom notes directly:
+    - Type or paste text
+    - Supports markdown
+    - Perfect for quick additions
+    
+    ::image-plain{src="/how-to/create-knowledge-stack/custom-notes-import.webp" alt="Custom notes interface"}
+    ::
+  ::
+
+  ::div{label="YouTube" icon="tabler:brand-youtube"}
+    ### YouTube Content
+    Add video knowledge:
+    - Paste single or multiple URLs (space-separated)
+    - Auto-fetches transcripts
+    - Includes video metadata
+    
+    ::image-plain{src="/how-to/create-knowledge-stack/youtube-import.webp" alt="YouTube import interface"}
+    ::
+  ::
+::
+
+## Getting Started
+
+::steps
+  ### 1. Create a Stack
+  Click the Knowledge Stack button in the sidebar
+  
+  ::image-plain{src="/how-to/create-knowledge-stack/new-stack-button.webp" alt="New Knowledge Stack button"}
+  ::
+
+  ### 2. Add Content
+  - Drag & drop files or folders
+  - Browse for content
+  - Paste YouTube links
+  - Type quick notes
+
+  ### 3. Configure Settings
+  Click the gear icon to adjust:
+  - Embedding model (local/remote)
+  - Chunk settings
+  - Processing options
+
+  ::alert{type="tip" icon="tabler:bulb"}
+  Start with default settings - they work great for most content! See [Advanced Features](/features/knowledge-stack/advanced-features) for customization.
+  ::
+
+  ### 4. Save & Activate
+  Choose your workflow:
+  - **Save Draft**: Store for later
+  - **Compose**: Build and activate
+  - **Three-dot Menu**: 
+    - "Compose new changes": Update with new content
+    - "Recompose Stack": Apply setting changes
+::
+
+## Using Your Stack
+
+1. **Start a Chat**
+   - Click the chat icon
+   - Select your Knowledge Stack(s)
+   - Multiple stacks can be active at once
+
+2. **Adjust Search**
+   - Set similarity (Low to Highest)
+   - Choose number of chunks (default: 15)
+   - Enable Jina AI reranking (optional)
+
+3. **Ask Questions**
+   - Use natural language
+   - Get sourced answers
+   - Reference multiple stacks
+
+::alert{type="info" icon="tabler:info-circle"}
+Want to understand how it works? Check out our guides on:
+- [Embeddings](/features/knowledge-stack/embeddings)
+- [RAG Technology](/features/knowledge-stack/rag-explained)
+- [Advanced Settings](/features/knowledge-stack/advanced-features)
+::
+
+## Best Practices
+
+1. **Organize Content**
+   - Group related materials
+   - Use clear names
+   - Keep stacks focused
+
+2. **Optimize Processing**
+   - Start with default settings
+   - Test with sample questions
+   - Adjust based on results
+
+3. **Manage Updates**
+   - Use "Compose new changes" for content updates
+   - "Recompose Stack" for setting changes
+   - Review results after major changes
+
+::alert{type="tip" icon="tabler:bulb"}
+New to Knowledge Stacks? Start with a small set of related content to get familiar with the features!
+::

content/1.features/knowledge-stack/2.embeddings.md

@@ -0,0 +1,83 @@
+---
+title: Understanding Embeddings
+description: Learn how Msty makes your documents searchable
+navTruncate: false
+---
+
+Embeddings are what make Knowledge Stacks "smarter" than a regular search. They help Msty understand the meaning behind your words, not just match exact phrases.
+
+## What Makes it Special?
+
+Think about how you'd search for a recipe:
+- Regular search looks for exact words like "chocolate cake"
+- Msty understands that "dessert with cocoa" means the same thing
+- It can even find recipes that never use the exact words you typed
+
+::image-plain{src="/how-to/create-knowledge-stack/search-settings-interface.webp" alt="Search settings showing similarity controls"}
+::
+
+## How Embeddings Work
+
+Imagine translating colors into numbers:
+- Red = [255, 0, 0]
+- Blue = [0, 0, 255]
+- Purple = [128, 0, 128]
+
+Embeddings do the same thing with words and ideas:
+- "happy" and "joyful" get similar numbers
+- "bank" (money) and "bank" (river) get different numbers
+- "I'm freezing" and "it's cold" get similar numbers
+
+## Embedding Models
+
+::image-plain{src="/how-to/create-knowledge-stack/chunk-settings-interface.webp" alt="Embedding model selection"}
+::
+
+### 1. Local Embeddings
+- ✅ Completely private - nothing leaves your computer
+- ✅ Free to use forever
+- ✅ Works offline
+- ⚠️ Slightly less accurate
+- ⚠️ Uses more CPU power
+
+### 2. Remote Embeddings (via OpenAI)
+- ✅ More accurate understanding
+- ✅ Less CPU usage
+- ⚠️ Requires API key
+- ⚠️ Small cost per use
+- ⚠️ Sends text to external service
+
+::alert{type="tip" icon="tabler:bulb"}
+Start with local embeddings - they work great for most uses! Switch to remote only if you need extra accuracy.
+::
+
+## Similarity Settings
+
+Control how strictly Msty matches content:
+
+1. **Similarity Threshold**
+   - Low: Broader matches, more results
+   - Medium: Balanced relevance
+   - High: Stricter matching
+   - Highest: Only very close matches
+
+2. **Number of Chunks**
+   - Default: 15 chunks
+   - Adjust based on needs:
+     - More chunks → broader context
+     - Fewer chunks → focused answers
+
+::alert{type="info" icon="tabler:info-circle"}
+These settings can be adjusted per chat session in the Knowledge Stack selector!
+::
+
+## Getting Started
+
+1. Create a new Knowledge Stack
+2. Click the gear icon
+3. Choose your embedding model
+4. Adjust similarity settings as needed
+
+::alert{type="tip" icon="tabler:bulb"}
+Want to fine-tune your results? Check out our guide on [Advanced Knowledge Stack Settings](/features/knowledge-stack/advanced-features).
+::

content/1.features/knowledge-stack/3.rag-explained.md

@@ -0,0 +1,80 @@
+---
+title: Understanding RAG in Knowledge Stacks
+description: Learn how Retrieval Augmented Generation works in Msty
+navTruncate: false
+---
+
+## What is RAG?
+
+RAG (Retrieval Augmented Generation) is the technology that powers Knowledge Stacks in Msty. It's important to understand that RAG doesn't "train" or "teach" the AI new information - instead, it's more like giving the AI a temporary reference book to consult while answering your questions.
+
+## How RAG Works
+
+Think of it like this:
+
+1. **You**: Ask a question about your documents
+2. **Msty**: 
+   - Searches your documents for relevant information
+   - Uses embeddings to find matches
+   - Selects the best chunks (default: 15)
+3. **AI**: 
+   - Receives your question and selected chunks
+   - Uses this default prompt:
+     ```
+     The following text has been extracted from a data source due to its probable relevance to the question. 
+     Please use the given information if it is relevant to come up with an answer and don't use anything else. 
+     The answer should be as concise and succinct as possible to answer the question.
+     ```
+   - Generates a focused answer
+
+::image-plain{src="/how-to/create-knowledge-stack/search-settings-interface.webp" alt="Search settings showing RAG controls"}
+::
+
+::alert{type="info" icon="tabler:info-circle"}
+The AI model itself never learns or remembers your documents. Each time you ask a question, Msty finds the relevant information fresh - like looking up answers in a book each time.
+::
+
+## Fine-tuning RAG
+
+Control how RAG works in the chat interface:
+
+1. **Similarity Threshold**:
+   - Low: Broader context, more results
+   - Medium: Balanced matching
+   - High: Strict matching
+   - Highest: Only exact matches
+
+2. **Number of Chunks**:
+   - Default: 15 chunks
+   - More chunks = broader context
+   - Fewer chunks = focused answers
+
+3. **Custom Prompt**:
+   - Modify the default prompt
+   - Guide AI response style
+   - Maintain answer focus
+
+::alert{type="tip" icon="tabler:bulb"}
+Pro Tip: You can select multiple Knowledge Stacks at once for cross-referencing information!
+::
+
+## Why Use RAG?
+
+1. **Accuracy**: 
+   - References specific facts
+   - Reduces "hallucinations"
+   - Provides sourced answers
+
+2. **Privacy**: 
+   - Documents stay local
+   - Only relevant snippets sent
+   - Full control over data
+
+3. **Cost Effective**:
+   - Sends minimal context
+   - No training needed
+   - Works with any AI model
+
+::alert{type="info" icon="tabler:info-circle"}
+Want to optimize your results? Learn about [chunk settings](/features/knowledge-stack/advanced-features) and [embedding options](/features/knowledge-stack/embeddings).
+::

content/1.features/knowledge-stack/4.advanced-features.md

@@ -0,0 +1,120 @@
+---
+title: Advanced Knowledge Stack Settings
+description: Fine-tune document processing and search quality
+navTruncate: false
+---
+
+Learn how to optimize your Knowledge Stack's search accuracy and processing with these advanced settings.
+
+## Processing Settings
+
+::image-plain{src="/how-to/create-knowledge-stack/chunk-settings-interface.webp" alt="Chunk settings interface"}
+::
+
+Access these settings by clicking the gear icon at the bottom of your Knowledge Stack:
+
+### Embedding Model
+Choose how Msty understands your content:
+- **Local**: Private, offline processing
+- **Remote**: Enhanced accuracy via OpenAI
+- Click to see all configured models
+
+### Splitter Type
+Choose how to divide your content:
+- **Recursive Character**: Like chapters in a book
+  - Best for normal documents
+  - Keeps paragraphs together
+  - Works well with mixed content
+- **Sentence**: Like individual sentences
+  - Perfect for technical docs
+  - Keeps code blocks intact
+  - Better for precise answers
+
+### Chunk Settings
+Control how Msty processes your text:
+
+1. **Chunk Size**
+   - Low: Quick, specific answers
+   - Medium: Balanced (recommended)
+   - High: More context
+   - Highest: Full context
+
+2. **Chunk Overlap**
+   - Low: Minimal connection
+   - Medium: Good balance
+   - High: Strong connection
+   - Highest: Maximum context
+
+3. **Ignore Small Chunks**
+   - Default: 50 characters
+   - Filters out headers, footers
+   - Skips single-line comments
+   - Removes short metadata
+
+::alert{type="tip" icon="tabler:bulb"}
+Start with Medium settings - they work great for most documents! Adjust only if you notice issues with answer quality.
+::
+
+## Search Settings
+
+::image-plain{src="/how-to/create-knowledge-stack/search-settings-interface.webp" alt="Search settings interface"}
+::
+
+Access these by clicking the sliders icon in the top right:
+
+### Results Control
+1. **Number of Chunks**
+   - Default: 15 chunks
+   - Adjust based on needs
+   - More chunks = broader context
+   - Fewer = focused answers
+
+2. **Similarity Threshold**
+   - Low to Highest options
+   - Controls match strictness
+   - Higher = more precise matches
+   - Lower = broader results
+
+### Jina AI Reranking
+Enhance search accuracy:
+- Free API key from [jina.ai/reranker](https://jina.ai/reranker)
+- Improves result ranking
+- Better handles complex queries
+
+::alert{type="warning" icon="tabler:alert-triangle"}
+Privacy Note: Jina AI processes search queries and metadata to improve results. Check their [privacy policy](https://jina.ai/legal/#privacy-policy) for details.
+::
+
+## Managing Changes
+
+::image-plain{src="/how-to/create-knowledge-stack/compose-save-as-draft-buttons.webp" alt="Compose and save buttons"}
+::
+
+### Save Options
+- **Save Draft**: Store unfinished work
+- **Compose**: Activate your stack
+- **Three-dot Menu**:
+  - "Compose new changes": Update content
+  - "Recompose Stack": Apply setting changes
+
+## When to Adjust Settings
+
+1. **Content Issues**
+   - Long documents → Increase chunk size
+   - Missing context → Increase overlap
+   - Too much noise → Raise similarity threshold
+   - Short answers → Add more chunks
+
+2. **Performance Tips**
+   - Test with real questions
+   - Change one setting at a time
+   - Note what works best
+   - Save successful configurations
+
+::alert{type="info" icon="tabler:info-circle"}
+Remember: You can adjust search settings per chat session in the Knowledge Stack selector!
+::
+
+::alert{type="success" icon="tabler:hand-love-you"}
+Thanks to our wonderful Discord user @Tdwag fo contributing almost all the content for this guide!
+::

content/2.how-to guides/create-knowledge-stacks.md

@@ -0,0 +1,131 @@
+---
+title: Creating a Knowledge Stack
+description: Step-by-step guide to creating and populating Knowledge Stacks
+navTruncate: false
+---
+
+::steps
+  ### 1. Start a New Stack
+  Click the Knowledge Stack button in the sidebar to begin
+  
+  ::image-plain{src="/how-to/create-knowledge-stack/new-stack-button.webp" alt="New Knowledge Stack button in sidebar"}
+  ::
+
+  ### 2. Add Your Content
+  ::tabs
+    ::div{label="Files" icon="tabler:file"}
+      #### Documents & Files
+      Drag and drop or browse for:
+      - PDFs, Word docs, text files
+      - Code files and spreadsheets
+      - EPUBs and RTF documents
+      
+      ::image-plain{src="/how-to/create-knowledge-stack/file-import.webp" alt="File import interface"}
+      ::
+    ::
+
+    ::div{label="Obsidian" icon="tabler:notebook"}
+      #### Vault Integration
+      Connect entire Obsidian vaults while preserving:
+      - Folder structure
+      - Internal links
+      - Metadata
+      
+      ::image-plain{src="/how-to/create-knowledge-stack/obsidian-import.webp" alt="Obsidian vault import"}
+      ::
+    ::
+
+    ::div{label="Folders" icon="tabler:folder"}
+      #### Bulk Import
+      Add entire folders with mixed content:
+      - Drag & drop folders
+      - Maintains original structure
+      - Processes all supported file types
+      
+      ::image-plain{src="/how-to/create-knowledge-stack/folder-import.webp" alt="Folder import interface"}
+      ::
+    ::
+
+    ::div{label="Notes" icon="tabler:pencil"}
+      #### Quick Add Notes
+      Type or paste custom text directly:
+      - Perfect for last-minute additions
+      - Supports markdown formatting
+      - Appears as "Custom Notes" in stack
+      
+      ::image-plain{src="/how-to/create-knowledge-stack/custom-notes-import.webp" alt="Custom notes field"}
+      ::
+    ::
+
+    ::div{label="YouTube" icon="tabler:brand-youtube"}
+      #### Video Content
+      Paste YouTube URLs to add:
+      - Automatic transcript processing
+      - Video metadata inclusion
+      - Multiple URLs supported
+      
+      ::image-plain{src="/how-to/create-knowledge-stack/youtube-import.webp" alt="YouTube URL field"}
+      ::
+    ::
+  ::
+
+  ### 3. Configure Processing
+  Click the gear icon to adjust chunk settings:
+  
+  ::image-plain{src="/how-to/create-knowledge-stack/chunk-settings-interface.webp" alt="Chunk processing settings"}
+  ::
+  
+  - **Embedding Model**: Choose local or cloud-based
+  - **Splitter Type**: Recursive vs sentence-based
+  - **Chunk Size**: Balance context vs specificity
+  - **Overlap**: Control context connections
+  
+  ::alert{type="tip" icon="tabler:bulb"}
+  Need help choosing settings? See our [Advanced Configuration Guide](/features/knowledge-stack/advanced-features)
+  ::
+
+  ### 4. Fine-tune Search
+  Click the sliders icon for precision controls:
+  
+  ::image-plain{src="/how-to/create-knowledge-stack/search-settings-interface.webp" alt="Search settings interface"}
+  ::
+  
+  - Results quantity and quality thresholds
+  - Custom prompt prefixes
+  - Jina AI reranking integration
+
+  ### 5. Save & Activate
+  Choose your workflow:
+  
+  ::image-plain{src="/how-to/create-knowledge-stack/compose-save-as-draft-buttons.webp" alt="Save and compose buttons"}
+  ::
+  
+  - **Save Draft**: Store unfinished stack
+  - **Compose**: Build ready-to-use stack
+  - **Three-dot Menu**: Update existing stacks
+
+::alert{type="info" icon="tabler:info-circle"}
+Remember: You can always [recompose your stack](/features/knowledge-stack/advanced-features#recomposing-stacks) later if you add new content or change settings!
+::
+
+  ### 6. Chat with Your Stack
+  Access your knowledge in any chat:
+
+  ::image-plain{src="/how-to/create-knowledge-stack/choose-knowledge-stack-in-chat-interface.webp" alt="Knowledge Stack selection in chat interface"}
+  ::
+
+  1. Start a new chat
+  2. Click the Knowledge Stack icon
+  3. Select stacks to reference
+  4. Adjust settings:
+     - **Similarity**: Match strictness (Low=Broad, High=Exact)
+     - **Chunks**: Number of references to use
+  5. Ask natural language questions
+
+  ::alert{type="tip" icon="tabler:bulb"}
+  Pro Tip: Combine multiple stacks for cross-reference queries! The selected stacks will show a checkmark badge when active.
+  ::
+
+  ::alert{type="info" icon="tabler:info-circle"}
+  Want deeper control? Learn about [optimizing search results](/features/knowledge-stack/embeddings) and [how RAG works](/features/knowledge-stack/rag-explained).
+  ::