Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-22 01:55:24 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #102] [Feature Request] mbox (Google Takeout for Gmail) Support - Initial code provided #69

New Issue
Closed
opened 2026-02-22 18:17:45 -05:00 by yindo · 2 comments
yindo commented 2026-02-22 18:17:45 -05:00
Owner
Copy Link
Copy Source

Originally created by @mplawner on GitHub (Jun 22, 2023).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/102

Hi folks - I'm not sure how to contribute to this project, but I started working on adding mbox support. Google Takeout using the mbox format, so if figured this would be a great way to interrogate emails.

Still working on it, but I've updated the filetypes.py file and created an as_mbox.py file. If someone can help importing it into this project, that would be appreciated. Thanks!

Updated filetypes.py:

from .convert.as_text import as_text
from .convert.as_markdown import as_markdown
from .convert.as_pdf import as_pdf
from .convert.as_docx import as_docx, as_odt
from .convert.as_mbox import as_mbox  # make sure to create this function

FILETYPES = {
    '.txt': as_text,
    '.md': as_markdown,
    '.pdf': as_pdf,
    '.docx': as_docx,
    '.odt': as_odt,
    '.mbox': as_mbox,  # add mbox file handling
}

ACCEPTED_MIMES = {
    'text/plain': ['.txt', '.md'],
    'application/vnd.openxmlformats-officedocument.wordprocessingml.document': ['.docx'],
    'application/vnd.oasis.opendocument.text': ['.odt'],
    'application/pdf': ['.pdf'],
    'application/mbox': ['.mbox'],  # add mbox MIME type

New as_mbox.py:

import os
import datetime  # Add this line
import email.utils
from mailbox import mbox
from slugify import slugify
from ..utils import guid, file_creation_time, write_to_server_documents, move_source
from ...utils import tokenize
from bs4 import BeautifulSoup

# Process all mbox-related documents.
def as_mbox(**kwargs):
    parent_dir = kwargs.get('directory', 'hotdir')
    filename = kwargs.get('filename')
    ext = kwargs.get('ext', '.mbox')
    remove = kwargs.get('remove_on_complete', False)
    fullpath = f"{parent_dir}/{filename}{ext}"

    print(f"-- Working {fullpath} --")
    box = mbox(fullpath)

    for message in box:
        content = ""
        if message.is_multipart():
            for part in message.get_payload():
                if part.get_content_type() == 'text/plain':
                    content = part.get_payload()
                elif part.get_content_type() == 'text/html':
                    soup = BeautifulSoup(part.get_payload(), 'html.parser')
                    content = soup.get_text()
        else:
            content = message.get_payload()

        date_tuple = email.utils.parsedate_tz(message['Date'])
        if date_tuple:
            local_date = datetime.datetime.fromtimestamp(email.utils.mktime_tz(date_tuple))
            date_sent = local_date.strftime("%a, %d %b %Y %H:%M:%S")
        else:
            date_sent = None
            
        data = {
            'id': guid(), 
            'url': "file://"+os.path.abspath(f"{parent_dir}/processed/{slugify(filename)}-{guid()}{ext}"),
            'title': f"{filename}{ext}",
            'description': "a custom file uploaded by the user.",
            'published': file_creation_time(fullpath),
            'sender': message['From'],
            'recipient': message['To'],
            'subject': message['Subject'],
            'date_sent': date_sent,
            'wordCount': len(content),
            'pageContent': content,
            'token_count_estimate': len(tokenize(content))
        }

        write_to_server_documents(data, f"{slugify(filename)}-{data.get('id')}")
    move_source(parent_dir, f"{filename}{ext}", remove=remove)
    print(f"[SUCCESS]: {filename}{ext} converted & ready for embedding.\n")
Originally created by @mplawner on GitHub (Jun 22, 2023). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/102 Hi folks - I'm not sure how to contribute to this project, but I started working on adding mbox support. Google Takeout using the mbox format, so if figured this would be a great way to interrogate emails. Still working on it, but I've updated the filetypes.py file and created an as_mbox.py file. If someone can help importing it into this project, that would be appreciated. Thanks! Updated filetypes.py: ``` from .convert.as_text import as_text from .convert.as_markdown import as_markdown from .convert.as_pdf import as_pdf from .convert.as_docx import as_docx, as_odt from .convert.as_mbox import as_mbox # make sure to create this function FILETYPES = { '.txt': as_text, '.md': as_markdown, '.pdf': as_pdf, '.docx': as_docx, '.odt': as_odt, '.mbox': as_mbox, # add mbox file handling } ACCEPTED_MIMES = { 'text/plain': ['.txt', '.md'], 'application/vnd.openxmlformats-officedocument.wordprocessingml.document': ['.docx'], 'application/vnd.oasis.opendocument.text': ['.odt'], 'application/pdf': ['.pdf'], 'application/mbox': ['.mbox'], # add mbox MIME type ``` New as_mbox.py: ``` import os import datetime # Add this line import email.utils from mailbox import mbox from slugify import slugify from ..utils import guid, file_creation_time, write_to_server_documents, move_source from ...utils import tokenize from bs4 import BeautifulSoup # Process all mbox-related documents. def as_mbox(**kwargs): parent_dir = kwargs.get('directory', 'hotdir') filename = kwargs.get('filename') ext = kwargs.get('ext', '.mbox') remove = kwargs.get('remove_on_complete', False) fullpath = f"{parent_dir}/{filename}{ext}" print(f"-- Working {fullpath} --") box = mbox(fullpath) for message in box: content = "" if message.is_multipart(): for part in message.get_payload(): if part.get_content_type() == 'text/plain': content = part.get_payload() elif part.get_content_type() == 'text/html': soup = BeautifulSoup(part.get_payload(), 'html.parser') content = soup.get_text() else: content = message.get_payload() date_tuple = email.utils.parsedate_tz(message['Date']) if date_tuple: local_date = datetime.datetime.fromtimestamp(email.utils.mktime_tz(date_tuple)) date_sent = local_date.strftime("%a, %d %b %Y %H:%M:%S") else: date_sent = None data = { 'id': guid(), 'url': "file://"+os.path.abspath(f"{parent_dir}/processed/{slugify(filename)}-{guid()}{ext}"), 'title': f"{filename}{ext}", 'description': "a custom file uploaded by the user.", 'published': file_creation_time(fullpath), 'sender': message['From'], 'recipient': message['To'], 'subject': message['Subject'], 'date_sent': date_sent, 'wordCount': len(content), 'pageContent': content, 'token_count_estimate': len(tokenize(content)) } write_to_server_documents(data, f"{slugify(filename)}-{data.get('id')}") move_source(parent_dir, f"{filename}{ext}", remove=remove) print(f"[SUCCESS]: {filename}{ext} converted & ready for embedding.\n") ```
yindo closed this issue 2026-02-22 18:17:45 -05:00
yindo commented 2026-02-22 18:17:45 -05:00
Author
Owner
Copy Link
Copy Source

@AntonioCiolino commented on GitHub (Jun 23, 2023):

pull request?

@AntonioCiolino commented on GitHub (Jun 23, 2023): pull request?
yindo commented 2026-02-22 18:17:45 -05:00
Author
Owner
Copy Link
Copy Source

@mplawner commented on GitHub (Jun 23, 2023):

Ok. I never did that before. Hope I did it correctly.

@mplawner commented on GitHub (Jun 23, 2023): Ok. I never did that before. Hope I did it correctly.
yindo changed title from [Feature Request] mbox (Google Takeout for Gmail) Support - Initial code provided to [GH-ISSUE #102] [Feature Request] mbox (Google Takeout for Gmail) Support - Initial code provided 2026-06-05 14:33:18 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#69
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?