* feat: add hyperlink extraction support to PPTXToDocument and XLSXToDocument
Add a `link_format` parameter to both converters, supporting "markdown",
"plain", and "none" (default) formats. This follows the same pattern
already available in DOCXToDocument.
Closes#9110
* fix: address review comments for PPTXToDocument
- Swap parameter order to keep store_full_path before link_format,
avoiding a breaking change for positional argument users
- Add _Paragraph type annotation to _process_paragraph method
* fix: remove unused pylint disable comment and run formatter
* fix: use single backticks in docstrings per project convention
* fix: move link_format to keyword-only parameter in XLSXToDocument
---------
Co-authored-by: yaowubarbara <[email protected]>
* extending original LLMDocumentContentExtractor to extract metadata
* updating class docstring
* simplifying code
* fixing
* updating tests
* adding release notes
* fixing release notes; adding new live test for metadata only
* refactoring code and adding more tests
* updating release notes
* refactoring code: only one prompt
* Update haystack/components/extractors/image/llm_document_content_extractor.py
Co-authored-by: Sebastian Husch Lee <[email protected]>
* Update haystack/components/extractors/image/llm_document_content_extractor.py
Co-authored-by: Sebastian Husch Lee <[email protected]>
* attending PR comments
* updating docstring prompt to include metadata extraction
* fxing tests
* Apply suggestion from @sjrl
Co-authored-by: Sebastian Husch Lee <[email protected]>
* fixing prompt validation
* fixing update
* attending PR comments
* updating release notes and docstring example
---------
Co-authored-by: Sebastian Husch Lee <[email protected]>
* Starting to refactor type util tests to be more systematic
* refactoring
* Expand tests
* Update to type utils
* Add missing subclass check
* Expand and refactor tests, introduce type_validation Literal
* More test refactoring
* Test refactoring, adding type validation variable to pipeline base
* Update relaxed version of type checking to pass all newly added tests
* trim whitespace
* Add tests
* cleanup
* Updates docstrings
* Add reno
* docs
* Fix mypy and add docstrings
* Changes based on advice from Tobi
* Remove unused imports
* Doc strings
* Add connection type validation to to_dict and from_dict
* Update tests
* Fix test
* Also save connection_type_validation at global pipeline level
* Fix tests
* Remove connection type validation from the connect level, only keep at pipeline level
* Formatting
* Fix tests
* formatting
* add component checks
* pipeline should run deterministically
* add FIFOQueue
* add agent tests
* add order dependent tests
* run new tests
* remove code that is not needed
* test: intermediate from cycle outputs are available outside cycle
* add tests for component checks (Claude)
* adapt tests for component checks (o1 review)
* chore: format
* remove tests that aren't needed anymore
* add _calculate_priority tests
* revert accidental change in pyproject.toml
* test format conversion
* adapt to naming convention
* chore: proper docstrings and type hints for PQ
* format
* add more unit tests
* rm unneeded comments
* test input consumption
* lint
* fix: docstrings
* lint
* format
* format
* fix license header
* fix license header
* add component run tests
* fix: pass correct input format to tracing
* fix types
* format
* format
* types
* add defaults from Socket instead of signature
- otherwise components with dynamic inputs would fail
* fix test names
* still wait for optional inputs on greedy variadic sockets
- mirrors previous behavior
* fix format
* wip: warn for ambiguous running order
* wip: alternative warning
* fix license header
* make code more readable
Co-authored-by: Amna Mubashar <[email protected]>
* Introduce content tracing to a behavioral test
* Fixing linting
* Remove debug print statements
* Fix tracer tests
* remove print
* test: test for component inputs
* test: remove testing for run order
* chore: update component checks from experimental
* chore: update pipeline and base from experimental
* refactor: remove unused method
* refactor: remove unused method
* refactor: outdated comment
* refactor: inputs state is updated as side effect
- to prepare for AsyncPipeline implementation
* format
* test: add file conversion test
* format
* fix: original implementation deepcopies outputs
* lint
* fix: from_dict was updated
* fix: format
* fix: test
* test: add test for thread safety
* remove unused imports
* format
* test: FIFOPriorityQueue
* chore: add release note
* fix: resolve merge conflict with mermaid changes
* fix: format
* fix: remove unused import
* refactor: rename to avoid accidental conflicts
* chore: remove unused inputs, add missing license header
* chore: extend release notes
* Update releasenotes/notes/fix-pipeline-run-2fefeafc705a6d91.yaml
Co-authored-by: Amna Mubashar <[email protected]>
* fix: format
* fix: format
* Update release note
---------
Co-authored-by: Amna Mubashar <[email protected]>
Co-authored-by: David S. Batista <[email protected]>
* Add draft of the Excel To Document converter
* Add license header
* Add release note
* Use Union instead of pipe
* Add openpyxl as additional dep
* Fix zip issue
* few updates from Bijay
* Update deps
* Add markdown test
* Adding more example excels and expanding tests
* Added more tests
* Fix windows test by setting lineterminator
* Addressing PR comments
* PR comments
* Fix linting
* Add log lines for PDF conversion and make skipping more explicit in DocumentSplitter
* Add logging statement for PDFMinerToDocument as well
* Add tests
* Remove unused line
* Remove unused line
* add reno
* Add in PDF file
* Update checks in PDF converters and add tests for document splitter
* Revert
* Remove line
* Fix comment
* Make mypy happy
* Make mypy happy
* Deprecate max_loops_allowed in favour of new argument max_runs_per_component
* Add missing test file
* Some enhancements
* Add version that will remove deprecate stuff
* fix: extract page breaks from .docx files
Context: Currently, DOCXToDocument does not extract page breaks from
word documents. This makes it impossible to do things like split by page
or get correct page number metadata after using something like
DocumentSplitter. For example, if you split by word, the 'page_number'
metadata field will be 1 for all documents.
Solution: Added a method to DOCXToDocument that extracts page breaks
from word documents as '\f' characters so that they are recognized by
DocumentSplitter.
Caveat: Due to the way the python-docx library is set up, you can only
accurately determine the location of the first page break for a given
paragraph. In the rare case that a paragraph contains more than one page
break (which means it is an extremely long paragraph spanning multiple
pages), the 2nd, 3rd, etc. page break locations are not known. To sort
of fix this, I just appended the page break characters to the end of
the paragraph text to keep the overall page number values for the
document consistent.
* Apply suggestions from code review
---------
Co-authored-by: Madeesh Kannan <[email protected]>
* first fucntioning DocxFileToDocument
* fix lazy import message
* add reno
* Add license headder
Co-authored-by: Sebastian Husch Lee <[email protected]>
* change DocxFileToDocument to DocxToDocument
* Update library install to the maintained version
Co-authored-by: Sebastian Husch Lee <[email protected]>
* clan try-exvept to only take non haystack errors into account
* Add wanring on docstring of component ignoring page brakes, mark test as skip
* make warnings lazy evaluations
Co-authored-by: Sebastian Husch Lee <[email protected]>
* make warnings lazy evaluations
Co-authored-by: Sebastian Husch Lee <[email protected]>
* Make warnings lazy evaluated
Co-authored-by: Sebastian Husch Lee <[email protected]>
* Solve f bug
* Get more metadata from docx files
* add 'python-docx' dependency and docs
* Change logging import
Co-authored-by: Sebastian Husch Lee <[email protected]>
* Fix typo
Co-authored-by: Sebastian Husch Lee <[email protected]>
* remake metadata extraction for docx
* solve bug regarding _get_docx_metadata method
* Update haystack/components/converters/docx.py
Co-authored-by: Sebastian Husch Lee <[email protected]>
* Update haystack/components/converters/docx.py
Co-authored-by: Sebastian Husch Lee <[email protected]>
* Delete unused test
---------
Co-authored-by: Sebastian Husch Lee <[email protected]>
* Add first pass at PPTXToDocument converter
* Add test and update code
* Add doc string
* Update docstrings
* Add release notes
* remove unused imports, add to api docs, update pyproject.toml
* Add a new test
* Add dep so tests can run
* Initial commit
* Remove old mock tests
* Fix current_last_page_number calculation
* Carry over unit tests from the other side
* Update pydocs, skip failing tests
* Fix pylint and mypy
* Minor adjustments
* Add release note
* Minor touch ups
* Resolve Document unique id issue by using custom id calculation
* Better hashing, add unit tests
* Small fixes
