Coverage for haystack/components/preprocessors/_page_numbers.py: 89%

1# SPDX-FileCopyrightText: 2022-present deepset GmbH <info@deepset.ai> 

2# 

3# SPDX-License-Identifier: Apache-2.0 

4 

5""" 

6The `page_number` convention shared by the splitters. 

7 

8Splitters record the page a chunk came from in its `page_number` metadata field, counting page breaks 

9(form feed, "\f") in the source document. The page reported is the one the chunk's first non-page-break 

10character lies on: a chunk that spans a page break belongs to the page it starts on, and page breaks a 

11chunk opens with belong to that chunk's own page rather than to the page it was carved out of. 

12""" 

13 

14 

15def _leading_page_breaks(text: str, page_break_character: str = "\f") -> int: 

16 """ 

17 Count how many pages a chunk's text starts past the page its first character is on. 

18 

19 This is the run of page breaks the chunk opens with, except for a chunk made up entirely of page 

20 breaks: that chunk is an empty page and stays on the page it starts on, rather than being pushed onto 

21 the following one. `split_by="page"` emits such chunks for every blank page. 

22 

23 :param text: The chunk to inspect. 

24 :param page_break_character: The character sequence marking a page break. 

25 :returns: The number of pages to advance, 0 if the chunk starts with anything but a page break or 

26 consists only of page breaks. 

27 """ 

28 if not page_break_character: 

29 return 0 

30 

31 offset = 0 

32 while text.startswith(page_break_character, offset): 

33 offset += len(page_break_character) 

34 

35 # Nothing but page breaks: an empty page, which belongs to the page it starts on. 

36 if offset >= len(text): 

37 return 0 

38 return offset // len(page_break_character)