feat: Add `page_number` to meta of Documents in `DocumentSplitter` about haystack

[davidsbatista]

Yes, that helps! So essentially, the DocumentSplitter should return a List[Document].

But since it has as input:

def run(self, documents: List[Document]):

It should return a List[List[Documents]]

do you agree?

[davidsbatista]

Hi Sebastian, I can pick up this again after finishing some high-priority issues I need to handle - maybe by the end of the week. Just to let you know, I haven't forgot it

[sjrl]

This issue #6706 is related since we currently do not keep page break information when converting a PDF file to a Haystack Document.

[davidsbatista]

Hi @sjrl :)

This is my first issue. I'm trying to understand the requisites better. It seems to me that to keep the page number and the associated text, I suppose we have to keep the chunks in the metadata, e.g.:

units = self._split_into_units(doc.content, self.split_by)
text_splits = self._concatenate_units(units, self.split_length, self.split_overlap)
metadata = deepcopy(doc.meta)
metadata["source_id"] = doc.id
metadata["page_number"] = units
split_docs += [Document(content=txt, meta=metadata) for txt in text_splits]

This has a few drawbacks:

• duplicated text, doc.content and doc.metadata['page_number'] now have the same information, a possible solution would be to have self._concatenate_units() being triggered only when doc.content is called/needed
• the metadata["page_number"] has the page number 0 - but this can be easily fixed

[sjrl]

Hi @davidsbatista!

Thanks for taking on this issue :)

I don't think we need to keep the associated text for the use case I am imagining. Basically what we are interested in Haystack terms would like this

1. Load a PDF File
2. Convert a PDF file to a single Document object --> PyPDFToDocument
3. Split the single Document into Chunked Documents (so Document to List of Documents) --> Document Splitter

• In this final step I would like to insert a page_number into the each Doc's metadata in the List of Documents that would tell me which page the chunked doc came from based on the original single Document. This tracking of page_number was done in Haystack v1 by counting and keeping track of page breaks (\f)

Does this make more sense?

[sjrl]

Hmm I'm not entirely sure. Initially I would say that it makes sense to return List[List[Documents]], but often we want a flattened list to be returned since we will often directly write these documents to a document store which I believe expects List[Document] as input.

So I think to keep that workflow working we should return List[Document] or have some way of flattening the list. What do you think?

[sjrl]

No problem! Thanks for the update

[lambda-science]

Would be interested by a follow up about this 👀 If something I could do ?

[davidsbatista]

@lambda-science there hasn't been any follow up, feel free to start working on it if you feel like it