Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Text extraction issue with extract_text_to_fp - Uncleaned CID characters #1056

Open
BaillySylvain opened this issue Oct 25, 2024 · 2 comments · May be fixed by #1070
Open

Text extraction issue with extract_text_to_fp - Uncleaned CID characters #1056

BaillySylvain opened this issue Oct 25, 2024 · 2 comments · May be fixed by #1070

Comments

@BaillySylvain
Copy link

Hello,

While using the extract_text_to_fp function with the latest version of pdfminer.six, I've encountered an issue where CID characters (e.g., CID(123)) appear in the extracted text. These characters seem to be associated with the fonts used in the PDF, but there is no straightforward way to clean them or make them readable.

This makes it difficult to obtain a clean and readable text from the PDF, as these CID characters are not converted into standard Unicode characters or intelligible text. I would like to know if there is a recommended way to handle this issue with pdfminer.six or if improvements can be made to the library to manage such cases more effectively.

Additional Information:

Version of pdfminer.six used: Version: 20240706
Operating system: linux

Example code used for extraction:

            extract_text_to_fp(srcFile, xmlFile, output_type='xml', codec='utf-8', laparams=LAParams(
                detect_vertical=False,
                word_margin=0.2,
                char_margin=3,
                boxes_flow=1,
                all_texts=True,
            ))

Thank you for your incomming answer and your time.
page10.pdf

@Some1Somewhere
Copy link

Some1Somewhere commented Oct 29, 2024

It's been discussed in euske/pdfminer#122 and on stack overflow : https://stackoverflow.com/questions/74416930/how-to-solve-cidx-pdfplumber-python-text-extraction

This is the code I've used to solve cid, might not work for all use cases, but works for mine!

def prune_text(text):
    """
    Replace (cid:x) patterns in the text with corresponding characters.

    Args:
        text (str): The input text containing (cid:x) patterns.

    Returns:
        str: The processed text with (cid:x) replaced.
    """

    def replace_cid(match):
        cid_num = int(match.group(1))
        # Define specific CID to character mappings
        cid_mapping = {
            0: "- ",  # Example: (cid:0) to bullet point
            # Add more mappings as needed
            # e.g., 66: 'B', etc.
        }
        try:

            return cid_mapping.get(
                cid_num, chr(cid_num)
            )  # Return mapped char or empty string if not found
        except:
            return ""

    # Regular expression to find all (cid:x) patterns
    cid_pattern = re.compile(r"\(cid:(\d+)\)")
    pruned_text = re.sub(cid_pattern, replace_cid, text)
    return pruned_text

@dhdaines
Copy link
Contributor

dhdaines commented Nov 27, 2024

The handling of unmapped glyphs is a method on PDFLayoutAnalyzer which can be overriden in a subclass or patched at runtime. So you can do this in your code for instance:

from pdfminer.converter import PDFLayoutAnalyzer
PDFLayoutAnalyzer.handle_undefined_char = lambda *args: ""

@dhdaines dhdaines linked a pull request Nov 27, 2024 that will close this issue
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants
@dhdaines @Some1Somewhere @BaillySylvain and others