sitemaps.py does not work with some .gz sitemaps

[internetyev]

Hi @eliasdabbas!
I got stuck with an error, trying to parse a booking.com sitemap.
The error message was given by sitemaps.py: "not well-formed (invalid token)" when trying to process a sitemap eg. https://www.booking.com/sitembk-beaches-pt-br.0000.xml.gz

I was able to go around this error by modifying the original code of sitemaps.py:

original code - lines 483-493:

    if sitemap_url.endswith('xml.gz'):
        xml_text = urlopen(Request(sitemap_url,
                                   headers={'Accept-Encoding': 'gzip',
                                            'User-Agent': 'advertools-' +
                                                          version}))
        resp_headers = xml_text.getheaders()
        xml_text = GzipFile(fileobj=xml_text)
    else:
        xml_text = urlopen(Request(sitemap_url, headers=headers))
        resp_headers = xml_text.getheaders()
    xml_string = xml_text.read()
    root = ElementTree.fromstring(xml_string)

modified code:

    if sitemap_url.endswith('xml.gz'):
        xml_text = urlopen(Request(sitemap_url,
                                   headers={'Accept-Encoding': 'gzip',
                                            'User-Agent': 'advertools-' +
                                                          version}))
        resp_headers = xml_text.getheaders()
        xml_text = GzipFile(fileobj=xml_text) 
        #convert to string
        with gzip.open(xml_text) as f:
            file_content = f.read()
        xml_string = file_content
    else:
        xml_text = urlopen(Request(sitemap_url, headers=headers))
        resp_headers = xml_text.getheaders()
        xml_string = xml_text.read()
    root = ElementTree.fromstring(xml_string)

To make it work, I had to import gzip in the beginning, and introduce a new variable file_content as I didn't want to risk, if there are some other dependencies.

Also, I didn't test this code with other non-gz sitemaps yet.

Definitely, the code is not optimal, but at least it got me what I wanted - to parse the booking.com .gz sitemaps with no issues.
I still don't know the exact reason why the .gz sitemap wasn't parsed properly, and would appreciate some advice.

[eliasdabbas]

Thanks for letting me know @internetyev

The sitemap_to_df function should work with zipped XML sitemaps as well.

In this case the error seems to be in the sitemap itself. The full traceback for this particular sitemap shows that the error was raised by the ElementTree.fromstring function. It could not parse this as valid XML, and pointed to the location of the error (line 1, column 0).

Traceback (most recent call last):

  File ~/venv311/lib/python3.11/site-packages/IPython/core/interactiveshell.py:3460 in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)

  Cell In[44], line 1
    adv.sitemap_to_df('https://www.booking.com/sitembk-beaches-pt-br.0000.xml.gz')

  File ~/venv311/lib/python3.11/site-packages/advertools/sitemaps.py:494 in sitemap_to_df
    root = ElementTree.fromstring(xml_string)

  File /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/xml/etree/ElementTree.py:1338 in XML
    parser.feed(text)

  File <string>
ParseError: not well-formed (invalid token): line 1, column 0

Double-checking here, I also got that the sitemap is invalid:

So, this is more of a business decision than a coding one.
Should errors be raised on sitemaps with errors? I think so. I believe this is the correct behavior, as we should be alerted to this, and not wrongly parse sitemaps.

But sometimes, like in this case, you are more interested in analyzing the content of a competitor, and don't care much whether or not it is valid. This is the price to pay for this behavior.

More than happy to discuss further if you have other suggestions to make this user-friendly, useful, but make sure it is correct.

[internetyev]

@eliasdabbas thank you for quick reply! I agree that this is more of a business decision - whether to parse sitemaps with errors, or not. I've noticed that you already give warnings if a sitemap does not follow the guidelines. In this particular case, I would consider allowing the erroneous sitemaps if the sitemaps are parseable (eg. wrong http header, wrongly specified encoding or file format, etc.) and the URLs can be distilled with confidence. Essentially, the same way as Screaming Frog does.
Honestly, I think my use case with Booking.com is quite rare and does not justify spending too much effort on this. Maybe, instead of adding new features, it would be ok to just add a few more comments to the code base, to help out with code modifications to parse wrongly built sitemaps?

[eliasdabbas]

@internetyev That's an interesting idea.

The current rule is very strict; errors: raise an exception, no errors: parse.

It might make sense to raise an exception only on parse errors. Otherwise parse, and report the other type of error.
So it would be something like this:

• No errors at all: parse normally
• Non parse errors (wrong header, etc): report the error, and parse
• Parse errors: raise an exception

Does that make sense?