what is the best way to access a .json thru HTTPs with header requirement

[BlazarKnight]

Body

I'm trying to access a specific website through Jupiter and I keep on running into errors with the header because it requires the header whenever you connect I have found three options

1. use requests with the built-in header function but that leads to problems because they send the data packed as a deflate and I haven't figured out how to inflate it.
2. Using SEC-Edgar leads to problems as I haven't figured out how to access their specific data types and turn them into something useful beyond the internal workings of the module.
3. use https://r.jina.ai/ because it didn't have any problems with the header or anything like that when I tried through the website and I'm hoping to look through the code and figure out what exactly they did for that and use that with credit.
4. might not have found other approaches that are far better than these 3 and I just don't know enough to implement. them

Guidelines

• I have read and understood this category's guidelines before making this post.

[MindOfBear]

Hello @BlazarKnight,

1. requests with Deflate Issue
   For the requests deflate issue, you can utilize Python's zlib library to handle the compressed data:

import requests
import zlib

headers = {
    'User-Agent': 'Your User Agent',
    'Accept-Encoding': 'deflate'
}

response = requests.get('https://example.com', headers=headers)

if response.headers.get('Content-Encoding') == 'deflate':
    data = zlib.decompress(response.content, -zlib.MAX_WBITS)
else:
    data = response.content

2. SEC-Edgar Data Types
   When dealing with SEC-Edgar's specific data types, consider using Python libraries like xml.etree.ElementTree or json for parsing structured data. This can help transform the data into a more usable format beyond the module's internal workings.

3. https://r.jina.ai/ Header Handling
   To understand how https://r.jina.ai/ handles headers without issues, inspect their network requests using browser developer tools. This can provide insights into their header management approach. You might find valuable information by examining their code and applying similar techniques with appropriate credit.

Also you may also want to explore web scraping frameworks like Scrapy.

Kind regards,
Ovidiu

[BlazarKnight]

thanks so much