Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test to fix SB download #1255

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open

Conversation

project-majo
Copy link

@project-majo project-majo commented Mar 26, 2019

Category

This change is exactly one of the following (please change [ ] to [x]) to indicate which:

  • a bug fix (Fix #...)
  • a new Ripper
  • a refactoring
  • a style change/fix
  • a new feature

Description

change how Spankbang pages are downloaded (by following requests in developer mode of chrome)
test is provided
needs integration in SpankbangRipper

Testing

Required verification:

  • I've verified that there are no regressions in mvn test (there are no new failures or errors).
  • I've verified that this change works as intended.
    • Downloads all relevant content.
    • Downloads content from multiple pages (as necessary or appropriate).
    • Saves content at reasonable file names (e.g. page titles or content IDs) to help easily browse downloaded content.
  • I've verified that this change did not break existing functionality (especially in the Ripper I modified).

Optional but recommended:

  • I've added a unit test to cover my change.

@project-majo
Copy link
Author

project-majo commented Mar 26, 2019

"description" in script[type=application/ld+json] sometimes has newlines and those need to be removed first .replace("\r","").replace("\n","") before giving it to new JSONObject(...)

it may be necessary to download actualUrl with the same connection used to download the original url and do the same for infoUrl

@GenuineSounds
Copy link
Contributor

Your test video is hilarious. Take care.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants