[YouTube] Refactor JavaScript usage and fix extraction of obfuscated signature deobfuscation function #1108
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
AudricV
added
bug
Stypox
requested changes
Sep 19, 2023
.../main/java/org/schabi/newpipe/extractor/services/youtube/YoutubeJavaScriptPlayerManager.java
Outdated
Show resolved
Hide resolved
...actor/src/main/java/org/schabi/newpipe/extractor/services/youtube/YoutubeSignatureUtils.java
Outdated
Show resolved
Hide resolved
...actor/src/main/java/org/schabi/newpipe/extractor/services/youtube/YoutubeSignatureUtils.java
Show resolved
Hide resolved
...actor/src/main/java/org/schabi/newpipe/extractor/services/youtube/YoutubeSignatureUtils.java
Outdated
Show resolved
Hide resolved
...g/schabi/newpipe/extractor/services/youtube/YoutubeThrottlingParameterDeobfuscationTest.java
Outdated
Show resolved
Hide resolved
3 tasks
…uscation function extraction The goal of this class is to decouple the extraction of signature timestamp and signature deobfuscation function from YoutubeStreamExtractor. The extraction of the signature deobfuscation function has been also adapted to support the latest YouTube player versions. This new class, YoutubeSignatureUtils, doens't store anything temporary such as a copy of the player code, which has to be passed where required. It is not public, as it will be used by a JavaScript player manager class in the future, in order to handle in a better way fetching, caching and resetting cache of the player code.
This commit is introducing breaking changes. For clients, everything is managed in a new class called YoutubeJavaScriptPlayerManager: - caching JavaScript base player code and its extracted code (functions and variables); - getting player signature timestamp; - getting deobfuscated signatures of streaming URLs; - getting streaming URLs with a throttling parameter deobfuscated, if applicable. The class delegates the extraction parts to external package-private classes: - YoutubeJavaScriptExtractor, to extract and download YouTube's JavaScript base player code: it always already present before and has been edited to mainly remove the previous caching system and made it package-private; - YoutubeSignatureUtils, for player signature timestamp and signature deobfuscation function of streaming URLs, added in a recent commit; - YoutubeThrottlingParameterUtils, which was originally YoutubeThrottlingDecrypter, for throttling parameter of streaming URLs deobfuscation function and checking whether this parameter is in a streaming URL. YoutubeJavaScriptPlayerManager caches and then runs the extracted code if it has been executed successfully. The cache system of throttling parameters deobfuscated values has been kept, its size can be get using the getThrottlingParametersCacheSize method and can be cleared independently using the clearThrottlingParametersCache method. If an exception occurs during the extraction or the parsing of a function property which is not related to JavaScript base player code fetching, it is stored until caches are cleared, making subsequent failing extraction calls of the requested function or property faster and consuming less resources, as the result should be the same until the base player code changes. All caches can be reset using the clearAllCaches method of YoutubeJavaScriptPlayerManager. Classes using JavaScript base player code and utilities directly (in the code and its tests) have been also updated in this commit.
The signature timestamp is used as a number by HTML5 clients, so it should be used in the same way by the extractor too instead of being a string. As the timestamp doesn't seem to exceed 5 digits, an integer is used to store its value.
…deobfuscation function extraction and execution
…on only where needed
AudricV
force-pushed
the
yt_refactor-js-usage
branch
from
63aec71 to
6ed2209
Compare
|
I can confirm it works, I tested in NewPipe |
6 tasks
This was referenced Oct 5, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR refactors the usage of YouTube's JavaScript base player file in the extractor.
It makes a single public class handling operations and caches of this file,
YoutubeJavaScriptPlayerManager, instead of being split in multiple files with questionable designs (such as the one of the deobfuscation function for obfuscated signatures of streaming URLs from HTML5 clients inYoutubeStreamExtractor).Extraction and parsing parts are delegated to different package-private classes:
YoutubeJavaScriptExtractor, handling the extraction of JavaScript's base player file;YoutubeThrottlingParameterUtils, handling the extraction of throttling parameter of streaming URLs from HTML5's clients and the detection of this parameter for a given streaming URL;YoutubeSignatureUtils, handling the extraction of the signature timestamp property and the deobfuscation function for obfuscated signatures of streaming URLs from HTML5 clients.Each extraction process has now a corresponding test.
It also improves the code on the fly, fixes the extraction of the deobfuscation function for obfuscated signatures, and makes use of a number when sending the signature timestamp in InnerTube's player requests instead of a string, in order to be consistent with HTML5 clients.
This PR introduces important breaking changes:
YoutubeJavaScriptExtractoris not public anymore, the extraction of YouTube's JavaScript base player file isn't intended to be used outside of the extractor;resetDeobfuscationCodemethod ofYoutubeStreamExtractorhas been removed. Use the methodclearAllCachesofYoutubeJavaScriptPlayerManagerinstead, which also clears other cached data;YoutubeThrottlingDecrypterhas been renamed toYoutubeThrottlingParameterUtils, which is a package-private class and the methodsgetCacheSizeandclearCachehave been removed. Use respectivelygetThrottlingParametersCacheSizeandclearThrottlingParametersCachemethods ofYoutubeJavaScriptPlayerManagerinstead.DeobfuscateExceptionnested exception class inYoutubeStreamExtractorhas been removed. A standardParsingExceptionwill be thrown when the deobfuscation function couldn't be parsed or executed instead.Fixes TeamNewPipe/NewPipe#10347.