Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Simplify the Script #11

Open
wants to merge 9 commits into
base: main
Choose a base branch
from
Open

Conversation

CGBassPlayer
Copy link

@CGBassPlayer CGBassPlayer commented Jul 10, 2023

While what the project had was a great start, this iteration has the same idea but accomplishes it in a much similar fashion as well as implements some better practices.

This PR fixes #6, closes #9, and fixes #10

How it works

The new script lives in an infinite loop. And goes as follows:

  1. Calls the PeerTube video search API endpoint searching for all videos that currently a live stream
  2. It will compare the list of currently running browsers to the list of new videos that live. If a video is no longer live, it will close that particular browser.
  3. Each video that gets returned (in the case of JB's peertube, that is usually 1 video at a time when they are live), it will start up a firefox browser with selenium if it does not already have an open browser.
  4. The program with pause a set amount of time (the ping_interval parameter sets this. Currently default is 300 seconds or 5 minutes) before iterating through the loop again.

Script Parameters

This script now takes 2 parameters as environment variables.

  1. api_url: this is the base url of the peertube instance you are calling. Do not include anything other than the base URL so https://jupiter.tube/.
  2. ping_interval: this is how often the script checks peertube for a live stream. The default is 300 seconds (5 minutes) for the main docker compose and 5 seconds for the development compose.

Things that need improvement

Time to wait between API calls

Using time.sleep() is far from the best way to handle the waiting period, but it works well enough in this first iteration of the "new" script. I would like to bring back in the schedule module that you were using originally, so that will be in a followup PR if someone does not tackle that.

Logging

Current the script just prints to the console as its way of logging while it is running, but adding real logging to the script would be helpful in case an error occurs or if a user does not wants to turn up or down how much the information is being captured.

### Type Hinting
I started this but as I was figuring stuff out I did not do it as much as I originally planned. Type hinting help the machine and people understand what a variable or function or parameter is suppose to be and can catch things we can miss. That'll be in a follow up PR or on this one depending on how long it stays open. I took care of this in this PR now

### Documentation
Eventually there should be docs adding how the script works, how to run it, and other things of that nature. Not a big deal yet, but something to think about I started working on this in this PR

Let me know if you have any questions or concerns 🙂

@CGBassPlayer CGBassPlayer changed the title Simplify the script Simplify the Script Jul 10, 2023
@CGBassPlayer CGBassPlayer marked this pull request as draft July 11, 2023 02:35
@CGBassPlayer
Copy link
Author

Temporarily converting to draft as I came across an issue with run in docker and the script not finding the Firefox web driver

@CGBassPlayer CGBassPlayer marked this pull request as ready for review July 12, 2023 00:35
@CGBassPlayer
Copy link
Author

I have fix the issues I was facing as well as put in some improvements

@tyrsarm
Copy link
Owner

tyrsarm commented Jul 20, 2023

Awesome work. I did not even think about using the API url to filter the video list. Not including the overhaul you did. I will make the container and test it this Sunday. If it passes, I will merge it in. Thank you

@tyrsarm
Copy link
Owner

tyrsarm commented Jul 23, 2023

When testing the container based off of this pull request I was getting an error.

This was fixed with adding these two lines in the Dockerfile at line 25:
mkdir /home/user &&
chown -R user:user /home/user

The error looks to be coming from installing GeckoDriverManager from line 32 from main.py as it looks to require /home/user directory to install.

Can you please verify my findings and add the fix to the pull request if needed. The issue only appears when there is a live stream active.

Log:

~ sudo docker-compose up
[+] Building 0.0s (0/0)
[+] Running 2/1
✔ Network peertube-headless-seeder-simplify_default Created 0.1s
✔ Container peertube-headless-seeder-simplify-peertube-seeder-1 Created 0.1s
Attaching to peertube-headless-seeder-simplify-peertube-seeder-1
peertube-headless-seeder-simplify-peertube-seeder-1 | Checking for live videos
peertube-headless-seeder-simplify-peertube-seeder-1 | Current live videos: 1
[WDM] - Downloading: 19.2kB [00:00, 19.2MB/s]
[WDM] - Downloading: 19.2kB [00:00, 41.7MB/s]
[WDM] - Downloading: 100%|██████████| 2.93M/2.93M [00:00<00:00, 12.4MB/s]
peertube-headless-seeder-simplify-peertube-seeder-1 | Traceback (most recent call last):
peertube-headless-seeder-simplify-peertube-seeder-1 | File "/app/./main.py", line 63, in
peertube-headless-seeder-simplify-peertube-seeder-1 | main(api_url, api_call_interval)
peertube-headless-seeder-simplify-peertube-seeder-1 | File "/app/./main.py", line 53, in main
Lpeertube-headless-seeder-simplify-peertube-seeder-1 | browser: WebDriver = run_browser_instance(video)
peertube-headless-seeder-simplify-peertube-seeder-1 | File "/app/./main.py", line 32, in run_browser_instance
peertube-headless-seeder-simplify-peertube-seeder-1 | browser: WebDriver = webdriver.Firefox(service=FirefoxService(GeckoDriverManager().install()),
peertube-headless-seeder-simplify-peertube-seeder-1 | File "/usr/local/lib/python3.10/site-packages/webdriver_manager/firefox.py", line 37, in install
peertube-headless-seeder-simplify-peertube-seeder-1 | driver_path = self._get_driver_path(self.driver)
peertube-headless-seeder-simplify-peertube-seeder-1 | File "/usr/local/lib/python3.10/site-packages/webdriver_manager/core/manager.py", line 31, in _get_driver_path
peertube-headless-seeder-simplify-peertube-seeder-1 | binary_path = self.driver_cache.save_file_to_cache(driver, file)
peertube-headless-seeder-simplify-peertube-seeder-1 | File "/usr/local/lib/python3.10/site-packages/webdriver_manager/core/driver_cache.py", line 40, in save_file_to_cache
peertube-headless-seeder-simplify-peertube-seeder-1 | archive = save_file(file, path)
peertube-headless-seeder-simplify-peertube-seeder-1 | File "/usr/local/lib/python3.10/site-packages/webdriver_manager/core/utils.py", line 38, in save_file
peertube-headless-seeder-simplify-peertube-seeder-1 | os.makedirs(directory, exist_ok=True)
peertube-headless-seeder-simplify-peertube-seeder-1 | File "/usr/local/lib/python3.10/os.py", line 215, in makedirs
peertube-headless-seeder-simplify-peertube-seeder-1 | makedirs(head, exist_ok=exist_ok)
peertube-headless-seeder-simplify-peertube-seeder-1 | File "/usr/local/lib/python3.10/os.py", line 215, in makedirs
peertube-headless-seeder-simplify-peertube-seeder-1 | makedirs(head, exist_ok=exist_ok)
peertube-headless-seeder-simplify-peertube-seeder-1 | File "/usr/local/lib/python3.10/os.py", line 215, in makedirs
peertube-headless-seeder-simplify-peertube-seeder-1 | makedirs(head, exist_ok=exist_ok)
peertube-headless-seeder-simplify-peertube-seeder-1 | [Previous line repeated 2 more times]
peertube-headless-seeder-simplify-peertube-seeder-1 | File "/usr/local/lib/python3.10/os.py", line 225, in makedirs
peertube-headless-seeder-simplify-peertube-seeder-1 | mkdir(name, mode)
peertube-headless-seeder-simplify-peertube-seeder-1 | PermissionError: [Errno 13] Permission denied: '/home/user'
peertube-headless-seeder-simplify-peertube-seeder-1 exited with code 0

@tyrsarm
Copy link
Owner

tyrsarm commented Jul 23, 2023

While testing the pull request, I found this JSON exception also.
It appears that the root issue is requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0).
I am troubleshooting the issue.

It appears after the browser starts during a live:

peertube-headless-seeder-simplify-peertube-seeder-1 | Checking for live videos
peertube-headless-seeder-simplify-peertube-seeder-1 | Current live videos: 1
peertube-headless-seeder-simplify-peertube-seeder-1 | Checking for live videos
peertube-headless-seeder-simplify-peertube-seeder-1 | Current live videos: 1
peertube-headless-seeder-simplify-peertube-seeder-1 | Checking for live videos
peertube-headless-seeder-simplify-peertube-seeder-1 | Traceback (most recent call last):
peertube-headless-seeder-simplify-peertube-seeder-1 | File "/usr/local/lib/python3.10/site-packages/requests/models.py", line 971, in json
peertube-headless-seeder-simplify-peertube-seeder-1 | return complexjson.loads(self.text, **kwargs)
peertube-headless-seeder-simplify-peertube-seeder-1 | File "/usr/local/lib/python3.10/json/init.py", line 346, in loads
peertube-headless-seeder-simplify-peertube-seeder-1 | return _default_decoder.decode(s)
peertube-headless-seeder-simplify-peertube-seeder-1 | File "/usr/local/lib/python3.10/json/decoder.py", line 337, in decode
peertube-headless-seeder-simplify-peertube-seeder-1 | obj, end = self.raw_decode(s, idx=_w(s, 0).end())
peertube-headless-seeder-simplify-peertube-seeder-1 | File "/usr/local/lib/python3.10/json/decoder.py", line 355, in raw_decode
peertube-headless-seeder-simplify-peertube-seeder-1 | raise JSONDecodeError("Expecting value", s, err.value) from None
peertube-headless-seeder-simplify-peertube-seeder-1 | json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
peertube-headless-seeder-simplify-peertube-seeder-1 |
peertube-headless-seeder-simplify-peertube-seeder-1 | During handling of the above exception, another exception occurred:
peertube-headless-seeder-simplify-peertube-seeder-1 |
peertube-headless-seeder-simplify-peertube-seeder-1 | Traceback (most recent call last):
peertube-headless-seeder-simplify-peertube-seeder-1 | File "/app/./main.py", line 63, in
peertube-headless-seeder-simplify-peertube-seeder-1 | main(api_url, api_call_interval)
peertube-headless-seeder-simplify-peertube-seeder-1 | File "/app/./main.py", line 43, in main
peertube-headless-seeder-simplify-peertube-seeder-1 | live_videos: list[str] = get_live_video_urls(url)
peertube-headless-seeder-simplify-peertube-seeder-1 | File "/app/./main.py", line 19, in get_live_video_urls
peertube-headless-seeder-simplify-peertube-seeder-1 | videos_json: dict = requests.get(search_url).json()
peertube-headless-seeder-simplify-peertube-seeder-1 | File "/usr/local/lib/python3.10/site-packages/requests/models.py", line 975, in json
peertube-headless-seeder-simplify-peertube-seeder-1 | raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
peertube-headless-seeder-simplify-peertube-seeder-1 | requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
peertube-headless-seeder-simplify-peertube-seeder-1 exited with code 1

@CGBassPlayer
Copy link
Author

CGBassPlayer commented Jul 24, 2023

PermissionError: [Errno 13] Permission denied: '/home/user'

This has been a pain point for me as well because I have not been able to figure out the best way to get this to run as a non-root user. I need to play with the order of the installs. Maybe do the pip installs at the user level could help, but I will have to look into it.

requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

This one is new to me. I did not come across that when I was working on this, but I tell you that that specific error has to do with expecting JSON back and getting something else instead, usually a string.

A good way to test that replacing requests.get(search_url).json() with a print statement with requests.get(search_url).text and see what you get. If it is not JSON that is the issue.

Last time I encountered an error like that, I was being throttled, so make sure you watch how often you ping their instance. Here is the default for a peertube API:
image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants