What is the best method to keep scrapyrt open/active even after reboot

[jordankendall86]

I have been exploring many options on how to keep scrapyrt open and active even after reboot, but I am unsure what is best. I was thinking of using immortal.run. I have used tmux, but it eventually kills the scrapyrt process after sometime. Not sure how long it persists. I have also tried without success to write a bash script to start scrapyrt within a tmux session (new or existing) if the scrapyrt process is not active. Is immortal.run a good option? Is a simple bash script and using crontab (for example: @reboot) possible? I am open to any option that will work. Please share an example. Thanks. (I am currently using Ubuntu 18.04 VM on Azure)

Here is an example of my attempt at writing a script in bash:

#!/bin/bash
if ! pgrep -x "scrapyrt" > /dev/null
then
	SESSION_NAME=coinSpider
	tmux has-session -t $SESSION_NAME 2>/dev/null
	if [ $? -ne 0 ]
	then
		tmux new-session -d -s $SESSION_NAME "$*"
	fi
	tmux attach -t $SESSION_NAME \; \
	send-keys -t 'scrapyrt -p 9080 -i 0.0.0.0' C-m \;
fi

[jordankendall86]

Below is my revised bash script labeled "scrapyrt-9080":

#!/bin/bash

if ! pgrep -x "scrapyrt" > /dev/null
then
        SESSION_NAME=coinSpider
	sudo -i -u jordan tmux has-session -t $SESSION_NAME 2>/dev/null
	if [ $? -ne 0 ]
	then
			sudo -i -u jordan tmux new-session -s $SESSION_NAME -d
			echo "New Session $SESSION_NAME"
	fi
	sleep 3
	sudo -i -u jordan tmux send-keys -t coinSpider C-z 'cd scrapy' C-m
	sleep 3
	sudo -i -u jordan tmux send-keys -t coinSpider C-z 'scrapyrt -p 9080 -i 0.0.0.0' C-m
	echo "Restart"
else
        echo "Running"
fi

I saved the bash file in the following path (also make the file executable): /usr/local/bin/
I then used crontab to run it @reboot and every minute (applied to root):
@reboot /usr/local/bin/scrapyrt-9080 >/dev/null 2>&1
* * * * * /usr/local/bin/scrapyrt-9080 >/dev/null 2>&1

[jordankendall86]

I was not able to get immortal.run to work. If anyone is familiar with it, then please provide an example of how to implement with scrapyrt instead of the bash script and crontab answer I provided in the previous comment. Thanks in advance.

[pawelmhm]

hey @jordankendall86 what exactly is your use case? What do you mean by reboot? You mean reboot of operating system, right? So you want scrapyrt to run as a service in background that is started by operating system every time you start your machine?

If this is your use case try supervisord http://supervisord.org/running.html I've used that in production in some contexts and it was working ok for me. You need to set it up according to documentation, specify configuration for scrapyrt and configure it to run on startup.

[jordankendall86]

I can't remember the details of how I got it to eventually work, but I know it was simple. I believe I programmed a bash script and used cron in order to ultimately keep the tmux session alive. So I did not use immortal.run. The problem I had was that the tmux session would time out or if the server crashed or had to be rebooted the tmux session would end. I needed something that would keep the tmux session alive no matter what happened. My simple program is still working, I just don't have the time to look under the hood. I will try come back to this comment at a later time and post the detailed solution I came up with. Although, thanks for tip, I will look into supervisord.

[pawelmhm]

There is wiki here that could contain useful info like this, otherwise it could go to readthedocs docs https://github.com/scrapinghub/scrapyrt/wiki/Deploy

[kennylajara]

There is wiki here that could contain useful info like this, otherwise it could go to readthedocs docs https://github.com/scrapinghub/scrapyrt/wiki/Deploy

That documentation has the deployment to ubuntu server marked as todo since 2016.

[jagdish1o1]

It's 2022 and it's still marked as todo since 2016