Distribute cron jobs in a cluster environment

[kiatng]

There are lots of cron jobs. The production has two app servers with a load balancer. Cron is setup to run in server1. Is there a way to configure cron jobs in the the xml files for server2 only?

[kiatng]

Ref https://github.com/AOEpeople/Aoe_Scheduler one of the feature is

Cron groups: Split you crons into different groups so you can run them in parallel or distribute them on multiple servers.

[kiatng]

Configure the crontab in servers:

server1

* * * * * ! test -e /{root}/maintenance.flag && /bin/bash /{root}/scheduler_cron.sh --mode default --excludeGroup server2

server2

* * * * * ! test -e /{root}/maintenance.flag && /bin/bash /{root}/scheduler_cron.sh --mode default --includeGroups server2

In module/etc/config.xml

<crontab>
    <jobs>
        <job_code>
            <schedule> ...</schedule>
            <run> ....</run>
            <groups>server2</groups> <!-- add this -->
        </job_code>
    </jobs>
</crontab>

[empiricompany]

yes with aoe_scheduler I usually group long running jobs to run them in parallel without blocking other primary jobs like the send of transactional emails

you can configure to run specific group only in server 2 and leave others in server1

[empiricompany]

I add a new cron job for each assigned group and exclude it from default cronjob

[empiricompany]

* * * * * ! test -e /{root}/maintenance.flag && /bin/bash /{root}/scheduler_cron.sh --mode always
* * * * * ! test -e /{root}/maintenance.flag && /bin/bash /{root}/scheduler_cron.sh --mode default --includeGroups crawler
* * * * * ! test -e /{root}/maintenance.flag && /bin/bash /{root}/scheduler_cron.sh --mode default --includeGroups feeds
* * * * * ! test -e /{root}/maintenance.flag && /bin/bash /{root}/scheduler_cron.sh --mode default --includeGroups xtento
* * * * * ! test -e /{root}/maintenance.flag && /bin/bash /{root}/scheduler_cron.sh --mode default --includeGroups ebizmarts_mailchimp
* * * * * ! test -e /{root}/maintenance.flag && /bin/bash /{root}/scheduler_cron.sh --mode default --includeGroups hubspot
* * * * * ! test -e /{root}/maintenance.flag && /bin/bash /{root}/scheduler_cron.sh --mode default --includeGroups indexer
* * * * * ! test -e /{root}/maintenance.flag && /bin/bash /{root}/scheduler_cron.sh --mode default --excludeGroups crawler,feeds,xtento,ebizmarts_mailchimp,hubspot,indexer

[kiatng]

That makes sense!

[kiatng]

We tested the shell commands, but needed to use the & backgrounding command to effect concurrency:

! test -e {root}/maintenance.flag && /bin/bash {root}/scheduler_cron.sh --mode default --includeGroups p1 &
! test -e {root}/maintenance.flag && /bin/bash {root}/scheduler_cron.sh --mode default --includeGroups p2 &
! test -e {root}/maintenance.flag && /bin/bash {root}/scheduler_cron.sh --mode default --includeGroups p3 &
! test -e {root}/maintenance.flag && /bin/bash {root}/scheduler_cron.sh --mode default --includeGroups p4 &
! test -e {root}/maintenance.flag && /bin/bash {root}/scheduler_cron.sh --mode default --includeGroups p5 &
! test -e {root}/maintenance.flag && /bin/bash {root}/scheduler_cron.sh --mode default --includeGroups p6 &
! test -e {root}/maintenance.flag && /bin/bash {root}/scheduler_cron.sh --mode default --includeGroups p7 &
! test -e {root}/maintenance.flag && /bin/bash {root}/scheduler_cron.sh --mode default --includeGroups p8 &
! test -e {root}/maintenance.flag && /bin/bash {root}/scheduler_cron.sh --mode default --includeGroups web02 &

[empiricompany]

ok maybe this is a container without a native cron? in my bare metal runs all cronjobjs in parallel