Crank is a library for building job-processing systems. It's main goals are simplicity and understandability.
- the only source for messages is kafka
- every job is just a single thread
(require '[crank.core :as crank])
(def monitor (crank/init {:report prn})) ; :report is optional
(crank/start monitor "mk order"
{:topic "mk_order"
:func prn
:timeout 10000 ;; ms, optional
:batch? true ;; Optional, `:func` will be passed a batch of messages
;; rather than a single message.
;; Batch size controlled by [:kafka :max.poll.records]
:kafka {:bootstrap.servers "kafka1:9192"
:group.id "crank"
:max.poll.records (int 10000)}}) ; upper limit on messages to be read from Kafka in one request (default 10000)
Crank will fire up thread for a job, and will pass data read from Kafka to supplied function. If a job dies because of something - an exception occured inside job or inside Crank itself - Crank's monitor will attempt to restart the job.
To stop job processing you can call (crank/stop monitor). Crank supports
having multiple monitors, there is no shared state.