Pipelinex is a simple library to build data pipelines in a clean and structured way.
It's mainly built for personal usage to help with structuring big data processing flows and automatically apply things like logging, which otherwise would make the code very unclean.
The package can be installed by adding pipelinex
to your list of dependencies in mix.exs
:
def deps do
[{:pipelinex, "~> 0.1.0"}]
end
Pipelinex provides two macros to build pipelines, task
and pipeline
.
Build simple pipelines using the task
macro:
defmodule MyPipeline do
use Pipelinex
task "add 5", num do
num + 5
end
task "add 10", num do
num + 10
end
end
Execute the pipeline via it's injected run/1
function:
result = MyPipeline.run(5)
result == 20
Pipelines can execute other pipelines using the pipeline
macro:
defmodule MyOtherPipeline do
use Pipelinex
pipeline MyPipeline
task "add 20", num do
num + 20
end
end
result = MyOtherPipeline.run(5)
result == 40
The task
macro supports pattern matching for its data argument.
defmodule MyPipeline do
use Pipelinex
task "add 5", %{:value => val} = map do
Map.put map, :value, val + 5
end
task "add 10", %{:value => val} = map do
Map.put map, :value, val + 10
end
end
If two or more tasks with the same name are defined one after another, the task will be called once and normal pattern matching behaviour will be applied.
defmodule MyPipeline do
use Pipelinex
task "add 5", %{:value => val} = map do
Map.put map, :value, val + 5
end
task "add 5", num do
num + 5
end
end
value = MyPipeline.run(%{:value => 5})
value == %{value: 10}
value = MyPipeline.run(5)
value == 10
Pipelinex logs when a task starts and ends using Logger.info
defmodule MyPipeline do
use Pipelinex
task "add 5", num do
num + 5
end
task "add 10", num do
num + 10
end
end
MyPipeline.run(5)
# 20
#
# 13:27:43.870 [info] Start: add 5
#
# 13:27:43.870 [info] End: add 5
#
# 13:27:43.870 [info] Start: add 10
#
# 13:27:43.870 [info] End: add 10