Now you may be wondering, how to implement the Mover.run/3 function with all these small Mover modules?
The simplest way is to return anonymous functions from these new functions.
The "concrete" movers are quite simple:
they just return an anonymous function that receives a source map and a destination map, then move the data from source to destination.
(and the data to be moved can be objects, files, or database records.)

defmodule FileMover do
  def new() do
    fn source, destination ->
      # move files from source to destination ...
    end
  end
end

What about the "abstract" movers that wraps around another mover?
Let's use CompareAfterwardsMover as an example:

defmodule CompareAfterwardsMover do
  def new(actual_mover, comparator) do
    fn source, destination ->
      with :ok <- Mover.run(actual_mover, source, destination),
           :same <- Comparator.run(comparator, source, destination) do
        :ok
      else
        :different -> {:error, :new_files_are_different}
      end
    end
  end
end

Note here the actual_mover is passed to Mover.run/3 to get called.
And the Comparator modules is also a Combinator module just like Mover.
Following the same pattern, we can implement RepoTransactionalMover, SequenceMover in a similar way:
Inside the anonymous functions they return, the mover to be wrapper would be passed to Mover.run/3 to do the real work.
And the "abstract" movers are just adding an extra non-functional layer upon it.

Finally, our Mover.run/3 function receives the anonymous function as mover, along side source and destination.
Then Mover.run/3 just delegates the work to mover by applying this anonymous function:

defmodule Mover do
  def run(mover, source, destination) do
    mover.(source, destination)
  end
end

From the examples above, we can see the power of this Combinator pattern:
By decomposing the work into smaller parts like CompareAfterwardsMover,
we composed a complex mover from a bunch of small and isolated mover's.
This process feels just like building Lego blocks.

With these tiny blocks, our Data Mover Combinator pattern meets our non-functional requirements perfectly:

1. We can easily switch between different methods to move files
   by implementing another AlternativeFileMover module and plugging this module in.
   If we want, we can easily implement a SplitterMover to randomly pick a method so we can compare different methods.
   
2. The comparator's are completely isolated from mover's.
   If I were writing this code before in a more procedural style, I would probably nest the comparing logic with the moving logic.
   This kind of nesting might make changing one of them harder since the other might be affected by the change.
   
3. Non-functional requirements like transactional operations can be encapsulated in separated mover's.
   So we can reuse this kind of non-functional logic easily,
   just like what we did with RepoTransactionalMover.
   

So what's making these Data Mover Combinators so powerful?
I think the main factor is that
these tiny blocks are completely isolated from each other, yet composable so we can build an extremely powerful data moving machine.

Isolated

Because each mover only needs to concern its own responsibilities, it can be very small and cohesive.
This isolation also brings us a nice side-effect: testability.
We can easily test the "concrete" mover's like FileMover since it's completely focusing on moving files.
We can also easily test the "abstract" mover's like CompareAfterwardsMover, or RepoTransactionalMover.

• CompareAfterwardsMoverTest
  

  defmodule CompareAfterwardsMoverTest do
    use ExUnit.Case, async: true
  
    test "returns :ok when both mover and comparator returns :ok"
  
    test "returns {:error, :new_files_are_different} when comparator returns :different"
  end
  

• RepoTransactionalMoverTest
  

  defmodule RepoTransactionalMoverTest do
    use TestSupport.DataCase, async: true
  
    test "returns result from actual_mover"
  
    test "reverts changes to repo when actual_mover returns an {:error, reason}"
  
    test "reverts changes to repo when actual_mover raises an exception"
  
    test "reverts changes to repo when actual_mover throws an error"
  
    test "reverts changes to multiple repos when transactions are nested"
  
    test "raises serialization_failure error when races with another SERIALIZABLE transaction"
  end
  

Composable (The Closure Property)

Mover's share the same API Mover.run/3.
So no matter it's a simple concrete mover like FileMover, or a complex composed mover like SequenceMover, as long as it can be accepted by Mover.run/3, it's a valid mover.
With this feature, the "depth" of Mover.run/3 is now dynamic:

• It can be infinitely deep (i.e. moves a lot of data behind the scenes) like what we do on production.
  
• It can also be extremely shallow (i.e. just return :ok) when we test an "abstract" mover, or try to understand a part of the system.
  

In another word, each mover can stay relatively simple, yet the whole system can be easily composed from these simple parts.

Most importantly, the final mover on production is built from ground up, layer by layer.
We can easily change the topology of these layers, to adjust the final behaviour with minimum changes.
For example, let's say the FileMover would take too much time to run.
We definitely don't want to wrap this step in repo transactions.
We can just change how we build the final mover to achieve this:

def run do
  [
    FileMover.new() |> CompareAfterwardsMover.new(MD5Comparator.new()),
    # Only wrap the following 2 steps in RepoTransactionalMover
    [
      Repo1Mover.new(),
      Repo2Mover.new()
    ]
    |> SequenceMover.new()
    |> RepoTransactionalMover.new(Repo1)
    |> RepoTransactionalMover.new(Repo2)
  ]
  |> SequenceMover.new()
  |> Mover.run(source, destination)
end

Actually I first started implementing "concrete" movers with Behaviour modules.
Because the callback function run(source, destination) was so obvious.
But with "abstract" movers like CompareAfterwardsMover, we need a more complex data structure:

defmodule CompareAfterwardsMover do
  def new(actual_mover, comparator) do
    [__MODULE__, actual_mover: actual_mover, comparator: comparator]
  end

  @impl Mover
  def run(source, destination, [actual_mover: actual_mover, comparator: comparator]) do
    with :ok <- Mover.run(actual_mover, source, destination),
         :same <- Comparator.run(comparator, source, destination) do
      :ok
    else
      :different -> {:error, :new_files_are_different}
    end
  end
end

defmodule Mover do
  @callback run(source(), destination(), keyword()) :: :ok | {:error, any()}

  def run([mod | opts], source, destination) do
    mod.run(source, destination, opts)
  end
end

• Pros
  
  • The contract is explicitly defined via the run/4 callback.
    
  • We can use Mox to define mocks.
    

  • We can do some meta analysis in Mover.run/3:
    

    defmodule Mover do
      def run([mod | opts], source, destination) do
        Logger.info("start mover #{mod}")
    
        # analyze how much time each mover takes with :time.tc ...
        mod.run(source, destination, opts)
      end
    end
    

• Cons
  
  • The callback run/4 needs an extra argument opts so that we can build a mover dynamically.
    
  • As you may see below, Behaviour is less flexible than Anonymous Function.
    Because a valid mover always needs to be a module.
    

I thought the Behaviour solution was a hack (attaching opts to a mod), and this hack required us to add an extra opts argument to our run callback function.
So I refactored the code to returned the anonymous functions just like above.
But the anonymous functions also have their Pros and Cons.

• Pros
  
  • Extremely flexible:
    CompareAfterwardsMover can be a module (with function new/2),
    but it can also be just a function Mover.compare_afterwards/2.
    

  • We can create mocks simply by defining an anonymous function in our test:
    

    fn :source, :destination -> :ok end
    

• Cons
  
  • The contract is implicitly defined as a convention.
    Meaning: the compiler won't help us detect an API violation if we decide to change the inputs/outputs convention
    (e.g. from (source, destination -> :ok | {:error | any()}) to (source, destination -> {:ok, any()} | {:error, any()}))
    This may cause some trouble in the long term.
    
  • Dialyzer cannot analyze anonymous functions deeply.
    
  • We cannot attach extra info in the anonymous functions.
    If we want to log when a mover gets called and its arguments, we may need to refactor to a data structure like the one with the callback modules (e.g. [fun, mover_name, opts]),
    which would take away much of the flexibility.
    

Finally, I was inspired by this great talk¹ by Quinn Wilton: Atlas Flexible Software Composition using Protocols
I experimented with using Protocol to implement the Combinator pattern:

defmodule CompareAfterwardsMover do
  defstruct [:actual_mover, :comparator]

  def new(actual_mover, comparator) do
    %__MODULE__{actual_mover: actual_mover, comparator: comparator}
  end

  defimpl MoverAlike do
    def run(%__MODULE__{actual_mover: actual_mover, comparator: comparator}, source, destination) do
      with :ok <- Mover.run(actual_mover, source, destination),
           :same <- Comparator.run(comparator, source, destination) do
        :ok
      else
        :different -> {:error, :new_files_are_different}
      end
    end
  end
end

defprotocol MoverAlike do
  def run(mover, source, destination)
end

defmodule Mover do
  defdelegate run(mover, source, destination), to: MoverAlike
end

• Pros
  
  • The contract is defined explicitly via a Protocol.
    
  • Unlike Behaviour's, we don't need an extra argument opts to store dynamic config.
    We can put these dynamic configs in the struct naturally.
    
  • We can also run meta analysis in Mover.run/3.
    
• Cons
  
  • There is no mock library for Protocols.
    (which is why I created one: dsdshcym/promox: Protocol-based mocks and explicit contracts in Elixir)
    
  • We cannot define extra functions under Mover module if it's a Protocol module.
    If we really want to do so, we may need to extract another Protocol module (MoverAlike).