midas / phil_columns-ex

A data seeding framework for Elixir/Ecto.

PhilColumns

A full featured Elixir/Ecto seeding solution providing means for dev and prod seeding.

Installation

Add phil_columns to your list of dependencies in mix.exs:

def deps do
  [{:phil_columns, "~> 0.1.0"}]
end

Ensure phil_columns is started before your application:

def application do
  [applications: [:phil_columns]]
end

Create a Seed module for you application:

# lib/my_app/seed.ex

defmodule MyApp.Seed do
  defmacro __using__(_opts) do
    quote do
      use PhilColumns.Seed

      # shared code here ...
    end
  end
end

Configuration

If you need to ensure applications are started before seeding, configure them like this:

config :phil_columns,
  ensure_all_started: ~w(timex)a

Usage

Seeding Quick Start

Use the generator to create a seed.

$ mix phil_columns.gen.seed add_things

The generator puts a seed file in place. Add your seeding logic to the up/1 and/or down/1 functions using any valid Elixir/Ecto code.

# priv/repo/seeds/20160624153032_add_things.exs

defmodule MyApp.Repo.Seeds.AddThings do
  use MyApp.Seed

  def up(_repo) do
    # seeding logic ...
  end
end

Execute the seed(s).

$ mix phil_columns.seed

The Seed Command

The simplest usage of the seed command defaults the environment to dev and the version to all.

$ mix phil_columns.seed

The env can be overridden by providing a switch. The env is used to select only seeds that have been specified for the specified env.

$ mix phil_columns.seed --env=prod
$ mix phil_columns.seed -e prod

Tags and Environments

Tags and environments can be applied to seeds and filtered in command usage. The seed generator adds the dev environment by default and no tags. This feature enables efficiency and adaptability in development seeding and the possibility to use PhilColumns seeding in production (see Production Seeding section below).

Specifying environment(s) for a seed is accomplished with the envs function.

defmodule MyApp.Repo.Seeds.AddThings do
  use MyApp.Seed

  envs [:dev, :prod]
  # ...
end

To change the environment use the env switch. When not specified the env defaults to dev.

$ mix phil_columns.seed -e prod

Similary, applying tag(s) is accomplished using the tags function.

defmodule MyApp.Repo.Seeds.AddThings do
  use MyApp.Seed

  envs [:dev, :prod]
  tags [:some_tag]
  # ...
end

To change the tag(s) provide them after the command command line.

$ mix phil_columns.seed --tags=users,settings,etc
$ mix phil_columns.seed -t users,settings,etc

Production Seeding

Why?

Systems often have system level data that must be seeded when bootstraping a system or as new features are rolled out. Some examples are settings, configurations, roles, licenses, etc.

PhilColumns provides the ability to apply these system data seedings and commit them with features, analgous to an Ecto migration. Committing seed data with features or bug fixes communicates the intention of the data more clearly than any other strategy can.

How?

Create a module specifically for dealing with seeding in production.

defmodule MyApp.Deployment.Seeder do
  import Mix.Ecto
  import Mix.PhilColumns

  def seed(opts, seeder \\ &PhilColumns.Seeder.run/4) do
    repos = parse_repo(opts)
            |> List.wrap

    # set env with current_env/0 overwriting provided arg
    opts = Keyword.put( opts, :env, current_env )

    opts =
      if opts[:to] || opts[:step] || opts[:all],
        do: opts,
        else: Keyword.put(opts, :all, true)

    opts =
      if opts[:log],
        do: opts,
        else: Keyword.put(opts, :log, :info)

    opts =
      if opts[:quiet],
        do: Keyword.put(opts, :log, false),
        else: opts

    Enum.each(repos, fn repo ->
      exec_task(repo, opts, fn ->
        seeder.(repo, seeds_path(repo), :up, opts)
      end)
    end)
  end

  defp current_env do
    # implement this
    # warning: do not use Mix.env if you are doing an erlang release
  end
end

Use the module in the production app's remote console.

MyApp.Deployment.Seeder.seed(tags: ~w(things stuff)a)

phil_columns-ex's Issues

Future changes in code referenced from migrations.

Hi, I'm curious about how to handle code changes in phil_columns migrations. Your project reminds me of when people wrote Rails migrations that actually invoked some ruby logic (instead of doing plain db migrations). The problem with this is the code referenced by the migration might change in the future, and will render the migration's code invalid.

Suppose you write a migration that creates a couple of users:

def up do
   SomeLogic.create_user(:homer)
   SomeLogic.create_user(:bart)
end

Six months later if someone tries to remove or change the number of params for create_user. The old migration code will no longer work. One alternative would be to use a macro to capture
the code and only execute it if the migration needs to be run. However this only postpones the problem as you wouldnt notice the error until a new developer tries to run all the historic migrations.

up do # only compiled if the migration needs to be run
   SomeLogic.create_user(:homer)
   SomeLogic.create_user(:bart)
end

Another approach would be to recommend users to write all the logic in the migration itself, (even at the price of duplicating some logic) for keeping each migration as an immutable unit of code.

Another even wilder approach would be to make migrations git-aware (somehow save the commit) and checkout it somewhere just to execute that version of the code. Dont believe this could be a sane option but just thinking out loud.

Running seeds tasks in a release/production

Thanks for the package, it's already been really useful.

When we use mix release to deploy to production the resulting package no longer has mix itself available for running tasks, so the example in the README doesn't work if you use releases in production.

Taking inspiration from the examples in the README and in the Ecto.Migrator documentation at https://hexdocs.pm/ecto_sql/Ecto.Migrator.html#module-example-running-migrations-in-a-release I've created a seeder task like this:

defmodule MyApp.Deployment.Seeder do
  import Ecto.Migrator, only: [migrations_path: 2, with_repo: 2]

  @app :my_app

  def seed(opts \\ [], seeder \\ &PhilColumns.Seeder.run/4) do
    load_app()
    # set env with current_env/0 overwriting provided arg
    opts = Keyword.put(opts, :env, current_env())
    opts = Keyword.put(opts, :tags, [])

    opts =
      if opts[:to] || opts[:step] || opts[:all],
        do: opts,
        else: Keyword.put(opts, :all, true)

    opts =
      if opts[:log],
        do: opts,
        else: Keyword.put(opts, :log, :info)

    opts =
      if opts[:quiet],
        do: Keyword.put(opts, :log, false),
        else: opts

    for repo <- repos() do
      {:ok, _, _} = with_repo(repo, &seeder.(&1, migrations_path(&1, "seeds"), :up, opts))
    end
  end

  defp current_env do
    ## Add this to config/config.exs:
    ##
    ## config :my_app, env: config_env()
    Application.fetch_env!(@app, :env)
  end

  defp repos do
    Application.fetch_env!(@app, :ecto_repos)
  end

  defp load_app do
    Application.load(@app)
  end
end

which can be run from the release like this:

bin/my_app eval "MyApp.Deployment.Seeder.seed"

Does this sound like the right approach? If so, would this be a useful addition to the README? Happy to provide a PR if it's like to useful to someone.

Can you please bump the inflex library dependency to 2.x

Thanx. I can submit a PR if needed. Just starting to use this package, and I needed to fork it to accomodate another library (ex_twilio)

thanx