cafebazaar / ecto-cassandra Goto Github PK

View Code? Open in Web Editor NEW

88.0 88.0 13.0 284 KB

Cassandra Ecto Adapter

License: Other

Elixir 100.00%

cassandra cassandra-adapter ecto

ecto-cassandra's People

Contributors

Stargazers

Watchers

Forkers

erobit hisea frm vfsoraki artemeff solvedata jacktday cibingeorge kraglik sevenmind strogo okarifrankline

ecto-cassandra's Issues

How to use cassandra's map with ecto-cassandra?

I am trying to create a collection for which a cql query looks like this.

create table groups (admin_id int, participants map<text, integer>);

My schema looks like this

defmodule HelloWorld.Repo.Migrations.CreateGroups do
  use Ecto.Migration

  def change do
    create table(:groups) do
      add :admin_id, :integer
      add :participants, {:map,"<text,integer>"}

      timestamps()
    end

  end
end

When I run mix ecto.migrate, I'm getting


    The following arguments were given to Ecto.Migration.validate_type!/1:

        # 1
        {:map, "<text,integer>"}

    Attempted function clauses (showing 4 out of 4):

        defp validate_type!(type) when is_atom(type)
        defp validate_type!({type, subtype}) when is_atom(type) and is_atom(subtype)
        defp validate_type!({type, subtype}) when is_atom(type) and is_tuple(subtype)
        defp validate_type!(%Ecto.Migration.Reference{} = reference)

    (ecto) lib/ecto/migration.ex:800: Ecto.Migration.validate_type!/1
    (ecto) lib/ecto/migration.ex:617: Ecto.Migration.add/3

Any pointers will be appreciated..

Set default consistency level

How can I set default consistency level for queries? I could not find options in docs for this repo, or cassandra driver.

When will V1 be released?

What is the reason that this library has been at 1.0.0-rc3 for eight months? How stable is it?

Support for Ecto.Changeset constraints?

I didn't see Changesets being used in the example code.

Is this a planned feature / already working?

Migration error: Unknown type keyspace.serial: ""

I tried to run a migration on a project and this error came up:

18:53:33.972 [info]  == Running Atmos.SensorRepo.Migrations.CreateSensorReadings.change/0 forward

18:53:33.972 [info]  create table sensor_readings
** (CQL.Error) [invalid] Unknown type atmos_dev.serial: ""
    lib/ecto_cassandra/adapter.ex:26: EctoCassandra.Adapter.execute_ddl/3
    (ecto) lib/ecto/migration/runner.ex:98: anonymous fn/2 in Ecto.Migration.Runner.flush/0
    (elixir) lib/enum.ex:1755: Enum."-reduce/3-lists^foldl/2-0-"/3
    (ecto) lib/ecto/migration/runner.ex:96: Ecto.Migration.Runner.flush/0
    (stdlib) timer.erl:181: :timer.tc/2
    (ecto) lib/ecto/migration/runner.ex:27: Ecto.Migration.Runner.run/6
    (ecto) lib/ecto/migrator.ex:127: Ecto.Migrator.attempt/6
    (ecto) lib/ecto/migrator.ex:73: anonymous fn/4 in Ecto.Migrator.do_up/4
    (ecto) lib/ecto/migrator.ex:250: anonymous fn/4 in Ecto.Migrator.migrate/4
    (elixir) lib/enum.ex:1229: Enum."-map/2-lists^map/1-0-"/2
    (ecto) lib/mix/tasks/ecto.migrate.ex:84: anonymous fn/4 in Mix.Tasks.Ecto.Migrate.run/2
    (elixir) lib/enum.ex:645: Enum."-each/2-lists^foreach/1-0-"/2
    (elixir) lib/enum.ex:645: Enum.each/2
    (mix) lib/mix/task.ex:294: Mix.Task.run_task/3
    (mix) lib/mix/cli.ex:58: Mix.CLI.run_task/2
    (elixir) lib/code.ex:370: Code.require_file/2

The project uses two repos: Atmos.Repo is the default postgres one, Atmos.SensorRepo is the one running the EctoCassandra adapter. Migrations on the former run smoothly. I tried checking if the Adapter module is running the migration on the correct repo and it is.

Here's the migration:

defmodule Atmos.SensorRepo.Migrations.CreateSensorReadings do
  use Ecto.Migration

  @clustering_order "WITH CLUSTERING ORDER BY (timestamp DESC)"

  def change do
    create table(:sensor_readings, options: @clustering_order) do
      add :team_id, :integer, partition_key: true
      add :sensor_id, :integer, partition_key: true
      add :type, :string
      add :value, :float
      add :timestamp, :timestamp, clustering_column: true

      timestamps()
    end
  end
end

I don't know if it helps but I'm running macOS Sierra 10.12.3 and Elixir 1.4.2.

Ecto could not retrieve schema migrations data from Cassandra

We are trying to replace Postgres adapter with this one, but simple drop-in did not work:

14:32:27.613 [error] Cassandra.Connection (#PID<0.220.0>) failed to connect: ** (Cassandra.ConnectionError) TCP Connect :timeout

14:32:27.613 [error] Cassandra.Connection (#PID<0.226.0>) failed to connect: ** (Cassandra.ConnectionError) TCP Connect :timeout

14:32:27.614 [error] Cassandra.Connection (#PID<0.222.0>) failed to connect: ** (Cassandra.ConnectionError) TCP Connect :timeout

14:32:27.614 [error] Cassandra.Connection (#PID<0.224.0>) failed to connect: ** (Cassandra.ConnectionError) TCP Connect :timeout

14:32:27.614 [error] Cassandra.Connection (#PID<0.225.0>) failed to connect: ** (Cassandra.ConnectionError) TCP Connect :timeout

14:32:27.614 [error] Cassandra.Connection (#PID<0.221.0>) failed to connect: ** (Cassandra.ConnectionError) TCP Connect :timeout

14:32:27.614 [error] Cassandra.Connection (#PID<0.223.0>) failed to connect: ** (Cassandra.ConnectionError) TCP Connect :timeout

14:32:27.614 [error] Cassandra.Connection (#PID<0.227.0>) failed to connect: ** (Cassandra.ConnectionError) TCP Connect :timeout

14:32:27.614 [error] Cassandra.Connection (#PID<0.219.0>) failed to connect: ** (Cassandra.ConnectionError) TCP Connect :timeout

14:32:27.614 [error] Cassandra.Connection (#PID<0.228.0>) failed to connect: ** (Cassandra.ConnectionError) TCP Connect :timeout

14:32:27.616 [error] Could not retrieve migrated versions. This error typically happens when the "schema_migrations" table, which Ecto uses for storing migrationinformation, is already used by another library or for other purposes.

You can fix this by running `mix ecto.drop` in the appropriate `MIX_ENV` to drop the existing database and let Ecto start a new one with a proper definition of "schema_migrations" or by configuring the repository to use another source:

    config :annon_api, Annon.DB.Configs.Repo,
      migration_source: "some_other_table_for_schema_migrations"

The full error is shown below.

** (Cassandra.ConnectionError) execute no connection
    lib/ecto_cassandra/adapter.ex:26: EctoCassandra.Adapter.execute_ddl/3
    (ecto) lib/ecto/migrator.ex:44: anonymous fn/2 in Ecto.Migrator.migrated_versions/2
    (ecto) lib/ecto/migrator.ex:276: Ecto.Migrator.verbose_schema_migration/3
    (ecto) lib/ecto/migrator.ex:148: Ecto.Migrator.run/4
    (ecto) lib/mix/tasks/ecto.migrate.ex:84: anonymous fn/4 in Mix.Tasks.Ecto.Migrate.run/2
    (elixir) lib/enum.ex:645: Enum."-each/2-lists^foreach/1-0-"/2
    (elixir) lib/enum.ex:645: Enum.each/2
    (mix) lib/mix/task.ex:294: Mix.Task.run_task/3

In tests i've used official docker Cassandra image, keyspaces were created successfully:

INFO  [OptionalTasks:1] 2017-04-30 11:24:29,883 CassandraRoleManager.java:351 - Created default superuser role 'cassandra'
INFO  [Native-Transport-Requests-1] 2017-04-30 11:24:35,208 MigrationManager.java:303 - Create new Keyspace: KeyspaceMetadata{name=annon_api_configs_test, params=KeyspaceParams{durable_writes=true, replication=ReplicationParams{class=org.apache.cassandra.locator.SimpleStrategy, replication_factor=1}}, tables=[], views=[], functions=[], types=[]}
INFO  [Native-Transport-Requests-2] 2017-04-30 11:24:35,345 MigrationManager.java:303 - Create new Keyspace: KeyspaceMetadata{name=annon_api_logger_test, params=KeyspaceParams{durable_writes=true, replication=ReplicationParams{class=org.apache.cassandra.locator.SimpleStrategy, replication_factor=1}}, tables=[], views=[], functions=[], types=[]}

Other things I've noticed:

This adapter doesn't use Ecto integration tests suite.
mix ecto.drop does not delete keyspaces
This adapter does not support SQL.Sandbox, which is bad if you want to run lots of tests in parallel:

** (EXIT from #PID<0.70.0>) shutdown: failed to start child: Annon.DB.Configs.Repo.CassandraRepo
    ** (EXIT) shutdown: failed to start child: Cassandra.Session
        ** (EXIT) shutdown: failed to start child: Cassandra.Session.ConnectionManager
            ** (EXIT) an exception was raised:
                ** (UndefinedFunctionError) function Ecto.Adapters.SQL.Sandbox.start_link/2 is undefined or private
                    (ecto) Ecto.Adapters.SQL.Sandbox.start_link(Cassandra.Connection, [balancer: %Cassandra.LoadBalancing.TokenAware{wrapped: %Cassandra.LoadBalancing.RoundRobin{max_tries: 3, num_connections: 10}}, idle_timeout: 30000, queue: false, otp_app: :annon_api, repo: Annon.DB.Configs.Repo, adapter: EctoCassandra.Adapter, priv: "priv/repos/configs", contact_points: ["localhost"], replication: [class: "SimpleStrategy", replication_factor: 1], keyspace: "annon_api_configs_test", pool: Ecto.Adapters.SQL.Sandbox, cluster: Annon.DB.Configs.Repo.CassandraRepo.Cassandra.Cluster, session: Annon.DB.Configs.Repo.CassandraRepo.Cassandra.Session, cache: Annon.DB.Configs.Repo.CassandraRepo.Cassandra.Session.Cache, connection_manager: Annon.DB.Configs.Repo.CassandraRepo.Cassandra.Session.ConnectionManager, host: {172, 17, 0, 2}, pool_size: 10])
                    (cassandra) lib/cassandra/session/connection_manager.ex:114: Cassandra.Session.ConnectionManager.start_connection/3
                    (elixir) lib/enum.ex:1229: Enum."-map/2-lists^map/1-0-"/2
                    (cassandra) lib/cassandra/session/connection_manager.ex:101: Cassandra.Session.ConnectionManager.connect_to_up_hosts/1
                    (cassandra) lib/cassandra/session/connection_manager.ex:38: Cassandra.Session.ConnectionManager.init/1
                    (stdlib) gen_server.erl:328: :gen_server.init_it/6
                    (stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3

Issues can be reproduced by code from this PR.

Not compatible with ecto 2.2

I think this library is not compatible with Ecto 2.2.

I had to add {:ecto, "~> 2.1.0", override: true} to my mix.exs to get it to work again.

When I have Ecto 2.2, I get this error:

** (exit) an exception was raised:
    ** (FunctionClauseError) no function clause matching in EctoCassandra.Adapter.execute/6
        (ecto_cassandra) lib/ecto_cassandra/adapter.ex:100: EctoCassandra.Adapter.execute(BackendPlatform.CassandraRepo, %{prefix: nil, preloads: [], select: %{assocs: [], postprocess: {:from, :any, {:source, :from}}, preprocess: [{:source, {"stats_counters", BackendPlatform.Stats.Counter}, [date: :string, key: :string, value: :integer, revenue: :integer]}], take: []}, sources: {{"stats_counters", BackendPlatform.Stats.Counter}}}, {:nocache, {94389754, :all, "SELECT date, key, value, revenue FROM stats_counters WHERE date IN (?) AND key IN (?, ?)"}}, ["2017-09-11", "supplier:1:verified", "supplier:1:click"], #Function<0.61005659/1 in Ecto.Repo.Queryable.execute/5>, [])
        (ecto) lib/ecto/repo/queryable.ex:133: Ecto.Repo.Queryable.execute/5
        (ecto) lib/ecto/repo/queryable.ex:37: Ecto.Repo.Queryable.all/4

(Only last lines of error)

I am using latest version of ecto_cassandra from Github.

Drop keyspaces in ecto.drop task

reported in #3

Failing inserts?

I don't know what happens exactly, but this is what I see in logs:

    exception exit: {#{'__exception__' => true,
                       '__struct__' => 'Elixir.ArgumentError',
                       message =>
                           <<"raise/1 expects a module name, string or exception as the first argument, got: 
                                 %CQL.Result.Rows{column_types: [:varchar, 
                                   :varchar, :uuid, :double], columns: [\"date\", \"key\", \"value\", \"unit_cost\"],
                                   columns_count: 4, paging_state: nil, rows: [], rows_count: 0}">>},
                     [{'Elixir.EctoCassandra.Adapter',exec,4,
                          [{file,"lib/ecto_cassandra/adapter.ex"},{line,163}]},

Last trace shows this function:

  defp exec(repo, cql, options, on_conflict \\ :error) do
    case exec_and_log(repo, cql, options) do
      %CQL.Result.Void{} ->
        {:ok, []}
      %CQL.Result.Rows{rows_count: 1, rows: [[true | _]], columns: ["[applied]"|_]} ->
        {:ok, []}
      %CQL.Result.Rows{rows_count: 1, rows: [[false | _]], columns: ["[applied]"|_]} ->
        if on_conflict == :nothing do
          {:ok, []}
        else
          {:error, :stale}
        end
      error -> raise error
    end
  end

It happens on an insert query, so I changed Repo.insert() to Repo.insert(on_conflict: :nothing). The error appears again.

Reading error, it seems somehow Cassandra sends 0 rows back (I'm OK with it, I don't need result), but the code is not handling it.

Am I right?

Version: {:ecto_cassandra, "~> 1.0.0-beta.3"}, Ecto is 2.1.6.

Cassandra is 3.11.0.

Set Data type.

Hi,

I am trying to get the set data type to work. I can successfully set it in the migration but can not seem to get it to work via the schema. The array data type fails with.

INSERT INTO chat_rooms (creator, participants, room_name, inserted_at, updated_at) VALUES (?, ?, ?, ?, ?) IF NOT EXISTS [1179, [1179, 15], "test room", ~N[2017-08-09 16:34:40.264752], ~N[2017-08-09 16:34:40.264763]]
[warn] Elixir.Cassandra.Session.Executor got error: %FunctionClauseError{arity: 1, function: :to_list, module: MapSet}

This is what I have in my schema for the set field. field :participants, {:array, :integer} I am curious if there is something I can do to make sets work via the schema.

Thanks

Occasional errors

I get this error occasionally:

** (exit) an exception was raised:
    ** (MatchError) no match of right hand side value: :ok
        (ecto) lib/ecto/repo/queryable.ex:130: Ecto.Repo.Queryable.execute/5
        (ecto) lib/ecto/repo/queryable.ex:35: Ecto.Repo.Queryable.all/4
        (ecto) lib/ecto/repo/queryable.ex:68: Ecto.Repo.Queryable.one/4

Tracing it back, lib/ecto/repo/queryable.ex:130:

  defp execute(operation, repo, adapter, query, opts) when is_list(opts) do
    {meta, prepared, params} = Planner.query(query, operation, repo, adapter, 0)

    case meta do
      %{fields: nil} ->
        adapter.execute(repo, meta, prepared, params, nil, opts)
      %{select: select, fields: fields, prefix: prefix, take: take,
        sources: sources, assocs: assocs, preloads: preloads} ->
        preprocess    = preprocess(prefix, sources, adapter)
        {count, rows} = adapter.execute(repo, meta, prepared, params, preprocess, opts) # THIS LINE
        postprocess   = postprocess(select, fields, take)
        {_, take_0}   = Map.get(take, 0, {:any, %{}})
        {count,
          rows
          |> Ecto.Repo.Assoc.query(assocs, sources)
          |> Ecto.Repo.Preloader.query(repo, preloads, take_0, postprocess, opts)}
    end
  end

And knowing adapter is this library, adapter.ex:

  def execute(repo, %{fields: fields} = meta, query, params, process, options) do
    [cql, options] = super(repo, meta, query, params, process, options)

    case exec_and_log(repo, cql, options) do
      %CQL.Result.Rows{rows_count: count, rows: rows} ->
        {count, Enum.map(rows, &process_row(&1, fields, process))}
      %CQL.Result.Void{} -> :ok # THIS LINE
      error              -> raise error
    end
  end

So, if result is %CQL.Result.Void{}, returning :ok throws a match error.

I can fix this, returning {0, []} should be enough, yes?

Rollbacks always time out

Rollbacks always timeout. This happens because Cassandra takes its time dropping tables and keysapces, and rollbacks or drops time out.

The problem happens after the first timeout, you can't run rollback again. Cassandra continued dropping table, but Elixir app thinks it does not and does not update schema_migrations. So next time, table does not exist, and rollback fails. Migrations go into an invalid state. You can never rollback past this point, but you may continue migrating forward.

Although the problem is not critical for ecto.drop, but it makes migrations invalid.

I searched to find how I can change the timeout value, and found nothing.

So the question is:

Is there a way (or a hack :) ) to change timeout when running migrations?

Undefined Enumerable for nil, not sure why happens

I get this error:

    exception exit: {#{'__exception__' => true,
                       '__struct__' => 'Elixir.Protocol.UndefinedError',
                       description => <<>>,protocol => 'Elixir.Enumerable',
                       value => nil},
                     [{'Elixir.Enumerable','impl_for!',1,
                          [{file,"lib/enum.ex"},{line,1}]},
                      {'Elixir.Enumerable',reduce,3,
                          [{file,"lib/enum.ex"},{line,116}]},
                      {lists,foreach,2,[{file,"lists.erl"},{line,1338}]},
                      {'Elixir.Stream',do_zip,3,
                          [{file,"lib/stream.ex"},{line,1077}]},
                      {'Elixir.Enum',zip,1,[{file,"lib/enum.ex"},{line,2732}]},
                      {'Elixir.EctoCassandra.Adapter',process_row,3,
                          [{file,"lib/ecto_cassandra/adapter.ex"},{line,8}]},
                      {'Elixir.Enum','-map/2-lists^map/1-0-',2,
                          [{file,"lib/enum.ex"},{line,1255}]},
                      {'Elixir.EctoCassandra.Adapter',execute,6,
                          [{file,"lib/ecto_cassandra/adapter.ex"},
                           {line,108}]}]}

I'm not sure why this happens, I don't even know if this happens because of this library, or because of queries, or Ecto, or something else.

Any help would be appreciated!

Can we update the ecto version to 3+

I am trying to use the ecto-cassandra along with my postgres database.
However the current ecto version I am using is {:ecto, "~> 3.4.0"} and the requirement for this project is 2.1.0.

Hence I am unable to use it due to dependencies issue.

Thanks in advance.

cafebazaar / ecto-cassandra Goto Github PK

ecto-cassandra's People

Contributors

Stargazers

Watchers

Forkers

ecto-cassandra's Issues

Recommend Projects

Recommend Topics

Recommend Org

Jobs