cafebazaar / ecto-cassandra Goto Github PK
View Code? Open in Web Editor NEWCassandra Ecto Adapter
License: Other
Cassandra Ecto Adapter
License: Other
I am trying to create a collection for which a cql query looks like this.
create table groups (admin_id int, participants map<text, integer>);
My schema looks like this
defmodule HelloWorld.Repo.Migrations.CreateGroups do
use Ecto.Migration
def change do
create table(:groups) do
add :admin_id, :integer
add :participants, {:map,"<text,integer>"}
timestamps()
end
end
end
When I run mix ecto.migrate
, I'm getting
The following arguments were given to Ecto.Migration.validate_type!/1:
# 1
{:map, "<text,integer>"}
Attempted function clauses (showing 4 out of 4):
defp validate_type!(type) when is_atom(type)
defp validate_type!({type, subtype}) when is_atom(type) and is_atom(subtype)
defp validate_type!({type, subtype}) when is_atom(type) and is_tuple(subtype)
defp validate_type!(%Ecto.Migration.Reference{} = reference)
(ecto) lib/ecto/migration.ex:800: Ecto.Migration.validate_type!/1
(ecto) lib/ecto/migration.ex:617: Ecto.Migration.add/3
Any pointers will be appreciated..
reported in #3
I didn't see Changesets being used in the example code.
Is this a planned feature / already working?
What is the reason that this library has been at 1.0.0-rc3 for eight months? How stable is it?
I have a Mysql App. using Mariaex adapter. How can i use that with cassandra adapter ?
I get this error occasionally:
** (exit) an exception was raised:
** (MatchError) no match of right hand side value: :ok
(ecto) lib/ecto/repo/queryable.ex:130: Ecto.Repo.Queryable.execute/5
(ecto) lib/ecto/repo/queryable.ex:35: Ecto.Repo.Queryable.all/4
(ecto) lib/ecto/repo/queryable.ex:68: Ecto.Repo.Queryable.one/4
Tracing it back, lib/ecto/repo/queryable.ex:130
:
defp execute(operation, repo, adapter, query, opts) when is_list(opts) do
{meta, prepared, params} = Planner.query(query, operation, repo, adapter, 0)
case meta do
%{fields: nil} ->
adapter.execute(repo, meta, prepared, params, nil, opts)
%{select: select, fields: fields, prefix: prefix, take: take,
sources: sources, assocs: assocs, preloads: preloads} ->
preprocess = preprocess(prefix, sources, adapter)
{count, rows} = adapter.execute(repo, meta, prepared, params, preprocess, opts) # THIS LINE
postprocess = postprocess(select, fields, take)
{_, take_0} = Map.get(take, 0, {:any, %{}})
{count,
rows
|> Ecto.Repo.Assoc.query(assocs, sources)
|> Ecto.Repo.Preloader.query(repo, preloads, take_0, postprocess, opts)}
end
end
And knowing adapter is this library, adapter.ex
:
def execute(repo, %{fields: fields} = meta, query, params, process, options) do
[cql, options] = super(repo, meta, query, params, process, options)
case exec_and_log(repo, cql, options) do
%CQL.Result.Rows{rows_count: count, rows: rows} ->
{count, Enum.map(rows, &process_row(&1, fields, process))}
%CQL.Result.Void{} -> :ok # THIS LINE
error -> raise error
end
end
So, if result is %CQL.Result.Void{}
, returning :ok
throws a match error.
I can fix this, returning {0, []}
should be enough, yes?
I think this library is not compatible with Ecto 2.2.
I had to add {:ecto, "~> 2.1.0", override: true}
to my mix.exs
to get it to work again.
When I have Ecto 2.2
, I get this error:
** (exit) an exception was raised:
** (FunctionClauseError) no function clause matching in EctoCassandra.Adapter.execute/6
(ecto_cassandra) lib/ecto_cassandra/adapter.ex:100: EctoCassandra.Adapter.execute(BackendPlatform.CassandraRepo, %{prefix: nil, preloads: [], select: %{assocs: [], postprocess: {:from, :any, {:source, :from}}, preprocess: [{:source, {"stats_counters", BackendPlatform.Stats.Counter}, [date: :string, key: :string, value: :integer, revenue: :integer]}], take: []}, sources: {{"stats_counters", BackendPlatform.Stats.Counter}}}, {:nocache, {94389754, :all, "SELECT date, key, value, revenue FROM stats_counters WHERE date IN (?) AND key IN (?, ?)"}}, ["2017-09-11", "supplier:1:verified", "supplier:1:click"], #Function<0.61005659/1 in Ecto.Repo.Queryable.execute/5>, [])
(ecto) lib/ecto/repo/queryable.ex:133: Ecto.Repo.Queryable.execute/5
(ecto) lib/ecto/repo/queryable.ex:37: Ecto.Repo.Queryable.all/4
(Only last lines of error)
I am using latest version of ecto_cassandra
from Github.
I tried to run a migration on a project and this error came up:
18:53:33.972 [info] == Running Atmos.SensorRepo.Migrations.CreateSensorReadings.change/0 forward
18:53:33.972 [info] create table sensor_readings
** (CQL.Error) [invalid] Unknown type atmos_dev.serial: ""
lib/ecto_cassandra/adapter.ex:26: EctoCassandra.Adapter.execute_ddl/3
(ecto) lib/ecto/migration/runner.ex:98: anonymous fn/2 in Ecto.Migration.Runner.flush/0
(elixir) lib/enum.ex:1755: Enum."-reduce/3-lists^foldl/2-0-"/3
(ecto) lib/ecto/migration/runner.ex:96: Ecto.Migration.Runner.flush/0
(stdlib) timer.erl:181: :timer.tc/2
(ecto) lib/ecto/migration/runner.ex:27: Ecto.Migration.Runner.run/6
(ecto) lib/ecto/migrator.ex:127: Ecto.Migrator.attempt/6
(ecto) lib/ecto/migrator.ex:73: anonymous fn/4 in Ecto.Migrator.do_up/4
(ecto) lib/ecto/migrator.ex:250: anonymous fn/4 in Ecto.Migrator.migrate/4
(elixir) lib/enum.ex:1229: Enum."-map/2-lists^map/1-0-"/2
(ecto) lib/mix/tasks/ecto.migrate.ex:84: anonymous fn/4 in Mix.Tasks.Ecto.Migrate.run/2
(elixir) lib/enum.ex:645: Enum."-each/2-lists^foreach/1-0-"/2
(elixir) lib/enum.ex:645: Enum.each/2
(mix) lib/mix/task.ex:294: Mix.Task.run_task/3
(mix) lib/mix/cli.ex:58: Mix.CLI.run_task/2
(elixir) lib/code.ex:370: Code.require_file/2
The project uses two repos: Atmos.Repo
is the default postgres one, Atmos.SensorRepo
is the one running the EctoCassandra
adapter. Migrations on the former run smoothly. I tried checking if the Adapter
module is running the migration on the correct repo and it is.
Here's the migration:
defmodule Atmos.SensorRepo.Migrations.CreateSensorReadings do
use Ecto.Migration
@clustering_order "WITH CLUSTERING ORDER BY (timestamp DESC)"
def change do
create table(:sensor_readings, options: @clustering_order) do
add :team_id, :integer, partition_key: true
add :sensor_id, :integer, partition_key: true
add :type, :string
add :value, :float
add :timestamp, :timestamp, clustering_column: true
timestamps()
end
end
end
I don't know if it helps but I'm running macOS Sierra 10.12.3 and Elixir 1.4.2.
We are trying to replace Postgres adapter with this one, but simple drop-in did not work:
14:32:27.613 [error] Cassandra.Connection (#PID<0.220.0>) failed to connect: ** (Cassandra.ConnectionError) TCP Connect :timeout
14:32:27.613 [error] Cassandra.Connection (#PID<0.226.0>) failed to connect: ** (Cassandra.ConnectionError) TCP Connect :timeout
14:32:27.614 [error] Cassandra.Connection (#PID<0.222.0>) failed to connect: ** (Cassandra.ConnectionError) TCP Connect :timeout
14:32:27.614 [error] Cassandra.Connection (#PID<0.224.0>) failed to connect: ** (Cassandra.ConnectionError) TCP Connect :timeout
14:32:27.614 [error] Cassandra.Connection (#PID<0.225.0>) failed to connect: ** (Cassandra.ConnectionError) TCP Connect :timeout
14:32:27.614 [error] Cassandra.Connection (#PID<0.221.0>) failed to connect: ** (Cassandra.ConnectionError) TCP Connect :timeout
14:32:27.614 [error] Cassandra.Connection (#PID<0.223.0>) failed to connect: ** (Cassandra.ConnectionError) TCP Connect :timeout
14:32:27.614 [error] Cassandra.Connection (#PID<0.227.0>) failed to connect: ** (Cassandra.ConnectionError) TCP Connect :timeout
14:32:27.614 [error] Cassandra.Connection (#PID<0.219.0>) failed to connect: ** (Cassandra.ConnectionError) TCP Connect :timeout
14:32:27.614 [error] Cassandra.Connection (#PID<0.228.0>) failed to connect: ** (Cassandra.ConnectionError) TCP Connect :timeout
14:32:27.616 [error] Could not retrieve migrated versions. This error typically happens when the "schema_migrations" table, which Ecto uses for storing migrationinformation, is already used by another library or for other purposes.
You can fix this by running `mix ecto.drop` in the appropriate `MIX_ENV` to drop the existing database and let Ecto start a new one with a proper definition of "schema_migrations" or by configuring the repository to use another source:
config :annon_api, Annon.DB.Configs.Repo,
migration_source: "some_other_table_for_schema_migrations"
The full error is shown below.
** (Cassandra.ConnectionError) execute no connection
lib/ecto_cassandra/adapter.ex:26: EctoCassandra.Adapter.execute_ddl/3
(ecto) lib/ecto/migrator.ex:44: anonymous fn/2 in Ecto.Migrator.migrated_versions/2
(ecto) lib/ecto/migrator.ex:276: Ecto.Migrator.verbose_schema_migration/3
(ecto) lib/ecto/migrator.ex:148: Ecto.Migrator.run/4
(ecto) lib/mix/tasks/ecto.migrate.ex:84: anonymous fn/4 in Mix.Tasks.Ecto.Migrate.run/2
(elixir) lib/enum.ex:645: Enum."-each/2-lists^foreach/1-0-"/2
(elixir) lib/enum.ex:645: Enum.each/2
(mix) lib/mix/task.ex:294: Mix.Task.run_task/3
In tests i've used official docker Cassandra image, keyspaces were created successfully:
INFO [OptionalTasks:1] 2017-04-30 11:24:29,883 CassandraRoleManager.java:351 - Created default superuser role 'cassandra'
INFO [Native-Transport-Requests-1] 2017-04-30 11:24:35,208 MigrationManager.java:303 - Create new Keyspace: KeyspaceMetadata{name=annon_api_configs_test, params=KeyspaceParams{durable_writes=true, replication=ReplicationParams{class=org.apache.cassandra.locator.SimpleStrategy, replication_factor=1}}, tables=[], views=[], functions=[], types=[]}
INFO [Native-Transport-Requests-2] 2017-04-30 11:24:35,345 MigrationManager.java:303 - Create new Keyspace: KeyspaceMetadata{name=annon_api_logger_test, params=KeyspaceParams{durable_writes=true, replication=ReplicationParams{class=org.apache.cassandra.locator.SimpleStrategy, replication_factor=1}}, tables=[], views=[], functions=[], types=[]}
Other things I've noticed:
mix ecto.drop
does not delete keyspacesSQL.Sandbox
, which is bad if you want to run lots of tests in parallel:** (EXIT from #PID<0.70.0>) shutdown: failed to start child: Annon.DB.Configs.Repo.CassandraRepo
** (EXIT) shutdown: failed to start child: Cassandra.Session
** (EXIT) shutdown: failed to start child: Cassandra.Session.ConnectionManager
** (EXIT) an exception was raised:
** (UndefinedFunctionError) function Ecto.Adapters.SQL.Sandbox.start_link/2 is undefined or private
(ecto) Ecto.Adapters.SQL.Sandbox.start_link(Cassandra.Connection, [balancer: %Cassandra.LoadBalancing.TokenAware{wrapped: %Cassandra.LoadBalancing.RoundRobin{max_tries: 3, num_connections: 10}}, idle_timeout: 30000, queue: false, otp_app: :annon_api, repo: Annon.DB.Configs.Repo, adapter: EctoCassandra.Adapter, priv: "priv/repos/configs", contact_points: ["localhost"], replication: [class: "SimpleStrategy", replication_factor: 1], keyspace: "annon_api_configs_test", pool: Ecto.Adapters.SQL.Sandbox, cluster: Annon.DB.Configs.Repo.CassandraRepo.Cassandra.Cluster, session: Annon.DB.Configs.Repo.CassandraRepo.Cassandra.Session, cache: Annon.DB.Configs.Repo.CassandraRepo.Cassandra.Session.Cache, connection_manager: Annon.DB.Configs.Repo.CassandraRepo.Cassandra.Session.ConnectionManager, host: {172, 17, 0, 2}, pool_size: 10])
(cassandra) lib/cassandra/session/connection_manager.ex:114: Cassandra.Session.ConnectionManager.start_connection/3
(elixir) lib/enum.ex:1229: Enum."-map/2-lists^map/1-0-"/2
(cassandra) lib/cassandra/session/connection_manager.ex:101: Cassandra.Session.ConnectionManager.connect_to_up_hosts/1
(cassandra) lib/cassandra/session/connection_manager.ex:38: Cassandra.Session.ConnectionManager.init/1
(stdlib) gen_server.erl:328: :gen_server.init_it/6
(stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3
Issues can be reproduced by code from this PR.
There must be a way to run lots of tests in parallel as reported in #3, using SQL.Sandbox
is the preferred way to do this
I get this error:
exception exit: {#{'__exception__' => true,
'__struct__' => 'Elixir.Protocol.UndefinedError',
description => <<>>,protocol => 'Elixir.Enumerable',
value => nil},
[{'Elixir.Enumerable','impl_for!',1,
[{file,"lib/enum.ex"},{line,1}]},
{'Elixir.Enumerable',reduce,3,
[{file,"lib/enum.ex"},{line,116}]},
{lists,foreach,2,[{file,"lists.erl"},{line,1338}]},
{'Elixir.Stream',do_zip,3,
[{file,"lib/stream.ex"},{line,1077}]},
{'Elixir.Enum',zip,1,[{file,"lib/enum.ex"},{line,2732}]},
{'Elixir.EctoCassandra.Adapter',process_row,3,
[{file,"lib/ecto_cassandra/adapter.ex"},{line,8}]},
{'Elixir.Enum','-map/2-lists^map/1-0-',2,
[{file,"lib/enum.ex"},{line,1255}]},
{'Elixir.EctoCassandra.Adapter',execute,6,
[{file,"lib/ecto_cassandra/adapter.ex"},
{line,108}]}]}
I'm not sure why this happens, I don't even know if this happens because of this library, or because of queries, or Ecto, or something else.
Any help would be appreciated!
How can I set default consistency level for queries? I could not find options in docs for this repo, or cassandra driver.
How can I insert with TTL?
Rollbacks always timeout. This happens because Cassandra takes its time dropping tables and keysapces, and rollbacks or drops time out.
The problem happens after the first timeout, you can't run rollback again. Cassandra continued dropping table, but Elixir app thinks it does not and does not update schema_migrations
. So next time, table does not exist, and rollback fails. Migrations go into an invalid state. You can never rollback past this point, but you may continue migrating forward.
Although the problem is not critical for ecto.drop
, but it makes migrations invalid.
I searched to find how I can change the timeout value, and found nothing.
So the question is:
Is there a way (or a hack :) ) to change timeout when running migrations?
I don't know what happens exactly, but this is what I see in logs:
exception exit: {#{'__exception__' => true,
'__struct__' => 'Elixir.ArgumentError',
message =>
<<"raise/1 expects a module name, string or exception as the first argument, got:
%CQL.Result.Rows{column_types: [:varchar,
:varchar, :uuid, :double], columns: [\"date\", \"key\", \"value\", \"unit_cost\"],
columns_count: 4, paging_state: nil, rows: [], rows_count: 0}">>},
[{'Elixir.EctoCassandra.Adapter',exec,4,
[{file,"lib/ecto_cassandra/adapter.ex"},{line,163}]},
Last trace shows this function:
defp exec(repo, cql, options, on_conflict \\ :error) do
case exec_and_log(repo, cql, options) do
%CQL.Result.Void{} ->
{:ok, []}
%CQL.Result.Rows{rows_count: 1, rows: [[true | _]], columns: ["[applied]"|_]} ->
{:ok, []}
%CQL.Result.Rows{rows_count: 1, rows: [[false | _]], columns: ["[applied]"|_]} ->
if on_conflict == :nothing do
{:ok, []}
else
{:error, :stale}
end
error -> raise error
end
end
It happens on an insert query, so I changed Repo.insert()
to Repo.insert(on_conflict: :nothing)
. The error appears again.
Reading error, it seems somehow Cassandra sends 0 rows back (I'm OK with it, I don't need result), but the code is not handling it.
Am I right?
Version: {:ecto_cassandra, "~> 1.0.0-beta.3"}
, Ecto is 2.1.6
.
Cassandra is 3.11.0
.
Hi,
I am trying to get the set data type to work. I can successfully set it in the migration but can not seem to get it to work via the schema. The array data type fails with.
INSERT INTO chat_rooms (creator, participants, room_name, inserted_at, updated_at) VALUES (?, ?, ?, ?, ?) IF NOT EXISTS [1179, [1179, 15], "test room", ~N[2017-08-09 16:34:40.264752], ~N[2017-08-09 16:34:40.264763]]
[warn] Elixir.Cassandra.Session.Executor got error: %FunctionClauseError{arity: 1, function: :to_list, module: MapSet}
This is what I have in my schema for the set field. field :participants, {:array, :integer}
I am curious if there is something I can do to make sets work via the schema.
Thanks
I am trying to use the ecto-cassandra along with my postgres database.
However the current ecto version I am using is {:ecto, "~> 3.4.0"}
and the requirement for this project is 2.1.0.
Hence I am unable to use it due to dependencies issue.
Thanks in advance.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.