GithubHelp home page GithubHelp logo

mongodb's Introduction

MongoDB

Build Status

Documentation for MongoDB is available online.

1.0.0 Released: March 22, 2023

There's one breaking change!

  • Mongo.find_one_and_replace/5, Mongo.find_one_and_update/5 now return {:ok, Mongo.FindAndModifyResult{:value, :matched_count, :upserted_id, :updated_existing}} instead of {:ok, doc} The change should be rather mechanical.

Features

  • Supports MongoDB versions 3.4, 3.6, 4.0, 4.2, 4.4, 5.0
  • Connection pooling (through db_connection)
  • Streaming cursors
  • Performant ObjectID generation
  • Follows driver specification set by 10gen
  • Safe (by default) and unsafe writes
  • Aggregation pipeline
  • Replica sets
  • Sessions and transactions

Tentative Roadmap

  • Use meta-driver test suite

Data representation

BSON Elixir
double 0.0
string "Elixir"
document [{"key", "value"}] | %{"key" => "value"} ¹
binary %BSON.Binary{binary: <<42, 43>>, subtype: :generic}
object id %BSON.ObjectId{value: <<...>>}
boolean true
UTC datetime %DateTime{}
null nil
regex %BSON.Regex{pattern: "..."}
JavaScript %BSON.JavaScript{code: "..."}
integer 42
symbol "foo" ²
min key :BSON_min
max key :BSON_max

¹ Since BSON documents are ordered Elixir maps cannot be used to fully represent them. This driver chose to accept both maps and lists of key-value pairs when encoding but will only decode documents to maps. This has the side-effect that the information about order of keys in a BSON document is lost when it's decoded. Additionally the driver will accept both atoms and strings for document keys but will only decode to strings.

² BSON symbols can only be decoded.

Writing your own encoding info

If you want to write a custom struct to your mongo collection - you can do that by implementing Mongo.Encoder protocol for your class. The output should be a map, which will be passed to the Mongo database.

Example:

defmodule CustomStruct do
  @fields [:a, :b, :c, :id]
  @enforce_keys @fields
  defstruct @fields

  defimpl Mongo.Encoder do
    def encode(%{a: a, b: b, id: id}) do
      %{
        _id: id,
        a: a,
        b: b,
        custom_encoded: true
      }
    end
  end
end

So, given the struct:

%CustomStruct{a: 10, b: 20, c: 30, id: "5ef27e73d2a57d358f812001"}

it will be written to database, as:

{
  "a": 10,
  "b": 20,
  "custom_encoded": true,
  "_id": "5ef27e73d2a57d358f812001"
}

Usage

Installation:

Add mongodb to your mix.exs deps.

defp deps do
  [
    {:mongodb, "~> 0.5.1"}
  ]
end

Then run mix deps.get to fetch dependencies.

Connection pooling

By default mongodb will start a single connection, but it also supports pooling with the :pool_size option.

# Starts an unpooled connection
{:ok, conn} = Mongo.start_link(url: "mongodb://localhost:27017/db-name")

# Gets an enumerable cursor for the results
cursor = Mongo.find(conn, "test-collection", %{})

cursor
|> Enum.to_list()
|> IO.inspect

If you're using pooling it is recommend to add it to your application supervisor:

def start(_type, _args) do
  children = [
    {Mongo, [name: :mongo, database: "test", pool_size: 2]}
  ]

  opts = [strategy: :one_for_one, name: MyApp.Supervisor]
  Supervisor.start_link(children, opts)
end

Simple start with pooling:

{:ok, conn} = Mongo.start_link(name: :mongo, database: "test", pool_size: 2)

Operate the mongodb with specify pool name in each query:

Mongo.find(:mongo, "collection", %{}, limit: 20)

More pool options in here.

Using with MongoDB Ecto

If you're using Mongo with the MongoDB Ecto library, where you have it defined in your config/runtime.exs like this:

config :my_app, MyApp.Repo,
  url: "mongo connection url"

You'll want to do reference mongo like this:

Mongo.find(MyApp.Repo.pool(), collection, %{_id: %{"$in" =>"some_ids"}})

Replica Sets

To connect to a MongoDB cluster that is using replica sets, it is recommended to use the :seeds list instead of a :hostname and :port pair.

{:ok, pid} = Mongo.start_link(database: "test", seeds: ["hostname1.net:27017", "hostname2.net:27017"])

This will allow for scenarios where the first "hostname1.net:27017" is unreachable for any reason and will automatically try to connect to each of the following entries in the list to connect to the cluster.

Auth mechanisms

For versions of MongoDB 3.0 and greater, the auth mechanism defaults to SCRAM. If you'd like to use MONGODB-X509 authentication, you can specify that as a start_link option.

{:ok, pid} = Mongo.start_link(database: "test", auth_mechanism: :x509)

AWS, TLS and Erlang SSL ciphers

Some MongoDB cloud providers (notably AWS) require a particular TLS cipher that isn't enabled by default in the Erlang SSL module. In order to connect to these services, you'll want to add this cipher to your ssl_opts:

{:ok, pid} = Mongo.start_link(database: "test",
      ssl_opts: [
        ciphers: ['AES256-GCM-SHA384'],
        cacertfile: "...",
        certfile: "...")
      ]
)

Examples

Using $and

Mongo.find(:mongo, "users", %{"$and" => [%{email: "[email protected]"}, %{first_name: "first_name"}]})

Using $or

Mongo.find(:mongo, "users", %{"$or" => [%{email: "[email protected]"}, %{first_name: "first_name"}]})

Using $in

Mongo.find(:mongo, "users", %{email: %{"$in" => ["[email protected]", "[email protected]"]}})

Contributing

The SSL test suite is enabled by default. You have two options. Either exclude the SSL tests or enable SSL on your MongoDB server.

Disable the SSL tests

mix test --exclude ssl

Enable SSL on your Mongo server

$ openssl req -newkey rsa:2048 -new -x509 -days 365 -nodes -out mongodb-cert.crt -keyout mongodb-cert.key
$ cat mongodb-cert.key mongodb-cert.crt > mongodb.pem
$ mongod --sslMode allowSSL --sslPEMKeyFile /path/to/mongodb.pem
  • For --sslMode you can use one of allowSSL or preferSSL
  • You can enable any other options you want when starting mongod

License

Copyright 2015 Justin Wood and Kobil Systems GmbH

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

mongodb's People

Contributors

ananthakumaran avatar ankhers avatar bogdanhabic avatar chakming avatar chowevtfy avatar deadtrickster avatar deepblue avatar elsifaka avatar ericmj avatar esse avatar evax avatar gianluca-nitti avatar gpad avatar hauleth avatar hbarisik avatar jocko avatar joeapearson avatar lau avatar michalmuskala avatar mindreframer avatar pchaussalet avatar rafaelm7o avatar rockerboo avatar sauvainr avatar scottmessinger avatar skinandbones avatar tomjoro avatar treye avatar tschmittni avatar zookzook avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

mongodb's Issues

Replica sets support

@ericmj I think that adding replica set support will really boost the elixir / phoenix community.
Using mongodb in production without replica set it's not really possible. My company has a project that I really would like to migrate to phoenix and it's impossible without replica set support. I think there is a lot of developers in the same position. If you need help I can try and add the replica set support myself. If you already working on it I would like to help anyway I can.

How to create a BSON DateTime struct?

Hi there, I'm an elixir noob trying to insert a date into mongo, but I can't figure out to to create the BSON.DateTime object properly. I'm currently doing this:

utc = :calendar.datetime_to_gregorian_seconds(:calendar.universal_time)
%BSON.DateTime{utc: get_utc_timestamp}

When it saves to mongo, it shows up as 1972-01-07....

What's the correct way to create a DateTime for now?

How to connect to MongoDB 3.2?

Hello, I'm trying to connect a MongoDB version 3.2 but it seems it isn't using SCRAM authentication, how can I specify that in my connection options?

{:ok, _} = MongoPool.start_link(
            hostname: server,
            username: username,
            password: password,
            database: dbname
        )
** (Mix) Could not start application ejobs: exited in: JobsMonitor.Start.start(:normal, [])
    ** (EXIT) exited in: GenServer.call(#PID<0.149.0>, {:find, "bz_administrators", %{}, nil, [batch_size: 1000]}, 5000)
        ** (EXIT) %Mongo.Error{code: 18, message: "auth failed for 'admin': Authentication failed."}

Values passed as connection parameters are ok

Stream.resource from cursor/Mongo.find

I'm sure this is due to my inexperience, but I'm trying to find an example of setting up a stream from a cursor.

I can see a number flaws in my approach and figure there is likely a better way to do this.

def doc_stream(some_criteria) do
  Stream.resource(
    fn -> iteration end,
    fn(iteration) ->
      IO.puts "iteration: #{iteration}"
      data = Mongo.find(SomePool, "collection_name", %{"field" => some_criteria, limit: 1, skip: iteration)
      if Enum.empty?(data) do
        {:halt, iteration}
      else
        {[Enum.at(data, 0)], iteration + 1}
      end
    end,
    fn(iteration) -> iteration end
  )
end
doc_stream |> Enum.take(10)

My goal is to not put a ton of pressure on the mongo servers when trying to work through massive collections in batches. I'd like to utilize streams and async processes to work on small chunks and only pull the next n-number of documents when its time to use them.

My issue with what I have is that if I take 10, its going to fire 10 queries at the database I assume. That is probably not ideal.

But before I sink more time into this and reinvent the wheel, I figured one of you may have a better approach or maybe this is silly altogether?

Mongo.delete_* naming

Was writing some more documentation and noticed naming of "remove" from Mongo is named as "delete_*". The main Mongo API uses remove(), so I was looking for that in the code. Connection is using remove though, so not sure why its like this. Could just be something that's going to change later.

Summary: Why is Mongo using delete_* instead of remove?

Using with a database connection string

I want to use this on Heroku where I am provided a full connection string as an environment variable. Is there any way to connect with a string in the following format?

mongodb://heroku_****:****@***.mlab.com:17545,***.mlab.com:17545/heroku_*****?replicaSet=rs-****

After insert_one BSON Object is Returned

After I insert some data I get this:

%Mongo.InsertOneResult{inserted_id: #BSON.ObjectId<57dc602dfa2d310e03f95be8>}

That's great, but I have no idea what to do with this BSON.Objectid I would like to get back maybe a map of all the data I just inserted or the Mongo ID so I can look up the record I just inserted.

Can anyone help me figure out how to do this?

Make connection error (:closed) a Mongo.Error

If mongo is inaccessible (offline), an attempt to use any Mongo functions results in an exception.

The Connection is returning a {:error, :closed} upstream, but the local function handler tries to raise all errors of the pattern {:error, e} causing the exception.

The Connection error should be converted to some kind of %Mongo.Error{} instead, so the raise function consistently reports these exceptions.

Authentication Database support

Hi @ericmj. Thanks for this. I was wondering if you would be interested in a PR to support authentication via an Authentication Database? If so, would master be the correct branch to submit the PR to? I've noticed some tests are failing on master and that there is another branch that seems to be active. Thanks :)

BSON.ObjectId.decode!/1 is undefined or private

hi, is there a way to make BSON.ObjectId.decode!/1 available?, I'm passing a string as parameter

15:01:10.138 [error] #PID<0.287.0> running Estorage.Router terminated
Server: localhost:4001 (http)
Request: POST /uploadfile
** (exit) an exception was raised:
    ** (UndefinedFunctionError) function BSON.ObjectId.decode!/1 is undefined or private
        (mongodb) BSON.ObjectId.decode!("579a2daf189fe3491028a3c6")
        (estorage) lib/estorage/users.ex:7: Estorage.Users.getById/1
        (estorage) lib/estorage/manager.ex:8: Estorage.Manager.upload_file/5
        (estorage) lib/estorage/router.ex:18: anonymous fn/1 in Estorage.Router.do_match/4
        (estorage) lib/estorage/router.ex:2: Estorage.Router.plug_builder_call/2
        (plug) lib/plug/adapters/cowboy/handler.ex:15: Plug.Adapters.Cowboy.Handler.upgrade/4
        (cowboy) src/cowboy_protocol.erl:442: :cowboy_protocol.execute/4

Easy way to convert BSON.ObjectId to String?

I'm sure I may be missing something here, but after an insert I get back a BSON.ObjectId like #BSON.ObjectId<1e0b239999a1014ed1b35a84>.

What is the easiest way to get a string representation of the actual _id for later use?

I've tried something like %BSON.ObjectId{value: object_id} = #BSON.ObjectId<1e0b239999a1014ed1b35a84>, but then I have a binary.

Feels like I'm missing something real simple here. Thanks!

JSON encoding

Would be useful to encode to JSON, e.g. map of strings:

BSON.encode_json(doc)
%{"_id" => "559c3c676b92c9bf5c0898fd", "name" => "Travis"}

I'm pretty new to Elixir and this is above my level still, but when I can do it I'll make a PR if there isn't already.

Setting a field equal to a function evaluation?

Say I have an incremental id, in elixir i set "_id"=> "incrId()", "timest"=>"db.eval("Date()")"

            Mongo.insert_one(MongoPool, "test", %{
                    "_id"=> "incrId()",
                    "timest"=> "db.eval(\"Date()\")"
                })

This makes the record saved without evaluating the function.Neither way works. Using the commandline for mongo this behavior does not happen as the functions get evaluated as they are inserted.

Add ability to specify database on the use Mongo.Pool options

Would like the ability to also define the database when defining the pool definition itself, instead of specifying the database on start_link. Since the pool is directly tied to the database, this keeps it cleaner.

The existing approach is:

defmodule DatabasePool do
use Mongo.Pool, name: MODULE, adapter: Mongo.Pool.Poolboy
end

but then elsewhere having to perform a DatabasePool.start_link(database: "my_database").

Would also like the ability to do this instead...

defmodule DatabasePool do
use Mongo.Pool, name: MODULE, adapter: Mongo.Pool.Poolboy, database: "my_database"
end

DatabasePool.start_link

Can't sort on sub fields

Sort is made with atoms, and can't do this with them. I think they should remain as a string keys.

Mongo.find(MongoPool, %{}, [sort: [game.name: -1]])

Thanks 😃

Correct endianness for all datatypes

BSON encoding/decoding of values does not pay attention to the endianness of the storage when doing binary pattern matching. All types which have a possible endianness should be serialized to little endian per the BSON spec.

Add collection drop and handle case exceptions

Add a .drop() function to drop a collection.

We performed it manually using a run_command but if the collection does not exist, an uncaught exception occurs...

** (CaseClauseError) no case clause matching: %{"errmsg" => "ns not found", "ok" => 0.0}

As a guidepost, within our wrapper lib, we're handling like this presently..

def drop_collection(collection) do
try do
%{"ok" => 1.0} = Mongo.run_command(@pool, %{:drop => collection})
:ok
rescue
e in CaseClauseError ->
Logger.debug "#{inspect e}"
:error
end
end

insert_one failing

When I follow the instructions on the Readme I get an error doing an insert:

iex(4)> Mongo.insert_one(MongoPool, "crawl_report", %{"field" => 10})
** (ArgumentError) argument error
(stdlib) :ets.lookup_element(Mongo.IdServer, :machineprocid, 2)
lib/mongo/id_server.ex:52: Mongo.IdServer.new/0
lib/mongo/connection.ex:134: Mongo.Connection.add_id/1
lib/mongo/connection.ex:102: Mongo.Connection.assign_ids/1
lib/mongo/connection.ex:79: Mongo.Connection.insert/4
(stdlib) timer.erl:181: :timer.tc/2
lib/mongo/pool/poolboy.ex:36: Mongo.Pool.Poolboy.run/2
lib/mongo/pool.ex:142: Mongo.Pool.run_with_log/5

However if I manually run Mongo.IdServer.start_link the insert works:

iex(3)> Mongo.insert_one(MongoPool, "crawl_report", %{"field" => 10})
** (ArgumentError) argument error
(stdlib) :ets.lookup_element(Mongo.IdServer, :machineprocid, 2)
lib/mongo/id_server.ex:52: Mongo.IdServer.new/0
lib/mongo/connection.ex:134: Mongo.Connection.add_id/1
lib/mongo/connection.ex:102: Mongo.Connection.assign_ids/1
lib/mongo/connection.ex:79: Mongo.Connection.insert/4
(stdlib) timer.erl:181: :timer.tc/2
lib/mongo/pool/poolboy.ex:36: Mongo.Pool.Poolboy.run/2
lib/mongo/pool.ex:142: Mongo.Pool.run_with_log/5
iex(3)> Mongo.IdServer.start_link
{:ok, #PID<0.203.0>}
iex(4)> Mongo.insert_one(MongoPool, "crawl_report", %{"field" => 10})
{:ok,
%Mongo.InsertOneResult{inserted_id: #BSON.ObjectId<1dfbc07783152906278464a1>}}

Is this expected behavior or is something wrong?

Query by ObjectID hexadecimal string

How can I query for document based on ObjectID as string? The only way I could get it to work is
to use Mongo.Ecto.ObjectID.dump from Mongo.Ecto project, like this:

Mongo.find(MongoPool, "some_collection", %{_id: elem(Mongo.Ecto.ObjectID.dump("55e724bf44f4173c876919f9"),1) })

is there a way to do it and not depend on an external project?

Is there anyway I can specify a clean connection without a database setting?

Currently I'm migrating a lib which using elixir-mongo as dependency which it doesn't require a database when creating a connection(which should be much more nature as well). The reason why I'm doing so is because elixir-mongo did purge all values's key into atom which might cause a huge memory leakage if it is running in a long term,and looks like their project is unmaintained for a while as well.

Everything looks fine for mongoldb as well, but I can't find anyway to create a clean connection to the database both with Mongo.Connection or MongoPool, is there any way I can circumvent with this restricts.

And also I did notice all find/update or db actions are taking collection as the first argument, so from my understanding this is a designed pattern, but indeed I need to maintain a single connection and talks to multiple db in the same time, is there anyway I can use it except for fork a new version?

Protocol Error

This is for the emj-dbconn branch

I believe my recent PR (#74) broke something. When trying to do simple queries, I am getting a protocol error.

However, if I execute the commands fast enough (copy / paste into iex) the query works properly.

Let me know if there is anything else I can do to debug this.

12:44:41 (emj-dbconn) mongodb$ iex -S mix
Erlang/OTP 19 [erts-8.0] [source] [64-bit] [smp:8:8] [async-threads:10] [hipe] [kernel-poll:false]

Interactive Elixir (1.3.0) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> {:ok, pid} = Mongo.start_link(database: "some_database")
{:ok, #PID<0.154.0>}
iex(2)> c = Mongo.find(pid, "users", %{})
%Mongo.Cursor{coll: "users", conn: #PID<0.154.0>, opts: [], query: %{},
 select: nil}
iex(3)> Enum.to_list(c)
** (Mongo.Error) tcp recv: invalid argument - :einval

12:45:27.973 [info]  Mongo.Protocol (#PID<0.154.0>) missed message: {:tcp, #Port<0.5043>, <<167, 25, 7, 0, 1, 245, 4, 0, 0, 0, 0, 0, 1, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 143, 1, 0, 0, 50, 24, 0, 0, 7, 95, 105, 100, 0, 82, 129, 95, ...>>}

12:45:27.973 [error] Mongo.Protocol (#PID<0.154.0>) disconnected: ** (Mongo.Error) tcp recv: invalid argument - :einval
    (mongodb) lib/mongo/cursor.ex:40: anonymous fn/6 in Enumerable.Mongo.Cursor.start_fun/6
     (elixir) lib/stream.ex:1121: anonymous fn/5 in Stream.resource/3
     (elixir) lib/enum.ex:1626: Enum.reduce/3
     (elixir) lib/enum.ex:2345: Enum.to_list/1
** (EXIT from #PID<0.152.0>) an exception was raised:
    ** (RuntimeError) disconnect/2 not implemented
        (mongodb) lib/db_connection.ex:240: Mongo.Protocol.disconnect/2
        (db_connection) lib/db_connection/connection.ex:178: DBConnection.Connection.disconnect/2
        (connection) lib/connection.ex:767: Connection.disconnect/3
        (stdlib) gen_server.erl:601: :gen_server.try_dispatch/4
        (stdlib) gen_server.erl:667: :gen_server.handle_msg/5
        (stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3

Interactive Elixir (1.3.0) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)>
12:45:27.978 [error] GenServer #PID<0.154.0> terminating
** (RuntimeError) disconnect/2 not implemented
    (mongodb) lib/db_connection.ex:240: Mongo.Protocol.disconnect/2
    (db_connection) lib/db_connection/connection.ex:178: DBConnection.Connection.disconnect/2
    (connection) lib/connection.ex:767: Connection.disconnect/3
    (stdlib) gen_server.erl:601: :gen_server.try_dispatch/4
    (stdlib) gen_server.erl:667: :gen_server.handle_msg/5
    (stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3
Last message: {:"$gen_cast", {:disconnect, #Reference<0.0.6.118>, %Mongo.Error{code: nil, message: "tcp recv: invalid argument - :einval"}, %{database: "some_database", request_id: 0, socket: #Port<0.5043>, timeout: 5000, wire_version: 4, write_concern: %{w: 1}}}}
State: {Mongo.Protocol, %{database: "some_database", request_id: 0, socket: #Port<0.5043>, timeout: 5000, wire_version: 4, write_concern: %{w: 1}}}

Move BSON to a separate lib

Hi,

I want to suggest you to move BSON lib to a separate HEX package, since your implementation works much (more than 50%) faster and predictable than bson package that have 10k downloads right now.

At the moment I'm used to copy-paste your code and make some cleanup, since making DB driver as dependency to parse BSON would be an ugly solution.

Mongo.Pool

I saw a previous issue that was the same as mine but the solution for them did not work for me (adding a name parameter to start_link). This is my first time using the library (version 2.0.0) and I followed the instructions in the README. I've got a MongoPool module:

defmodule MongoPool do
  use Mongo.Pool, name: __MODULE__, adapter: Mongo.Pool.Poolboy
end

and when the app tries to compile I get the following error:

** (CompileError) lib/intake_zen/mongo_pool.ex:2: module Mongo.Pool is not loaded and could not be found
    (elixir) expanding macro: Kernel.use/2
    lib/intake_zen/mongo_pool.ex:2: MongoPool (module)
    (elixir) lib/kernel/parallel_compiler.ex:116: anonymous fn/4 in Kernel.ParallelCompiler.spawn_compilers/1

I searched through the driver repo and can't find Mongo.Pool anywhere. Is this module no longer available?

Monitoring tailable cursor events

So, if you create a tailable cursor, you can use the cursor as a Stream; however, the functionality that a tailable cursor adds (allowing you to see newly inserted documents for a cursor as they are added) doesn't really support streams. We'd need some way to have a supervised process that sends new results to another process as they are available. Here's the code for how RethinkDB handles this:

https://github.com/hamiltop/rethinkdb_changefeed

Unable to query oplog

I'm getting an error running queries against the oplog on a MongoDB replica set. MongoDB v 2.6.5 / Elixir 1.0.5.

An iex session that illustrates this:

-> % iex -S mix
Erlang/OTP 18 [erts-7.0.2] [source] [64-bit] [smp:4:4] [async-threads:10] [hipe] [kernel-poll:false] [dtrace]

Interactive Elixir (1.0.5) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> defmodule MongoPool do use Mongo.Pool, name: __MODULE__, adapter: Mongo.Pool.Poolboy end                        {:module, MongoPool,
 <<70, 79, 82, 49, 0, 0, 9, 80, 66, 69, 65, 77, 69, 120, 68, 99, 0, 0, 1, 111, 131, 104, 2, 100, 0, 14, 101, 108, 105, 120, 105, 114, 95, 100, 111, 99, 115, 95, 118, 49, 108, 0, 0, 0, 2, 104, 2, ...>>,
 :ok}
iex(2)> {:ok, _} = MongoPool.start_link(database: "local")
{:ok, #PID<0.134.0>}
iex(3)> Mongo.find(MongoPool, "oplog.rs", %{}) |> Enum.take(1)
** (exit) exited in: GenServer.call(#PID<0.150.0>, {:find, "oplog.rs", %{}, nil, [batch_size: 1000]}, 5000)
    ** (EXIT) an exception was raised:
        ** (FunctionClauseError) no function clause matching in Mongo.Connection.handle_info/2
            lib/mongo/connection.ex:346: Mongo.Connection.handle_info({:tcp, #Port<0.5509>, <<0, 3, 0, 0, 0, 82, 85, 0, 2, 49, 56, 48, 0, 3, 0, 0, 0, 82, 87, 0, 2, 49, 56, 49, 0, 3, 0, 0, 0, 82, 69, 0, 2, 49, 56, 50, 0, 3, 0, 0, 0, 66, 76, 0, 2, 49, 56, 51, ...>>}, {:ok, %{auth: [], database: "local", opts: [backoff: 1000, port: 27017, hostname: 'localhost'], queue: %{0 => %{command: :find, from: {#PID<0.127.0>, #Reference<0.0.3.911>}, params: nil}}, request_id: 1, socket: #Port<0.5509>, tail: <<84, 120, 47, 0, 85, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 8, 0, 0, 0, 138, 133, 8, 97, 208, 0, 0, 0, 0, 0, 0, 0, 232, 3, 0, 0, 242, 11, 0, 0, 17, 116, ...>>, timeout: 5000, wire_version: 2, write_concern: [w: 1]}})
            lib/connection.ex:798: Connection.handle_async/3
            (stdlib) gen_server.erl:615: :gen_server.try_dispatch/4
            (stdlib) gen_server.erl:681: :gen_server.handle_msg/5
            (stdlib) proc_lib.erl:239: :proc_lib.init_p_do_apply/3

18:55:47.367 [error] GenServer #PID<0.150.0> terminating
Last message: {:tcp, #Port<0.5509>, <<0, 3, 0, 0, 0, 82, 85, 0, 2, 49, 56, 48, 0, 3, 0, 0, 0, 82, 87, 0, 2, 49, 56, 49, 0, 3, 0, 0, 0, 82, 69, 0, 2, 49, 56, 50, 0, 3, 0, 0, 0, 66, 76, 0, 2, 49, 56, 51, ...>>}
State: {:ok, %{auth: [], database: "local", opts: [backoff: 1000, port: 27017, hostname: 'localhost'], queue: %{0 => %{command: :find, from: {#PID<0.127.0>, #Reference<0.0.3.911>}, params: nil}}, request_id: 1, socket: #Port<0.5509>, tail: <<84, 120, 47, 0, 85, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 8, 0, 0, 0, 138, 133, 8, 97, 208, 0, 0, 0, 0, 0, 0, 0, 232, 3, 0, 0, 242, 11, 0, 0, 17, 116, ...>>, timeout: 5000, wire_version: 2, write_concern: [w: 1]}}
** (exit) an exception was raised:
    ** (FunctionClauseError) no function clause matching in Mongo.Connection.handle_info/2
        lib/mongo/connection.ex:346: Mongo.Connection.handle_info({:tcp, #Port<0.5509>, <<0, 3, 0, 0, 0, 82, 85, 0, 2, 49, 56, 48, 0, 3, 0, 0, 0, 82, 87, 0, 2, 49, 56, 49, 0, 3, 0, 0, 0, 82, 69, 0, 2, 49, 56, 50, 0, 3, 0, 0, 0, 66, 76, 0, 2, 49, 56, 51, ...>>}, {:ok, %{auth: [], database: "local", opts: [backoff: 1000, port: 27017, hostname: 'localhost'], queue: %{0 => %{command: :find, from: {#PID<0.127.0>, #Reference<0.0.3.911>}, params: nil}}, request_id: 1, socket: #Port<0.5509>, tail: <<84, 120, 47, 0, 85, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 8, 0, 0, 0, 138, 133, 8, 97, 208, 0, 0, 0, 0, 0, 0, 0, 232, 3, 0, 0, 242, 11, 0, 0, 17, 116, ...>>, timeout: 5000, wire_version: 2, write_concern: [w: 1]}})
        lib/connection.ex:798: Connection.handle_async/3
        (stdlib) gen_server.erl:615: :gen_server.try_dispatch/4
        (stdlib) gen_server.erl:681: :gen_server.handle_msg/5
        (stdlib) proc_lib.erl:239: :proc_lib.init_p_do_apply/3
    (elixir) lib/gen_server.ex:356: GenServer.call/3
    (stdlib) timer.erl:181: :timer.tc/2
             lib/mongo/pool/poolboy.ex:34: Mongo.Pool.Poolboy.run/2
             lib/mongo/pool.ex:140: Mongo.Pool.run_with_log/5
             lib/mongo/cursor.ex:33: anonymous fn/6 in Enumerable.Mongo.Cursor.start_fun/6
    (elixir) lib/stream.ex:1014: anonymous fn/5 in Stream.resource/3
    (elixir) lib/enum.ex:1740: Enum.take/2

Add database option support for CRUD functions

distinct/find/insert_one/insert_many/delete_one/delete_many/replace_one/update_one/update_many/save_one/save_many including MongoCursors should support :database option to specify a database to talk with.

Mongo.Pool

Is pool still available? Can't seem to use it with v2.1.. And if it's not used anymore, what is the alternative?

Error using Poolboy

Hi!
I'm trying to use the package in a project that I'm working on, but if I try to use DBConnection.Poolboy as specified in the instructions, I get the following error:

iex(11)> Mongo.find(:mongo, "products", %{}, limit: 20, pool: DBConnection.Poolboy) |> Enum.to_list
** (UndefinedFunctionError) function :invalid_message.exception/1 is undefined (module :invalid_message is not available)
                    :invalid_message.exception([])
    (db_connection) lib/db_connection.ex:925: DBConnection.checkout/2
    (db_connection) lib/db_connection.ex:741: DBConnection.run/3
    (db_connection) lib/db_connection.ex:1132: DBConnection.run_meter/3
    (db_connection) lib/db_connection.ex:636: DBConnection.execute/4
          (mongodb) lib/mongo.ex:386: Mongo.kill_cursors/3
           (elixir) lib/stream.ex:1129: Stream.do_resource/5
           (elixir) lib/enum.ex:1627: Enum.reduce/3
           (elixir) lib/enum.ex:2346: Enum.to_list/1

These are my deps:

* db_connection 1.1.0 (Hex package) (mix)
  locked at 1.1.0 (db_connection) b2b88db6
  ok
* mongodb 0.2.0 (Hex package) (mix)
  locked at 0.2.0 (mongodb) 95f85074
  ok
* poolboy 1.5.1 (Hex package) (rebar)
  locked at 1.5.1 (poolboy) 6b461639
  ok

And this is how I have added the Mongo worker to the main supervision tree:

defp children do
    [
        ...
        worker(Mongo, [mongo_config]),
        ...
    ]
  end

defp mongo_config do
    [
      name:     :mongo,
      pool:     DBConnection.Poolboy,
      username: System.get_env("MONGODB_USERNAME"),
      password: System.get_env("MONGODB_PASSWORD"),
      database: System.get_env("MONGODB_DBNAME"),
    ]
  end

The thing is that if I remove the pool from the config, and run the same query without the pool, it actually works. What might be going wrong?
Thanks in advance!

Issue with BSON decoder and ObjectId

I get an error when the BSON decoder tries to decode the machine identifier bytes of an ObjectId.

** (MatchError) no match of right hand side value: <<111, 230, 80, 74, 154, 74, 223, 8>>
stacktrace:
  (mongodb) lib/bson/decoder.ex:98: BSON.Decoder.document/1
  (mongodb) lib/bson/decoder.ex:6: BSON.Decoder.decode/1
  test/controllers/user_controller_test.exs:27

The original binary was <<86, 206, 224, 104, 111, 230, 80, 74, 154, 74, 223, 8>>

BSON decoder error

Hi, on one of my collections I get the following error:

 ** (FunctionClauseError) no function clause matching in BSON.Decoder.type/2
            (mongodb) lib/bson/decoder.ex:10: BSON.Decoder.type(1, <<1, 0, 0, 0, 0, 0, 240, 127, 1, 49, 0, 1, 0, 0, 0, 0, 0, 240, 127>>)
 lib/bson/decoder.ex:10: BSON.Decoder.type(1, <<0, 0, 0, 0, 0, 0, 240, 127, 1, 112, 114, 111, 106, 101, 99, 116, 101, 100, 69, 84, 99, 70, 114, 111, 109, 104, 105, 115, 116, 111, 114, 121, 0, 170, 241, 210, 77, 98, 16, 201, 63, 2, 67, 73, 77, 73, 83, 83, 116, 97, ...>>)

I am pulling from master since it seems a similar issue was resolved recently, but it still breaks. I'm connecting to an existing database so I need to be able to access the values. Is there a workaround for this type of issue before it is fixed?

I think it is caused by a 2 element array that has nans with typedouble.

Thanks.

Mongo.Aggregate: Disallowed field type Array in object expression

When performing a sum aggregation and excluding _id from the final output I get an exception. The same operation can be completed from Mongo Client with out issue.

MongoDB shell version: 2.6.10

Mongo Client

db.www_test_com.aggregate( 
  [ 
    { $group: { _id: '', total: { $sum: "$bytes_sent" } } }, 
    { $project: { _id: 0, total: '$total' } } 
  ] 
);

{ "total" : 85000 }

Using Mongo.aggregate with project

resp = Mongo.aggregate(MongoPool, "www_test_com",
      [
        %{'$group': %{ '_id': false, 'total': %{ '$sum': "$bytes_sent" }}},
        %{'$project': %{ '_id': 0, 'total': '$total' }}
      ]
    ) |> Enum.to_list

** (EXIT) an exception was raised:
   ** (CaseClauseError) no case clause matching: {:ok, %Mongo.ReadResult{cursor_id: 0, docs: [%{"code" => 15992, "errmsg" => "exception: disallowed field type Array in object expression (at 'total')", "ok" => 0.0}], from: 0, num: 1}}

Using Mongo.aggregate without project

resp = Mongo.aggregate(MongoPool, "www_test_com",
      [
        %{'$group': %{ '_id': false, 'total': %{ '$sum': "$bytes_sent" }}}
      ]
    ) |> Enum.to_list

IO.inspect(resp)
[%{"_id" => false, "total" => 85000}]

protocol Enumerable not implemented for %Mongo.Cursor

Hi there. Thanks for this great library! I have been struggling with a few other libraries that are basically dead. Everything works as expected until this issue came up. I can't figure out how to get past it, and using the code for find that you're using in the tests. I understand this is basically pre-alpha, but my requirements are limited at the moment.

  defmodule MongoPool do
    use Mongo.Pool, name: __MODULE__, adapter: Mongo.Pool.Poolboy
  end

collection = "broadcasts0"
Mongo.find(MongoPool, collection, %{} |> Enum.to_list
** (exit) an exception was raised:
    ** (Protocol.UndefinedError) protocol Enumerable not implemented for %Mongo.Cursor{coll: "broadcasts-0", opts: [], pool: MongoPool, query: %{}, select: nil}
        (elixir) lib/enum.ex:1: Enumerable.impl_for!/1
        (elixir) lib/enum.ex:112: Enumerable.reduce/3
        (elixir) lib/enum.ex:1265: Enum.reduce/3
        (elixir) lib/enum.ex:1807: Enum.to_list/1

Thanks 😃

GenServer Timeout

Hi @ericmj! I have been recently working with massive datasets and more or less complex aggregate functions to retrieve data, operations that sometimes last 2-3 minutes. I've set the HTTP timeout high but even with this option if my process lasts more than five secs the GenServer.call raises an exception because it is always called with the default timeout for GenServer operations(5000ms), do you think is worth to send the same time as in HTTP timeout for mongo? For the moment I have this in my local project:

GenServer.call(conn, {:find, coll, query, select, opts}, 60_000)

60_000 is a hardcoded limit I put only to try, and it worked.

:get_more timeout error

If I run a Mongo.find query on a large mongo collection in a busy server with:

cursor = Mongo.find(MongoPool, "userdata", %{state: "ready"}, batch_size: 1000)

After a few hours of running I always get a timeout into :get_more:

** (exit) exited in: GenServer.call(#PID<0.169.0>, {:get_more, "userdata", 17595059170, [batch_size: 1000]}, 5000)
    ** (EXIT) time out
    (elixir) lib/gen_server.ex:564: GenServer.call/3
    (stdlib) timer.erl:181: :timer.tc/2
    lib/mongo/pool/poolboy.ex:36: Mongo.Pool.Poolboy.run/2
    lib/mongo/pool.ex:142: Mongo.Pool.run_with_log/5
    lib/mongo/cursor.ex:65: anonymous fn/3 in Enumerable.Mongo.Cursor.next_fun/2
    (elixir) lib/stream.ex:1099: Stream.do_resource/5
    (elixir) lib/enum.ex:1477: Enum.reduce/3
    (elixir) lib/enum.ex:1092: Enum.map/2

The :timeout connection parameter is not used for :get_more so the only way to fix it is to manually edit this line: https://github.com/ericmj/mongodb/blob/943f6bcefff916e2e8b572fad1c9d79d55235490/lib/mongo/connection.ex#L64

and change it to:

GenServer.call(conn, {:get_more, coll, cursor_id, opts}, 50000)

With 50000 everything works fine and I no longer get any timeouts. So I think the :timeout connection parameter should also be use here or maybe create a new parameter.

List collections?

Hi,
I'd like to query the connected database to get a list of collections?
Can you give an example of how to do that?
Thanks
-John

How to catch/rescue errors?

The code below would throw a duplicate ID error, but the rescue doesn't work.
What's the right way to catch/rescue an operation so that I can log the message?

try do
  Mongo.insert_one(MongoPool, "articles",%{"_id" => "aaa"})
  Mongo.insert_one(MongoPool, "articles",%{"_id" => "aaa"})
rescue
  e in RuntimeError -> IO.inspect e
end

insert_many crash on blank document set

If you are just recursion processing results, and pass a blank result set, it crashes on this.

** (FunctionClauseError) no function clause matching in Mongo.many_docs/1
             (mongodb) lib/mongo.ex:449: Mongo.many_docs([])
             (mongodb) lib/mongo.ex:130: Mongo.insert_many/4
    (twitch_discovery) lib/index/stream.ex:2: TwitchDiscovery.Index.Stream.mongo_save_many/1
    (twitch_discovery) lib/index/stream.ex:2: TwitchDiscovery.Index.Stream.get_next/1
    (twitch_discovery) lib/index.ex:41: TwitchDiscovery.Index.index/0

Add support for NaN and +/-Inf

Elixir (and Erlang) doesn't natively support those special floating point values so if we see any such value from Mongo, we'll simply throw up an error.

        ** (FunctionClauseError) no function clause matching in BSON.Decoder.type/2
            (mongodb) lib/bson/decoder.ex:10: BSON.Decoder.type(1, <<0, 0, 0, 0, 0, 0, 248, 127>>)

It might be possible to binary match to detect those values and return our own special atoms as discussed here: https://groups.google.com/forum/#!topic/capnproto/T6jEFu38rtI

and here: kaos/ecapnp@a4e3221#diff-d28a6e8472db5dc7149ae3347d12566aR84

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.