kirill5k / mongo4cats

MongoDB client wrapper for Cats Effect & FS2

mongo4cats

MongoDB Java client wrapper compatible with Cats-Effect/FS2 and ZIO. Available for Scala 2.12, 2.13 and 3.3.

Documentation is available on the mongo4cats microsite.

Dependencies

Add this to your build.sbt (depends on cats-effect and FS2):

libraryDependencies += "io.github.kirill5k" %% "mongo4cats-core" % "<version>"
libraryDependencies += "io.github.kirill5k" %% "mongo4cats-embedded" % "<version>" % Test

Alternatively, for ZIO 2, add this:

libraryDependencies += "io.github.kirill5k" %% "mongo4cats-zio" % "<version>"
libraryDependencies += "io.github.kirill5k" %% "mongo4cats-zio-embedded" % "<version>" % Test

Optional support for circe or zio-json can be enabled with:

// circe
libraryDependencies += "io.github.kirill5k" %% "mongo4cats-circe" % "<version>"
// zio-json
libraryDependencies += "io.github.kirill5k" %% "mongo4cats-zio-json" % "<version>"

Quick start with Cats Effect

import cats.effect.{IO, IOApp}
import mongo4cats.client.MongoClient
import mongo4cats.operations.{Filter, Projection}
import mongo4cats.bson.Document
import mongo4cats.bson.syntax._

object Quickstart extends IOApp.Simple {

  override val run: IO[Unit] =
    MongoClient.fromConnectionString[IO]("mongodb://localhost:27017").use { client =>
      for {
        db   <- client.getDatabase("my-db")
        coll <- db.getCollection("docs")
        _    <- coll.insertMany((0 to 100).map(i => Document("name" := s"doc-$i", "index" := i)))
        docs <- coll.find
          .filter(Filter.gte("index", 10) && Filter.regex("name", "doc-[1-9]0"))
          .projection(Projection.excludeId)
          .sortByDesc("name")
          .limit(5)
          .all
        _ <- IO.println(docs.mkString("[\n", ",\n", "\n]"))
      } yield ()
    }
}

Quick start with ZIO

import mongo4cats.bson.Document
import mongo4cats.bson.syntax._
import mongo4cats.operations.{Filter, Projection}
import mongo4cats.zio.{ZMongoClient, ZMongoCollection, ZMongoDatabase}
import zio._

object Zio extends ZIOAppDefault {

  val client     = ZLayer.scoped[Any](ZMongoClient.fromConnectionString("mongodb://localhost:27017"))
  val database   = ZLayer.fromZIO(ZIO.serviceWithZIO[ZMongoClient](_.getDatabase("my-db")))
  val collection = ZLayer.fromZIO(ZIO.serviceWithZIO[ZMongoDatabase](_.getCollection("docs")))

  val program = for {
    coll <- ZIO.service[ZMongoCollection[Document]]
    _    <- coll.insertMany((0 to 100).map(i => Document("name" := s"doc-$i", "index" := i)))
    docs <- coll.find
      .filter(Filter.gte("index", 10) && Filter.regex("name", "doc-[1-9]0"))
      .projection(Projection.excludeId)
      .sortByDesc("name")
      .limit(5)
      .all
    _ <- Console.printLine(docs.mkString("[\n", ",\n", "\n]"))
  } yield ()

  override def run = program.provide(client, database, collection)
}

If you find this library useful, consider giving it a ⭐!

mongo4cats's Issues

Document id forced to ObjectId

Hi!

It seems that mongo4cats imposes the limitation that stored document must have their _id field be of the ObjectId type:

mongo4cats/modules/kernel/src/main/scala/mongo4cats/codecs/ContainerValueWriter.scala

Line 34 in 2ddd402

idFieldName.flatMap(idName => value.getObjectId(idName).map(idName -> _)).foreach { case (idName, idValue) =>

.

Where I work, we're trying to incorporate more of the Typelevel ecosystem (cats, CE...), but we're explicitly not using auto generated values for the _id field, or ObjectId instances in any way. What's the reasoning behind mongo4cats requiring the ObjectId type, or did it just end up being that way?

I could submit a PR to change this, but first I just want to raise the issue before doing the work, in case the limitation is intentional and the PR would never be accepted..

Question: How to use this in context of an http4s app wish FS2

Hi there! Apologies for asking a question as an issue, but I see that discussions aren't enabled on your repo.

Firstly, thanks for providing this library, so far, it looks great, and I hope it will be able to integrate into a new app I'm writing.

I've spun up a small sample app with http4s, but I've run into a wall using it when it isn't an IOApp.Simple application. I am using it in the context of F in the following pseudocode example:

def stream[F[_] : Async]: Stream[F, Nothing] = {
    for {
      mongoClient: MongoClient[F] <- Stream.resource(MongoClient.fromConnectionString[F]("mongodb://localhost:27017"))
      versionsAlg = VersionRegistry.impl[F](mongoClient)
      ...
      httpApp = ???
      exitCode <- Stream.resource(
        EmberServerBuilder.default[F]
          .withHost(ipv4"0.0.0.0")
          .withPort(port"8080")
          .withHttpApp(finalHttpApp)
          .build >>
          Resource.eval(Async[F].never)
      )
    } yield (exitCode)
  }.drain

At this point, there is no problem as I now have MongoClient[F]. The problem comes inside my VersionRegistry.impl[F]:

  def impl[F[_] : Concurrent](M: MongoClient[F]): VersionRegistry[F] = new VersionRegistry[F] :
    def get(beta: String): F[VersionRegistry.Versions] =
      val db: F[MongoDatabase[F]] = M.getDatabase("sdkman")

The problem I'm now facing is that I'm not able to flatMap over F[MongoDatabase[F]] as this is merely an F, not an IO. I think the Mongo client wants you to get a new database instance on every request; therefore, I'm re-using the singleton client at the above level.

This is most probably an idiotic question with a straightforward answer. Still, I'm just not seeing it right now 😄 Once I get the answer, I'd be happy to contribute a PR to your docs to clarify this for future travellers who want to integrate this into an http4s app.
I've also thrown my sample code project up in a repository for you to check out and play with.
I hope you can shed some light, thanks!

Mongo 3.6 required, but I don't have that

Hi, I know this is not actually a bug.

But I get error Server at localhost:27017 reports wire version 4, but this version of the driver requires at least 6 (MongoDB 3.6)

Is there any version or any way to use the wire version 4? I can only install MongoDB 3.4

Hard to close a session

It seems like mongo4cats.client.ClientSession implements the methods declared in com.mongodb.reactivestreams.client.ClientSession but not the ones from that interface's superinterface, com.mongodb.session.ClientSession. As far as I know, it is only possible to close a session by calling close() on underlying. I assume that is an oversight and not a deliberate design choice. Am I incorrect?

Scala 3 & Circe: Can't upgrade since 0.6.8 due to circe

I know this is a bug in circe, but this seems to block me from upgrading from 0.6.6:

circe/circe#2089

On Scala 3.2.2 & ZIO 2 app start I get:

core timestamp=2023-03-08T09:01:56.263939006Z level=ERROR thread=#zio-fiber-0 message="" cause="Exception in thread "zio-fiber-4" java.lang.NoSuchMethodError: 'scala.collection.Iterable io.circe.DerivedEncoder.encodedIterable$(io.circe.DerivedEncoder, scala.Product)'
core 	at sttp.apispec.openapi.internal.InternalSttpOpenAPICirceEncoders$$anon$22.encodedIterable(InternalSttpOpenAPICirceEncoders.scala:28)

ContainerValueReader Unable to read TimeStamp

The JsonDocumentFindAndUpdate.scala example throws an exception because it is unable to map the TimeStamp type which is unimplemented in readBsonValue() l. 92:

I can shift to a Date_Time in the meanwhile but just wondering if you're planning on implementing this anytime soon?

org.bson.BsonInvalidOperationException: ReadBSONType can only be called when State is TYPE, not when State is VALUE.
at org.bson.AbstractBsonReader.throwInvalidState(AbstractBsonReader.java:668)
at org.bson.BsonBinaryReader.readBsonType(BsonBinaryReader.java:88)
at mongo4cats.codecs.ContainerValueReader$.go$1(ContainerValueReader.scala:32)
at mongo4cats.codecs.ContainerValueReader$.readBsonDocument(ContainerValueReader.scala:40)
at mongo4cats.codecs.DocumentCodec.decode(DocumentCodecProvider.scala:37)
at mongo4cats.codecs.DocumentCodec.decode(DocumentCodecProvider.scala:25)

Dispatcher closing prematurely

I think the stream call is closing Dispatcher resource prematurely. Subscribed callbacks can happen after the Dispatcher is closed if the Stream returned by stream is compiled. Here is the exception thrown when this happens:

reactor.core.Exceptions$ErrorCallbackNotImplemented: java.lang.IllegalStateException: dispatcher already shutdown
Caused by: java.lang.IllegalStateException: dispatcher already shutdown
	at cats.effect.std.Dispatcher$$anon$2.unsafeToFutureCancelable(Dispatcher.scala:283)
	at cats.effect.std.DispatcherPlatform.unsafeRunTimed(DispatcherPlatform.scala:60)
	at cats.effect.std.DispatcherPlatform.unsafeRunTimed$(DispatcherPlatform.scala:59)
	at cats.effect.std.Dispatcher$$anon$2.unsafeRunTimed(Dispatcher.scala:190)
	at cats.effect.std.DispatcherPlatform.unsafeRunSync(DispatcherPlatform.scala:52)
	at cats.effect.std.DispatcherPlatform.unsafeRunSync$(DispatcherPlatform.scala:51)
	at cats.effect.std.Dispatcher$$anon$2.unsafeRunSync(Dispatcher.scala:190)
	at mongo4cats.database.helpers$PublisherOps$$anon$4.onComplete(helpers.scala:77)

I think we can prevent this if we use fs2-reactive-streams.

The stream implementation would change to:

    def stream[F[_]: Async]: Stream[F, T] =
      fromPublisher(publisher)

And I think with this approach the boundedStreams may not be required anymore.

Exception when using `$slice` in aggregations

Hi,

I'm getting an exception on $slice but I can't tell how I'm using it wrong:

Aggregate
  .project(Projection.excludeId)
  .unwind("$name", UnwindOptions().preserveNullAndEmptyArrays(false))
  .group("$_id", Accumulator.addToSet("uniqueNames", "$name"))
  .project(Projection.slice("uniqueNames", 1, 2))

com.mongodb.MongoCommandException: Command failed with error 28724 (Location28724): 'Failed to optimize pipeline :: caused by :: First argument to $slice must be an array, but is of type: int' on server localhost:27017. The full response is {"ok": 0.0, "errmsg": "Failed to optimize pipeline :: caused by :: First argument to $slice must be an array, but is of type: int", "code": 28724, "codeName": "Location28724"}

If I remove the skipargument and do Projection.slice("uniqueNames", 2), I get:

com.mongodb.MongoCommandException: Command failed with error 28667 (Location28667): 'Invalid $project :: caused by :: Expression $slice takes at least 2 arguments, and at most 3, but 1 were passed in.' on server localhost:27017. The full response is {"ok": 0.0, "errmsg": "Invalid $project :: caused by :: Expression $slice takes at least 2 arguments, and at most 3, but 1 were passed in.", "code": 28667, "codeName": "Location28667"}
at com.mongodb.internal.connection.ProtocolHelper.getCommandFailureException(ProtocolHelper.java:198)

From the errors, it looks like the fieldName is ignored in favor of the second argument.

This is my collection:

[
  { _id: 1, age: 44, name: "A" },
  { _id: 2, age: 29, name: "B" },
  { _id: 3, age: 95, name: "B" },
  { _id: 4, age: 50, name: "C" },
  { _id: 5, age: 36, name: "D" },
  { _id: 6, age: 36, name: "B" },
]

This is the aggregation that I'm trying to build. I want to get the unique names, skip the first one, and return the next two:

db.foo.aggregate(
  //  Expected after this stage:
  //      [ { _id: 0, names: [ 'A', 'B', 'B', 'C', 'B' ], total: 5 } ]
    {
      $group: {
        _id: 0,
        uniqueNames: { $push: {$toString: '$name'} },
      }
    },

  //  Expected after this stage:
  //      [ { uniqueNames: [ 'B', 'C' ] } ]
    {
      $project: {
        _id: 0,
        uniqueNames: {
          $slice: [{ $setIntersection: ['$uniqueNames'] }, 1, 2]
        }
      }
    },
)

If it helps, I'm using mongo4cats-core:0.6.6 and also tried 0.6.4, along with the latest MongoDB 6.0.4.

Have I stumbled upon a bug or am I doing something wrong?

Potential slowness when converting to a fs2.Stream

Calling .stream on a collection and then materializing the stream (that will retrieve a finite amount of values, say a couple of hundreds) seems to be a lot slower than calling .all

Potential connection pool outage

This is more likely related to the underlying mongo-java-driver.
With the default settings, the connection pool will reach its max given a moderate amount of traffic and requests would start to timeout due to the connection pool being full. (It seems that the connection are not properly re-used, I'm following some issues/PRs in the mongo-java-driver and I can raise a PR once a new library is released)

Typo in `witReadConcern` in `MongoCollection`

CirceJsonMapper doesn't support json integers large than MaxLong

The CirceJsonMapper checks for decimal values by the existence of a decimal and then tries to convert all values without a decimal to an integer (int32) or long (int64). This fails for the case that a json integer is provided that exceeds the 64 bit long value.

Example that will fail:

import mongo4cats.bson.syntax._
import mongo4cats.circe._
import io.circe.Json

Json.obj("decimal_value" -> Json.fromBigInt(BigInt(Long.MaxValue) + 1)).toBson

With the following

java.util.NoSuchElementException: None.get
  scala.None$.get(Option.scala:627)
  scala.None$.get(Option.scala:626)
  mongo4cats.circe.CirceJsonMapper$JsonNumberSyntax$.toBsonValue$extension(CirceJsonMapper.scala:67)
  mongo4cats.circe.CirceJsonMapper$.toBson(CirceJsonMapper.scala:37)
  mongo4cats.circe.CirceJsonMapper$.$anonfun$toBson$2(CirceJsonMapper.scala:43)
  scala.collection.immutable.List.map(List.scala:246)
  mongo4cats.circe.CirceJsonMapper$.toBson(CirceJsonMapper.scala:43)
  mongo4cats.circe.MongoJsonCodecs.$anonfun$deriveJsonBsonValueEncoder$1(MongoJsonCodecs.scala:39)
  mongo4cats.bson.syntax$ValueSyntax$.toBson$extension(syntax.scala:26)

Distinct query breaks typed collections

for {
  collection <- db.getCollection("name")
  res <- collection.distinct("column").first
} yield ()

This will fail if the column is not a Document which I'd expect it normally is not. So I ended up having to do this to make it work:

for {
  collection <- db.getCollection[String]("name", MongoDatabaseF.DefaultCodecRegistry)
  res <- collection.distinct("column").first
} yield ()

The issue with this is that now the type of the collection has to change to conform to the distinct query, so I can't do a find using the same collection. It seems like it might be a good idea to make aggregations pass a Decoder implicit for distinct that will decode the BsonValue into the required type.

Question: $facet support in mongo4cats

Hi @Kirill5k ,

Thanks for all the hard work you put into this great project.

I'm looking to build out a pagination feature so that it returns the total number of rows while I page over the results, using the same query (stage) to give me data as well as the full row count. I read about $facet in Mongodb that I'd like to use in mongo4cats, but I don't see any support for facet. Am I overlooking the support or is there a way I can achieve the same functionality using mongo4cats?

ZIO Json integration

It would be nice to see ZIO Json integration module added to the library.

Support for Circe

It would be nice to be able to provide Circe codecs to the mongo client.

ZClientSessionLive#commitTransaction and #abortTransaction Ignore Result

ZClientSessionLive#commitTransaction and ZClientSessionLive#abortTransaction ignore their results and always return ().
The problem stems from the fact that unlike com.mongodb.reactivestreams.client.ClientSession#startTransaction(), which just returns void, com.mongodb.reactivestreams.client.ClientSession#commitTransaction actually returns Publisher<Void> but ZClientSessionLive#commitTransaction is implemented as if it also just returned void. Same for #abortTransaction.
The mongo4cats.client.LiveClientSession does not share this problem.

Intellij marks ZMongo usages as errors

I know Intellij is buggy sometimes when working with Scala 3 but can we somehow bypass the display of these errors? I suppose it ocurres because Intellij finds object and trait in the package simultaneously and cannot decide which one it is referring to :)