zebresel-com / mongodm

A golang object document mapper (ODM) for MongoDB

What is mongodm?

The mongodm package is an object document mapper (ODM) for mongodb written in Go which uses the official mgo adapter. You can find an example API application here.

Features

• 1:1, 1:n struct relation mapping and embedding
• call Save(),Update(), Delete() and Populate() directly on document instances
• call Select(), Sort(), Limit(), Skip() and Populate() directly on querys
• validation (default and custom with regular expressions) followed by translated error list (customizable)
• population instruction possible before and after querys
• Find(), FindOne() and FindID()
• default handling for ID, CreatedAt, UpdatedAt and Deleted attribute
• extends *mgo.Collection
• default localisation (fallback if none specified)
• database authentication (user and password)
• multiple database hosts on connection

Todos

• recursive population
• add more validation presets (like "email")
• benchmarks
• accept plain strings as objectID value
• virtuals and hooks (like in mongoose)

Usage

Note(!)

Collection naming in this package is switched to Model.

Fetch (terminal)

go get github.com/zebresel-com/mongodm

Import

Add import "github.com/zebresel-com/mongodm" in your application file.

Define your own localisation for validation

First step is to create a language file in your application (skip if you want to use the english defaults). This is necessary for document validation which is always processed. The following entrys are all keys which are currently used. If one of the keys is not defined the output will be the key itself. In the next step you have to specify a translation map when creating a database connection.

For example:

{
    "en-US": {
        "validation.field_required": "Field '%s' is required.",
        "validation.field_invalid": "Field '%s' has an invalid value.",
        "validation.field_invalid_id": "Field '%s' contains an invalid object id value.",
        "validation.field_minlen": "Field '%s' must be at least %v characters long.",
        "validation.field_maxlen": "Field '%s' can be maximum %v characters long.",
        "validation.entry_exists": "%s already exists for value '%v'.",
        "validation.field_not_exclusive": "Only one of both fields can be set: '%s'' or '%s'.",
        "validation.field_required_exclusive": "Field '%s' or '%s' required."
    }
}

Create a database connection

Subsequently you have all information for mongodm usage and can now connect to a database. Load your localisation file and parse it until you get a map[string]string type. Then set the database host and name. Pass the config reference to the mongodm Connect() method and you are done. (You dont need to set a localisation file or credentials)

	file, err := ioutil.ReadFile("locals.json")

	if err != nil {
		fmt.Printf("File error: %v\n", err)
		os.Exit(1)
	}

	var localMap map[string]map[string]string
	json.Unmarshal(file, &localMap)

	dbConfig := &mongodm.Config{
		DatabaseHosts: []string{"127.0.0.1"},
		DatabaseName: "mongodm_sample",
		DatabaseUser: "admin",
		DatabasePassword: "admin",
		// The option `DatabaseSource` is the database used to establish 
		// credentials and privileges with a MongoDB server. Defaults to the value
		// of `DatabaseName`, if that is set, or "admin" otherwise.
		DatabaseSource: "admin",
		Locals:       localMap["en-US"],
	}

	connection, err := mongodm.Connect(dbConfig)

	if err != nil {
		fmt.Println("Database connection error: %v", err)
	}

You can also pass a custom DialInfo from mgo (*mgo.DialInfo). If used, all config attributes starting with Database will be ignored:

	file, err := ioutil.ReadFile("locals.json")

	if err != nil {
		fmt.Printf("File error: %v\n", err)
		os.Exit(1)
	}

	var localMap map[string]map[string]string
	json.Unmarshal(file, &localMap)

	dbConfig := &mongodm.Config{
		DialInfo: &mgo.DialInfo{
			Addrs:    []string{"127.0.0.1"},
			Timeout:  3 * time.Second,
			Database: "mongodm_sample",
			Username: "admin",
			Password: "admin",
			Source:   "admin",
		},
		Locals:       localMap["en-US"],
	}

	connection, err := mongodm.Connect(dbConfig)

	if err != nil {
		fmt.Println("Database connection error: %v", err)
	}

Create a model

package models

import "github.com/zebresel-com/mongodm"

type User struct {
	mongodm.DocumentBase `json:",inline" bson:",inline"`

	FirstName string       `json:"firstname" bson:"firstname"`
	LastName  string       `json:"lastname"	 bson:"lastname"`
	UserName  string       `json:"username"	 bson:"username"`
	Messages  interface{}  `json:"messages"	 bson:"messages" 	model:"Message" relation:"1n" autosave:"true"`
}

It is important that each schema embeds the IDocumentBase type (mongodm.DocumentBase) and make sure that it is tagged as 'inline' for json and bson. This base type also includes the default values id, createdAt, updatedAt and deleted. Those values are set automatically from the ODM. The given example also uses a relation (User has Messages). Relations must always be from type interface{} for storing bson.ObjectId OR a completely populated object. And of course we also need the related model for each stored message:

type Message struct {
	mongodm.DocumentBase `json:",inline" bson:",inline"`

	Sender 	  string       `json:"sender" 	 bson:"sender"`
	Receiver  string       `json:"receiver"	 bson:"receiver"`
	Text  	  string       `json:"text"	 bson:"text"`
}

Note that when you are using relations, each model will be stored in his own collection. So the values are not embedded and instead stored as object ID or array of object ID's.

To configure a relation the ODM understands three more tags:

model:"Message"

	This must be the struct type you want to relate to.

	Default: none, must be set

relation:"1n"

	It is important that you specify the relation type one-to-one or one-to-many because the ODM must decide whether it accepts an array or object.

	Possible: "1n", "11"
	Default: "11"

autosave:"true"

	If you manipulate values of the message relation in this example and then call 'Save()' on the user instance, this flag decides if this is possible or not.
	When autosave is activated, all relations will also be saved recursively. Otherwise you have to call 'Save()' manually for each relation.

	Possible: "true", "false"
	Default: "false"

But it is not necessary to always create relations - you also can use embedded types:

type Customer struct {
	mongodm.DocumentBase `json:",inline" bson:",inline"`

	FirstName string       `json:"firstname" bson:"firstname"`
	LastName  string       `json:"lastname"	 bson:"lastname"`
	Address   *Address     `json:"address"	 bson:"address"`
}

type Address struct {

	City 	string       `json:"city" 	 bson:"city"`
	Street  string       `json:"street"	 bson:"street"`
	ZipCode	int16	     `json:"zip"	 bson:"zip"`
}

Persisting a customer instance to the database would result in embedding a complete address object. You can embed all supported types.

Now that you got some models and a connection to the database you have to register these models for the ODM for working with them.

Register your models (collections)

It is necessary to register your created models to the ODM to work with. Within this process the ODM creates an internal model and type registry to work fully automatically and consistent. Make sure you already created a connection. Registration expects a pointer to an IDocumentBase type and the collection name where the docuements should be stored in. Register your collections only once at runtime!

For example:

connection.Register(&User{}, "users")
connection.Register(&Message{}, "messages")
connection.Register(&Customer{}, "customers")

Working on a model (collection)

To create actions on each collection you have to request a model instance. Make sure that you registered your collections and schemes first, otherwise it will panic.

For example:

User := connection.Model("User")

User.Find() ...

Persist a new document in a collection

Save() persists all changes for a document. Populated relations are getting converted to object ID's / array of object ID's so you dont have to handle this by yourself. Use this function also when the document was newly created, if it is not existent the method will call insert. During the save process createdAt and updatedAt gets also automatically persisted.

For example:

User := connection.Model("User")

user := &models.User{}

User.New(user) //this sets the connection/collection for this type and is strongly necessary(!) (otherwise panic)

user.FirstName = "Max"
user.LastName = "Mustermann"

err := user.Save()

FindOne

If you want to find a single document by specifing query options you have to use this method. The query param expects a map (e.g. bson.M{}) and returns a query object which has to be executed manually. Make sure that you pass an IDocumentBase type to the exec function. After this you obtain the first matching object. You also can check the error if something was found.

For example:

User := connection.Model("User")

user := &models.User{}

err := User.FindOne(bson.M{"firstname" : "Max", "deleted" : false}).Populate("Messages").Exec(user)

if _, ok := err.(*mongodm.NotFoundError); ok {
	//no records were found
} else if err != nil {
	//database error
} else {
	fmt.Println("%v", user)
}

Find

UseFind() if you want to fetch a set of matching documents. Like FindOne, a map is expected as query param, but you also can call this method without any arguments. When the query is executed you have to pass a pointer to a slice of IDocumentBase types.

For example:

User := connection.Model("User")

users := []*models.User{}

err := User.Find(bson.M{"firstname" : "Max", "deleted" : false}).Populate("Messages").Exec(&users)

if _, ok := err.(*mongodm.NotFoundError); ok { //you also can check the length of the slice
	//no records were found
} else if err != nil {
	//database error
} else {
	for user, _ := range users {
		fmt.Println("%v", user)
	}
}

FindId

If you have an object ID it is possible to find the matching document with this param.

For example:

User := connection.Model("User")

user := &models.User{}

err := User.FindId(bson.ObjectIdHex("55dccbf4113c615e49000001")).Select("firstname").Exec(user)

if _, ok := err.(*mongodm.NotFoundError); ok {
	//no records were found
} else if err != nil {
	//database error
} else {
	fmt.Println("%v", user)
}

Populate

This method replaces the default object ID value with the defined relation type by specifing one or more field names. After it was succesfully populated you can access the relation field values. Note that you need type assertion for this process.

For example:

User := connection.Model("User")

user := &models.User{}

err := User.Find(bson.M{"firstname" : "Max"}).Populate("Messages").Exec(user)

if err != nil {
	fmt.Println(err)
}

for _, user := range users {

	if messages, ok := user.Messages.([]*models.Message); ok {

		for _, message := range messages {

			fmt.Println(message.Sender)
		}
	} else {
		fmt.Println("something went wrong during type assertion. wrong type?")
	}
}

or after your query only for single users:

User := connection.Model("User")

user := &models.User{}

err := User.Find(bson.M{"firstname" : "Max"}).Exec(user)

if err != nil {
	fmt.Println(err)
}

for index, user := range users {

	if user.FirstName == "Max" {

		err := user.Populate("Messages")

		if err != nil {
		
			fmt.Println(err)
			
		} else if messages, ok := user.Messages.([]*models.Message); ok {
	
			for _, message := range messages {
	
				fmt.Println(message.Text)
			}
		} else {
			fmt.Println("something went wrong during type assertion. wrong type?")
		}
	}
}

Note: Only the first relation level gets populated! This process is not recursive.

Default document validation

To validate model attributes/values you first have to define some rules. Therefore you can add tags:

type User struct {
	mongodm.DocumentBase `json:",inline" bson:",inline"`

	FirstName    string   `json:"firstname"  bson:"firstname" minLen:"2" maxLen:"30" required:"true"`
	LastName     string   `json:"lastname"  bson:"lastname" minLen:"2" maxLen:"30" required:"true"`
	UserName     string   `json:"username"  bson:"username" minLen:"2" maxLen:"15"`
	Email        string   `json:"email" bson:"email" validation:"email" required:"true"`
	PasswordHash string   `json:"-" bson:"passwordHash"`
	Address      *Address `json:"address" bson:"address"`
}

This User model defines, that the firstname for example must have a minimum length of 2 and a maximum length of 30 characters (minLen, maxLen). Each required attribute says, that the attribute can not be default or empty (default value is required:"false"). The validation tag is used for regular expression validation. Currently there is only one preset "email". A use case would be to validate the model after a request was mapped:

User := self.db.Model("User")
user := &models.User{}

err, _ := User.New(user, self.Ctx.Input.RequestBody)

if err != nil {
	self.response.Error(http.StatusBadRequest, err)
	return
}

if valid, issues := user.Validate(); valid {

		err = user.Save()
		
		if err != nil {
			self.response.Error(http.StatusInternalServerError)
			return
		}
		
		// Go on..
		
	} else {
		self.response.Error(http.StatusBadRequest, issues)
		return
	}

This example maps a received Ctx.Input.RequestBody to the attribute values of a new user model. Continuing with calling user.Validate() we detect if the document is valid and if not what issues we have (a list of validation errors). Each Save call will also validate the current state. The document gets only persisted when there were no errors.

Custom document validation

In some cases you may want to validate request parameters which do not belong to the model itself or you have to do advanced validation checks. Then you can hook up before default validation starts:

func (self *User) Validate(values ...interface{}) (bool, []error) {

	var valid bool
	var validationErrors []error

	valid, validationErrors = self.DefaultValidate()

	type m map[string]string

	if len(values) > 0 {

		//expect password as first param then validate it with the next rules
		if password, ok := values[0].(string); ok {

			if len(password) < 8 {

				self.AppendError(&validationErrors, mongodm.L("validation.field_minlen", "password", 8))

			} else if len(password) > 50 {

				self.AppendError(&validationErrors, mongodm.L("validation.field_maxlen", "password", 50))
			}

		} else {

			self.AppendError(&validationErrors, mongodm.L("validation.field_required", "password"))
		}
	}

	if len(validationErrors) > 0 {
		valid = false
	}

	return valid, validationErrors
}

Simply add a Validate method in your IDocumentBase type model with the signature Validate(...interface{}) (bool, []error). Within this you can implement any checks that you want. You can call the DefaultValidate method first to run all default validations. You will get a valid and validationErrors return value. Now you can run your custom checks and append some more errors with AppendError(*[]error, message string). Also have a look at the mongodm.L method if you need language localisation! The next example shows how we can use our custom validate method:

User := self.db.Model("User")
	user := &models.User{}

	// NOTE: we now want our request body get back as a map (requestMap)..
	err, requestMap := User.New(user, self.Ctx.Input.RequestBody)

	if err != nil {
		self.response.Error(http.StatusBadRequest, err)
		return
	}

	//NOTE: ..and validate the "password" parameter which is not part of the model/document
	if valid, issues := user.Validate(requestMap["password"]); valid {
		err = user.Save()
		
		if err != nil {
			self.response.Error(http.StatusInternalServerError)
			return
		}
	} else {
		self.response.Error(http.StatusBadRequest, issues)
		return
	}

	self.response.AddContent("user", user)
	self.response.SetStatus(http.StatusCreated)
	self.response.ServeJSON()
}

In this case we retrieve a requestMap and forward the password attribute to our Validate method (example above). If you want to use your own regular expression as attribute tags then use the following format: validation:"/YOUR_REGEX/YOUR_FLAG(S)" - for example: validation:"/[a-z0-9._%+\-]+@[a-z0-9.\-]+\.[a-z]{2,4}/"

Contribute

Read this and feel free :-)

Dockerized

With docker, you do not need to install go and mongodb on your local machine, it's easy to setup a development environment for this repository. Thanks to @centsent who helped to dockerize this project.

Prerequisites

Make sure you already installed below programs on your local machine:

• git
• docker
• docker-compose

dep

dep is a prototype dependency management tool for Go. To use dep in the container, prefix make for all dep commands, for example:

$ make dep "ensure -add github.com/some/repos"

Beware that the double quotes are required after make dep command.

Making Changes

Puppet has some good and simple contribution rules so lets adopt them in an adjusted way.

• Create a topic branch from where you want to base your work.
  • This is usually the master branch.
  • To quickly create a topic branch based on master, run git checkout -b fix/master/my_contribution master. Please avoid working directly on the master branch.
• Make commits of logical and atomic units.
• Check for unnecessary whitespace with git diff --check before committing.
• Make sure your commit messages are in a proper format:

  chore: add Oyster build script
  docs: explain hat wobble
  feat: add beta sequence
  fix: remove broken confirmation message
  refactor: share logic between 4d3d3d3 and flarhgunnstow
  style: convert tabs to spaces
  test: ensure Tayne retains clothing
  

  (Source)
• Make sure you have added the necessary tests for your changes.
• Run all the tests with make test to assure nothing else was accidentally broken (it will build the docker container and run go test)

Questions?

Are there any questions or is something not clear enough? Simply open up a ticket or send me an email :)

mongodm's Issues

Fatal error, mongodb initialization failed: open /home/user/work/pkg/mod/github.com/zebresel-com/[email protected]+incompatible/locals.json: no such file or directory"

I have built an executable of my go application which uses zebresel-com/mongodm and am trying to deploy it in docker container, but I am getting the following error.It is working great on my linux box but it fails when I try to deploy it inside docker.
" initialization failed: open /home/myuser/work/pkg/mod/github.com/zebresel-com/[email protected]+incompatible/locals.json: no such file or directory"
Steps to reproduce:

1. Create a go executable of work repo which is using zebresel-com/mongodm
2. Copy the executable in docker env via Docker copy command through Dockerfile

Assigning a 1n relation

currently i have a model that is as the following

	ConfirmedGuests      []interface{} `model:"Guest" relation:"1n" autosave:"true"`

In the flow that I have in my backend, I get the data for the guests and the data for the model that ConfirmedGuests lives in. I first create all guests and put their bson.ObjectId's in a slice. then I wish to assign them to ConfirmedGuests. But since ConfirmedGuests is of type []interface{} and my guests slice is of type []bson.ObjectId I cannot assign them when creating the struct ConfirmedGuests lives in. Is there any way of assigning this through a clever cast or something ?

Regards, Duck

Slices always viewed as relations

Hello there,

I was wondering if there is a possible bug with how the relation tag is handled. I have an Email model that holds the sender's email and a slice of recipients. I only want to store the recipient's email, and not the entire user object. The model is written as follows:

type Email struct {
	mongodm.DocumentBase `json:",inline" bson:",inline"`
	From	string `json:"from" bson:"from"`
	To 	[]string `json:"to" bson:"to"`
}

I am failing validation on the To field with validation.field_invalid_relation1n . Looking through the document_base.go file I found the following code:

if fieldValue.Kind() == reflect.Slice && relationTag != REL_1N {
  self.AppendError(&validationErrors, L("validation.field_invalid_relation1n", validationName))
} else if fieldValue.Kind() != reflect.Slice && relationTag == REL_1N {
  self.AppendError(&validationErrors, L("validation.field_invalid_relation11", validationName))
}

Shouldn't there be another check inside the if block to see if relationTag != ""? Without it, every slice would have to be a relation even though it doesn't need to be.

Need to add `sparse` annotation support to model field

Problem:

type ModelA struct {
	mongodm.DocumentBase `json:",inline" bson:",inline"`

	Target bson.ObjectId `json:"target" bson:"target"`
}

if create a ModelA but leave target field empty, when try to save the document, error occured:
ObjectIDs must be exactly 12 bytes long (got 0). So, maybe need add something like this:

Target bson.ObjectId `sparse:"true"`

?

1n relation. saving objectId's and populating them

Currently, I have an array of object id's which are coming in from the frontend to my backend API. I wish to save these as ObjectId's in the backend so i can later populate them when i need to send the whole object back to the frontend. But as it currently stands, if i save the raw objectId's, it will save them as type String and not ObjectId. Is there a way around this ?

Many thanks

Expanding mongodm.Config

Hi there!

I was reading up on mongodm.Config, and noticed that SetSocketTimeOut, SetSafe, and SetMode were hardcoded instead of allowing users to pass them in via mongodm.Config. Are there any plans to include this functionality in future releases? If you accept PRs, I'd be more than happy to add that in.

Change license to MIT

.. would be more community friendly. Any other suggestions?

Getting Error does not implement DefaultValidate

Hi, I am new with golang and I tried to use your library and I get the following error

middlewares/main.go:76: cannot use model (type models.Model) as type mongodm.IDocumentBase in argument to ModelCollection.New:
	models.Model does not implement mongodm.IDocumentBase (DefaultValidate method has pointer receiver)

I have emmbeded the struct mongodm.DocumentBase on my definition, what is my problem ?

Populating on .Find()

Currently when I call the .Find() method on a model, and follow that up with a .Populate(), I get the following error: Given models were not correctly initialized with 'DocumentBase' interface type.

This is weird as calling .Populate() on the same model but instead of .Find() I use .FindId() it works ?

Virtuals and Hooks (like in mongoose)

According to Todos, I will spend some days to work on virtuals and hooks (like in mongoose) task.

Put docker infos back to readme?

Are you sure you want to keep the informations related to docker into the contribute file?

It's a bit unnatural to look for these infos there.

Just a thought

using Documentbase.Update()

I am trying to update a defined model with the following piece of code

	connection := context.MustGet("dbConnection").(*mongodm.Connection)
	Reservation := connection.Model("reservation")
	reservation := &models.Reservation{}

	err := context.BindJSON(r)
	if err != nil {
		context.AbortWithStatusJSON(500, apierrors.JsonBindingError)
		return
	}

	err = Reservation.FindId(id).Exec(reservation)
	if err != nil {
		context.AbortWithStatusJSON(500, apierrors.DBLookupFailedError)
	}

	err, _ = reservation.Update(bson.M{
		"IsSharedRoom":      r.IsSharedRoom,
		"Beds":              r.Beds,
		"PartySize":         r.PartySize,
		"notes":             r.Notes,
		"ConfirmedGuest":    r.ConfirmedGuest,
		"RoomType":          r.RoomType,
		"CheckinDate":       r.CheckinDate,
		"CheckoutDate":      r.CheckoutDate,
		"ReservationNumber": r.ReservationNumber,
	})

however, when this code is ran, nothing in the database is updated on the model, is there something I am missing ? I also could not find any documentation on how to use this function other than that it can be used on an IDocumentBase

Regards Duck

How to create index on model?

Index topic not found in any code, issue or document...

Allow nested fields to be populated

Currently it is not possible to populate a field in a model that is not top level.
Not allowing this sort of defeats the perpose of mongodb as it does not allow for proper nesting.

A solution for this would be to allow dot notated routes for structures.

mongodb3.0 SCRAM-SHA-1?

Database connection error: %v server returned error on SASL authentication step: Authentication failed.

About contributing

In my opinion, For more developers to contributing code to this repository easily like most open source project, I think you should do something as below:

1. Dockerized. To unified environment of development.
2. Writing test. To make CI easily.
3. Building by CI.
4. Create a CONTRIBUTE.md. To guide those guys who want to contribute to this repository.

BTW, I can submit PRs for step 1-3, let me know your decision by reply, thank you.

find by relation

how to find by relation?

Replace unmaintained mgo dependency with community fork

For details see https://github.com/go-mgo/mgo