Mongoproxy
A server that speaks the MongoDB wire protocol, and can analyze and transform requests and responses to and from a client and a MongoDB server instance. All requests go through a module pipeline, and modules include a proxy to send requests to mongod
, as well as a module to collect real-time analytics about the requests passing through the server.
Building and running
To grab dependencies (should not be necessary, as dependencies are checked into the repository):
chmod 755 ./vendor.sh # only needs to be done once
./vendor.sh
Note that currently, mongoproxy requires a specific fork of mgo because it requires some features not yet pulled into the main repo.
To run:
chmod 755 ./start.sh # only needs to be done once
./start.sh <options>
start.sh
sets up the go path, and runs the main/server.go
file. It's equivalent to the following two commands:
. ./set_gopath.go
go run main/server.go
Configurations
Configurations tell the server which modules to load and run, and also specifies configurations for each module. They can either be a document in a MongoDB instance (as BSON), or a JSON file in the local directory.
By default, the server expects a mongod
instance to be running on localhost:27017
with a configuration document in the test.config
collection. The location for the configuration document can be set as a command line option, and can also be a file.
Configurations have one field modules
, which is an array. Each object in the array has a name
field for the name of the module, and a config
field for the module's configuration.
A configuration can be found in the project directory named example_bi_config.json
, which is run with the following command:
./start.sh -f example_bi_config.json
Command Line Options
-port Port number to run the server on. Defaults to 8124.
-logLevel Sets verbosity of the logs from 1 to 5, with 1 being least verbose and 5 being the most. Defaults to 3.
-m URL of a mongod server to connect to to retrieve configuration information from. Defaults to localhost:27017
-c Namespace of the collection in the mongod server to retrieve configuration information from. Defaults to test.config
-f Path to a configuration file. If set, the m and c flags are ignored.
Tests
To run unit tests:
chmod 755 ./test.sh
./test.sh
To run integration tests:
# single test
node tests/test <js file to test>
# a directory full of test files
node tests/test_dir <directory of files to test>
Modules
The only thing that the server does to requests is to translate them into a Go struct. Everything else happens in a module.
Usage
The server can use modules that are defined in its configuration.
All modules have a unique name to identify them, which is also used in the name
field of the configuration. Requests are passed from module to module in the order defined in the configuration, and the response sent to the server, and finally back up to the client.
The following modules are implemented and included in the source:
mockule A mock module that stores insert requests in memory and can dump them back out. It also pretends it is a 1-node replica set.
mongod A module that forwards the request to a MongoDB instance and passes back the response to the server.
bi A module with pre-configured rules that analyzes requests and aggregates them into metrics.
Developing Modules
All modules implement the Module
interface, defined in server/modules.go
. Configure()
is called at the server startup, and Process(req, res, next)
is called every time a request passes through the server.
A module is responsible for calling the next module in the pipeline via the next
argument in the Process
function, which is a function that takes two arguments: a request and a response.
Modules also have to be added to the registry in order for the server to know they exist. Each module should live in their own package, and have an init
function with the following line:
server.Publish(<Module>)
where server
is the imported github.com/mongodbinc-interns/mongoproxy/server
package.
Then, in the server/registry.go
file, add the import path of the module to the file preceded by an underscore, to add the module to the registry.
Example Module
package examplemodule
import (
"github.com/mongodbinc-interns/mongoproxy/messages"
"github.com/mongodbinc-interns/mongoproxy/server"
"gopkg.in/mgo.v2/bson"
)
type ExampleModule struct {
}
func init() {
server.Publish(ExampleModule{})
}
func (m ExampleModule) Name() string {
return "find-only"
}
func (m ExampleModule) Configure(config bson.M) error {
// configure the module here. Returns an error if configuration fails
return nil
}
// this module will drop all requests except for Find requests
func (m ExampleModule) Process(req messages.Requester, res messages.Responder,
next server.PipelineFunc) {
switch req.Type() {
case messages.FindType:
// send the request and response to the next module
next(req, res)
default:
res.Write(<Request dropped>)
return
}
This module can then be run with a configuration like the following:
{
"modules": [
...
{
"name": "find-only",
"config": {}
},
...
]
}