reverb / wordnik-oss Goto Github PK

Been banging my head all day trying to figure out. if i kick off the incremental util then run several inserts and create a stop.txt i get something like this in the console:

including* learn.BigGreenFurryCreaturesFromMars
oplog: 1 records, 0 req/sec, 0 skips
including* learn.BigGreenFurryCreaturesFromMars
including* learn.BigGreenFurryCreaturesFromMars
including* learn.BigGreenFurryCreaturesFromMars
found stop file, exiting

At this moment the process is still running, not until i run one more insert that the process actually dies and the console then reads

including* learn.BigGreenFurryCreaturesFromMars
oplog: 1 records, 0 req/sec, 0 skips
including* learn.BigGreenFurryCreaturesFromMars
including* learn.BigGreenFurryCreaturesFromMars
including* learn.BigGreenFurryCreaturesFromMars
found stop file, exiting
including* learn.BigGreenFurryCreaturesFromMars
oplog: 5 records, 0 req/sec, 0 skips
exiting thread

Does this make sense? Is this the intended behavior?

Thanks!

This is the result of running maven against mongo-admin-utils:

$ mvn
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Building mongo-admin-utils
[INFO] task-segment: [install]
[INFO] ------------------------------------------------------------------------
[INFO] [enforcer:enforce {execution: enforce-maven}]
[INFO] [resources:resources {execution: default-resources}]
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /home/anchor/new-dumps/wordnik-wordnik-oss-6b46f20/modules/mongo-admin-utils/src/main/resources
[INFO] [compiler:compile {execution: default-compile}]
[INFO] Compiling 8 source files to /home/anchor/new-dumps/wordnik-wordnik-oss-6b46f20/modules/mongo-admin-utils/target/classes
[INFO] ------------------------------------------------------------------------
[ERROR] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Compilation failure
IncrementalBackupUtil.java:[47,84] cannot access scala.Option
class file for scala.Option not found
OplogTailThread thd = new OplogTailThread(util, MongoDBConnectionManager.getOplog("oplog", DATABASE_HOST, DATABASE_USER_NAME, DATABASE_PASSWORD).get());

[INFO] ------------------------------------------------------------------------
[INFO] For more information, run Maven with the -e switch
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 2 seconds
[INFO] Finished at: Sat May 05 17:25:16 EDT 2012
[INFO] Final Memory: 35M/440M
[INFO] ------------------------------------------------------------------------

RestoreUtil fails to restore BSON dump which contains single DBRef in a document.

Steps to reproduce:

1. Insert following sample documents into new collection:
   db.user.insert( { name : "John Doe" } ); var user = db.user.findOne( { name : "John Doe" } ); db.user.insert( { name : "Jane Doe" , father : DBRef("user", user._id) } );
2. Make a BSON dump of user collection:
   mongodump -d test -o ~/tmp/
3. Invoke RestoreUtil passing the created dump:
   ./bin/run.sh com.wordnik.system.mongodb.RestoreUtil -i ~/tmp/test/ -D -d test

The last step finishes with following exception:

restoring file user.bson to collection user java.lang.IllegalArgumentException: fields stored in the db can't start with '$' (Bad Key: '$ref') at com.mongodb.DBCollection.validateKey(DBCollection.java:1444) at com.mongodb.DBCollection._checkKeys(DBCollection.java:1425) at com.mongodb.DBCollection._checkKeys(DBCollection.java:1415) at com.mongodb.DBCollection._checkObject(DBCollection.java:1400) at com.mongodb.DBCollection.save(DBCollection.java:799) at com.mongodb.DBCollection.save(DBCollection.java:785) at com.wordnik.system.mongodb.RestoreUtil.write(RestoreUtil.java:162) at com.wordnik.system.mongodb.RestoreUtil.restore(RestoreUtil.java:129) at com.wordnik.system.mongodb.RestoreUtil.run(RestoreUtil.java:75) at com.wordnik.system.mongodb.RestoreUtil.main(RestoreUtil.java:61)

Possible solution:

Use DefaultDBDecoder's method decode instead of readObject at line 125 (in this case MongoDB Java API correctly reconstructs objects with DBRefs from BSON dump).

Looks like the Incremental Backup tool does not gzip the output files even tho I am using the -Z option.

Right now it is not documented how to do an initial backup with the snapshot tool then start the incremental backup with this second tool.

Talking with Tony on the IRC he suggested getting the last timestamp from the oplog _id field from the time the snapshot was made.

Would be nice if the snapshot tool could do that for us, maybe as a command line argument, to save the trouble of manually querying the database for that.

Hi!

I actually was having an issue with restoring compressed files (made with the snapshot util) but i was in the process of decommissioning a server and bringing up a new one so thought I would log that issue after the new server was up. BUT, I was bringing the new server up today and unfortunately was not able to build the project. I was hoping someone could give me a pointer as to why the build failed. Maybe something on our end.....not sure.

maven info:
Apache Maven 3.0.4 (r1232337; 2012-01-17 08:44:56+0000)
Maven home: /usr/local/apache-maven/apache-maven-3.0.4
Java version: 1.6.0_31, vendor: Sun Microsystems Inc.
Java home: /usr/java/jdk1.6.0_31/jre
Default locale: en_US, platform encoding: ANSI_X3.4-1968
OS name: "linux", version: "2.6.18-308.8.2.el5.centos.plusxen", arch: "amd64", family: "unix"

error info:
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] wordnik-oss-project ............................... SUCCESS [8.916s]
[INFO] common-utils ...................................... SUCCESS [6.635s]
[INFO] mongo-utils ....................................... FAILURE [3.843s]
[INFO] mongo-admin-utils ................................. SKIPPED
[INFO] mongo-admin-utils-distribution .................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 21.995s
[INFO] Finished at: Fri Sep 07 01:23:47 UTC 2012
[INFO] Final Memory: 9M/25M

.......

T E S T S

Running com.wordnik.test.mongo.connection.ConnectionManagerTest
Sep 7, 2012 1:23:46 AM com.mongodb.DBTCPConnector fetchMaxBsonObjectSize
WARNING: Exception determining maxBSON size using0
java.io.IOException: couldn't connect to --edited our IP-- bc:java.net.ConnectException: Connection refused
at com.mongodb.DBPort._open(DBPort.java:228)

I don't understand the test failures because we are not using all the ports in the output. We are not sharding, just one replica set.

Any ideas as to why this happens? I can provide more info if needed.

Thanks!

Hi! For swagger-play2-utils project we need 2.11 build of com.wordnik#common-utils, otherwise 2.11 build of swagger-play2-utils if blocked

I am attempting to run an incremental backup against a small oplog and the process never completes. A jstack has the following in the output and I noticed that the "Finalizer" and "Reference Handler" are in a waiting state. Can you direct me to what it is waiting on and ultimately how to get it to close?

Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.12-b01-434 mixed mode):

"Attach Listener" daemon prio=9 tid=7fdd211e7000 nid=0x11056c000 waiting on condition [00000000]
java.lang.Thread.State: RUNNABLE

"DestroyJavaVM" prio=5 tid=7fdd212f7800 nid=0x1065c8000 waiting on condition [00000000]
java.lang.Thread.State: RUNNABLE

"Thread-5" prio=5 tid=7fdd212f6800 nid=0x1102ce000 waiting on condition [1102cd000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at com.wordnik.system.mongodb.StopFileMonitor.run(StopFileMonitor.java:14)

"oplog" prio=5 tid=7fdd212f6000 nid=0x1101cb000 runnable [1101ca000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
- locked <7bc4461c8> (a java.io.BufferedInputStream)
at org.bson.io.Bits.readFully(Bits.java:35)
at org.bson.io.Bits.readFully(Bits.java:28)
at com.mongodb.Response.(Response.java:39)
at com.mongodb.DBPort.go(DBPort.java:128)
- locked <7bc438ad8> (a com.mongodb.DBPort)
at com.mongodb.DBPort.call(DBPort.java:79)
at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:218)
at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:189)
at com.mongodb.DBApiLayer$Result._advance(DBApiLayer.java:452)
at com.mongodb.DBApiLayer$Result.hasNext(DBApiLayer.java:418)
at com.mongodb.DBCursor._hasNext(DBCursor.java:509)
at com.mongodb.DBCursor.hasNext(DBCursor.java:529)
at com.wordnik.system.mongodb.OplogTailThread.run(OplogTailThread.java:143)

"MongoCleaner1234403385" daemon prio=5 tid=7fdd212f3000 nid=0x1100c8000 waiting on condition [1100c7000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at com.mongodb.Mongo$DBCleanerThread.run(Mongo.java:681)

"ReplicaSetStatus:Updater" daemon prio=5 tid=7fdd212f2800 nid=0x10ffc5000 waiting on condition [10ffc4000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at com.mongodb.ReplicaSetStatus$Updater.run(ReplicaSetStatus.java:428)

"MongoCleaner1751161119" daemon prio=5 tid=7fdd26069800 nid=0x10fe84000 waiting on condition [10fe83000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at com.mongodb.Mongo$DBCleanerThread.run(Mongo.java:681)

"Low Memory Detector" daemon prio=5 tid=7fdd2501a800 nid=0x10f9e9000 runnable [00000000]
java.lang.Thread.State: RUNNABLE

"C2 CompilerThread1" daemon prio=9 tid=7fdd25019800 nid=0x10f8e6000 waiting on condition [00000000]
java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" daemon prio=9 tid=7fdd25019000 nid=0x10f7e3000 waiting on condition [00000000]
java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" daemon prio=9 tid=7fdd25018000 nid=0x10f6e0000 runnable [00000000]
java.lang.Thread.State: RUNNABLE

"Surrogate Locker Thread (Concurrent GC)" daemon prio=5 tid=7fdd25017000 nid=0x10f5dd000 waiting on condition [00000000]
java.lang.Thread.State: RUNNABLE

"Finalizer" daemon prio=8 tid=7fdd211dd800 nid=0x10f3d3000 in Object.wait() [10f3d2000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <7bb401300> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
- locked <7bb401300> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

"Reference Handler" daemon prio=10 tid=7fdd211dc800 nid=0x10f2d0000 in Object.wait() [10f2cf000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <7bb4011d8> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:485)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
- locked <7bb4011d8> (a java.lang.ref.Reference$Lock)

"VM Thread" prio=9 tid=7fdd211d8000 nid=0x10f1cd000 runnable

"Gang worker#0 (Parallel GC Threads)" prio=9 tid=7fdd21001800 nid=0x1099cb000 runnable

"Gang worker#1 (Parallel GC Threads)" prio=9 tid=7fdd21002000 nid=0x109ace000 runnable

"Gang worker#2 (Parallel GC Threads)" prio=9 tid=7fdd21002800 nid=0x109bd1000 runnable

"Gang worker#3 (Parallel GC Threads)" prio=9 tid=7fdd21003000 nid=0x109cd4000 runnable

"Gang worker#4 (Parallel GC Threads)" prio=9 tid=7fdd21004000 nid=0x109dd7000 runnable

"Gang worker#5 (Parallel GC Threads)" prio=9 tid=7fdd21004800 nid=0x109eda000 runnable

"Gang worker#6 (Parallel GC Threads)" prio=9 tid=7fdd21005000 nid=0x109fdd000 runnable

"Gang worker#7 (Parallel GC Threads)" prio=9 tid=7fdd21005800 nid=0x10a0e0000 runnable

"Concurrent Mark-Sweep GC Thread" prio=9 tid=7fdd210e3000 nid=0x10ee1f000 runnable
"Gang worker#0 (Parallel CMS Threads)" prio=9 tid=7fdd210e2000 nid=0x10e419000 runnable

"Gang worker#1 (Parallel CMS Threads)" prio=9 tid=7fdd210e2800 nid=0x10e51c000 runnable

"VM Periodic Task Thread" prio=10 tid=7fdd211ef000 nid=0x10faec000 waiting on condition

"Exception Catcher Thread" prio=10 tid=7fdd24001800 nid=0x1067f3000 runnable
JNI global references: 1091

Hi,

The process just stalls when using ReplayUtil to restore data collected using the following command.

./bin/run.sh com.wordnik.system.mongodb.IncrementalBackupUtil -o /root/mongoincre mongoserver -s 100

There are 74 oplog files in

ls /root/mongoincre/|wc -l
75

Now I restore the data

./bin/run.sh com.wordnik.system.mongodb.ReplayUtil -i /root/mongoincre/ -h localhost
replaying file oplog.0066.bson
inserts: 2,531, updates: 2,026, deletes: 266, skips: 0 (482 req/sec)
inserts: 3,382, updates: 3,842, deletes: 294, skips: 0 (713 req/sec)
inserts: 3,413, updates: 4,471, deletes: 300, skips: 0 (801 req/sec)

It does not exit gracefully

The issues I face are

1. I could see the process stalls and consumed entire cpu.
2. It does the process only on one oplog file. There are another 73 in which the operation did not happen.

My Java

java -version
java version "1.6.0_16"
Java(TM) SE Runtime Environment (build 1.6.0_16-b01)
Java HotSpot(TM) 64-Bit Server VM (build 14.2-b01, mixed mode)

First off, thank you for this amazing toolset. I know people have been using it successfully and I was having great hopes to use it with our latest mongo 2.2.2.

This ticket is just to acknowledge the following and warn people when using this utility with the latest versions of mongo (2.2.1 and above).

The following mongo issue has caused most mongo drivers (except the native C++ one) to break when attempting to read the oplog:

https://jira.mongodb.org/browse/SERVER-1606

This includes the mongo-java-driver that this project uses. We found this out recently while trying to migrate data from one server to another and realizing that we are missing all $set operations.

Note that in https://jira.mongodb.org/browse/DOCS-923, LazyDBObject is mentioned as a working alternative but in our local tests, we have yet to manage to find a viable way of using it.

Have you guys switched to the latest mongo versions in-house? If so, what has your experience been using this toolset?

when I make a snashot,that's some problems

t_Item: 87.02% (19.51mb/s)
t_Item: 93.75% (19.59mb/s)
t_User: 1.99% (13.05mb/s)
Jun 30, 2011 9:17:37 AM com.mongodb.DBTCPConnector$MyPort error
SEVERE: MyPort.error called
java.lang.ArrayIndexOutOfBoundsException: -117
at org.bson.BSONDecoder$Input.readCStr(BSONDecoder.java:315)
at org.bson.BSONDecoder.decodeElement(BSONDecoder.java:90)
at org.bson.BSONDecoder.decodeElement(BSONDecoder.java:174)
at org.bson.BSONDecoder.decode(BSONDecoder.java:71)
at org.bson.BSONDecoder.decode(BSONDecoder.java:55)
at org.bson.BSONDecoder.decode(BSONDecoder.java:42)
at com.mongodb.Response.(Response.java:63)
at com.mongodb.DBPort.go(DBPort.java:101)
at com.mongodb.DBPort.go(DBPort.java:66)
at com.mongodb.DBPort.call(DBPort.java:56)
at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:211)
at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:195)
at com.mongodb.DBApiLayer$Result._advance(DBApiLayer.java:391)
at com.mongodb.DBApiLayer$Result.hasNext(DBApiLayer.java:374)
at com.mongodb.DBCursor._hasNext(DBCursor.java:425)
at com.mongodb.DBCursor.hasNext(DBCursor.java:445)
at com.wordnik.system.mongodb.SnapshotUtil$SnapshotThread.run(SnapshotUtil.java:199)
java.lang.ArrayIndexOutOfBoundsException: -117
at org.bson.BSONDecoder$Input.readCStr(BSONDecoder.java:315)
at org.bson.BSONDecoder.decodeElement(BSONDecoder.java:90)
at org.bson.BSONDecoder.decodeElement(BSONDecoder.java:174)
at org.bson.BSONDecoder.decode(BSONDecoder.java:71)
at org.bson.BSONDecoder.decode(BSONDecoder.java:55)
at org.bson.BSONDecoder.decode(BSONDecoder.java:42)
at com.mongodb.Response.(Response.java:63)
at com.mongodb.DBPort.go(DBPort.java:101)
at com.mongodb.DBPort.go(DBPort.java:66)
at com.mongodb.DBPort.call(DBPort.java:56)
at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:211)
at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:195)
at com.mongodb.DBApiLayer$Result._advance(DBApiLayer.java:391)
at com.mongodb.DBApiLayer$Result.hasNext(DBApiLayer.java:374)
at com.mongodb.DBCursor._hasNext(DBCursor.java:425)
at com.mongodb.DBCursor.hasNext(DBCursor.java:445)
at com.wordnik.system.mongodb.SnapshotUtil$SnapshotThread.run(SnapshotUtil.java:199)

Hi,
I think that the reapply of the oplogs from the incremetanl backup couldn't be reapply for a specific database. Correct ?
Should not useful to be able to tell which database you want to 'recover' with a parameter like -d <database_name> or -d <database_name1>, <database_name2>

Regards,

Hi,
While i am trying to compile i got this error. Can you help me figure out why this is happening?

./sbt compile
Detected sbt version 0.12.2
Downloading sbt launcher 0.12.2:
From http://typesafe.artifactoryonline.com/typesafe/ivy-releases/org.scala-sbt/sbt-launch/0.12.2/sbt-launch.jar
To ./.lib/0.12.2/sbt-launch.jar
Using /home/ismet/.sbt/0.12.2 as sbt dir, -sbt-dir to override.
[info] Loading project definition from /home/ismet/wordnik-oss/project
[info] Updating {file:/home/ismet/wordnik-oss/project/}default-ae286f...
[info] Resolving org.scala-sbt#precompiled-2_10_0;0.12.2 ...
[info] Done updating.
[info] Compiling 4 Scala sources to /home/ismet/wordnik-oss/project/target/scala-2.9.2/sbt-0.12/classes...
[info] Set current project to wordnik-oss (in build file:/home/ismet/wordnik-oss/)
[info] Defining /:log-level
[info] The new value will be used by no settings or tasks.
[info] Reapplying settings...
[info] Set current project to wordnik-oss (in build file:/home/ismet/wordnik-oss/)
[info] Defining /:console-project::trace-level, /:console::trace-level and 2 others.
[info] The new values will be used by no settings or tasks.
[info] Run last for details.
[info] Reapplying settings...
[info] Set current project to wordnik-oss (in build file:/home/ismet/wordnik-oss/)
[info] Updating {file:/home/ismet/wordnik-oss/}mongo-utils...
[info] Updating {file:/home/ismet/wordnik-oss/}common-utils...
[info] Resolving com.typesafe.akka#akka-actor;2.0.5 ...
[warn] module not found: com.typesafe.akka#akka-actor;2.0.5
[warn] ==== local: tried
[warn] /home/ismet/.ivy2/local/com.typesafe.akka/akka-actor/2.0.5/ivys/ivy.xml
[warn] ==== public: tried
[warn] http://repo1.maven.org/maven2/com/typesafe/akka/akka-actor/2.0.5/akka-actor-2.0.5.pom
[info] Resolving log4j#log4j;1.2.17 ...
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: UNRESOLVED DEPENDENCIES ::
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: com.typesafe.akka#akka-actor;2.0.5: not found
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[info] Resolving log4j#log4j;1.2.17 ...
[info] Done updating.
[info] Updating {file:/home/ismet/wordnik-oss/}mongo-admin-utils...
[info] Resolving org.scala-lang#scala-library;2.9.2 ...
[info] Compiling 2 Scala sources and 1 Java source to /home/ismet/wordnik-oss/modules/mongo-utils/target/scala-2.9.2/classes...
[info] Resolving log4j#log4j;1.2.17 ...
[info] Done updating.
[warn] warning: [options] bootstrap class path not set in conjunction with -source 1.6
[warn] 1 warning
[info] Compiling 2 Scala sources and 19 Java sources to /home/ismet/wordnik-oss/modules/mongo-admin-utils/target/scala-2.9.2/classes...
[warn] warning: [options] bootstrap class path not set in conjunction with -source 1.6
[warn] 1 warning
sbt.ResolveException: unresolved dependency: com.typesafe.akka#akka-actor;2.0.5: not found
at sbt.IvyActions$.sbt$IvyActions$$resolve(IvyActions.scala:214)
at sbt.IvyActions$$anonfun$update$1.apply(IvyActions.scala:122)
at sbt.IvyActions$$anonfun$update$1.apply(IvyActions.scala:121)
at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:114)
at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:114)
at sbt.IvySbt$$anonfun$withIvy$1.apply(Ivy.scala:102)
at sbt.IvySbt.liftedTree1$1(Ivy.scala:49)
at sbt.IvySbt.action$1(Ivy.scala:49)
at sbt.IvySbt$$anon$3.call(Ivy.scala:58)
at xsbt.boot.Locks$GlobalLock.withChannel$1(Locks.scala:75)
at xsbt.boot.Locks$GlobalLock.withChannelRetries$1(Locks.scala:58)
at xsbt.boot.Locks$GlobalLock$$anonfun$withFileLock$1.apply(Locks.scala:79)
at xsbt.boot.Using$.withResource(Using.scala:11)
at xsbt.boot.Using$.apply(Using.scala:10)
error sbt.ResolveException: unresolved dependency: com.typesafe.akka#akka-actor;2.0.5: not found
[error] Total time: 8 s, completed Mar 11, 2013 10:29:10 PM

Its always painful to explicitly exclude a logging-framework dependency from a library, so please only depend on slf4j-api. The selection of the logging framework should be made by the user of the library (ie. the application). If you need the slf4j-log4j binding for logging during unit-tests, bind it to test scope.

Taken from www.slf4j.org/manual.html:
"Please note that embedded components such as libraries or frameworks should not declare a dependency on any SLF4J binding but only depend on slf4j-api."

kind regards

I am getting the following exception on all my oplog files when I use the ReplayUtil.

java.io.IOException: unexpected EOF
at org.bson.BasicBSONDecoder$BSONInput._need(BasicBSONDecoder.java:327)
at org.bson.BasicBSONDecoder$BSONInput.readInt(BasicBSONDecoder.java:340)
at org.bson.BasicBSONDecoder._decode(BasicBSONDecoder.java:74)
at org.bson.BasicBSONDecoder.decode(BasicBSONDecoder.java:57)
at org.bson.BasicBSONDecoder.readObject(BasicBSONDecoder.java:42)
at com.wordnik.system.mongodb.ReplayUtil.run(ReplayUtil.java:154)
at com.wordnik.system.mongodb.ReplayUtil.main(ReplayUtil.java:70)

I am restoring a snapshot and then replaying five oplogs. I have programmatically verified that the data is being restored without gaps, so I am confident the data is there.

Have you seen that? Do you have any suggestions for me?

Here is a link to one of the oplogs.
http://static.integrityops.net.s3.amazonaws.com/1381330862_oplog.bson.gz

I mistakenly thought this project had been abandoned and made some mods to the code and republished at https://github.com/qbert65536/mongo-incremental-backuper .

The only real changes are 1) looking for oplog.rs instead of $main and 2) IncrementalBackupUtil no longer tails the oplog , it runs once and quits.

I am working on a MongoDB implementation, and I am trying to use your tools. They are exactly what I am looking for, but I have having a problem. I need to backup multiple databases.

This creates a problem in the stopping for the oplog monitor. Touching the stop.txt file in the CWD doesn't work that well since I really need to be able to stop the oplog monitor for individual databases. I would even settle for stopping all, but since it deletes the stop file after it stops, I would have to keep adding a file and checking for whether or not it deletes it within a second or so to see if there are any more left.

It seems like it would be better if it responded to signals or if the stop file could go in the output directory.

Hi,

Thank you for this awesome tool!
I have an issue when trying to use SnapshotUtil for a MongoLab database instance (https://mongolab.com/). I'm not sure exactly what's wrong, but here is the exception:

ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console.
Jun 25, 2015 10:00:29 PM com.wordnik.mongo.connection.MongoDBConnectionManager$ getConnection
SEVERE: can't get connection to dsXXXXXX.mongolab.com:31XXX/XXXXX with username Some(XXXXX), password XXXXXXXXX
Exception in thread "main" java.lang.RuntimeException: com.wordnik.mongo.connection.PersistenceException
at com.wordnik.system.mongodb.SnapshotUtil.getCollections(SnapshotUtil.java:161)
at com.wordnik.system.mongodb.SnapshotUtil.run(SnapshotUtil.java:70)
at com.wordnik.system.mongodb.SnapshotUtil.main(SnapshotUtil.java:65)
at com.epickur.api.dump.MongoDBDump.exportMongo(MongoDBDump.java:114)
at com.epickur.api.cron.MongoDBDumpJob.execute(MongoDBDumpJob.java:26)
at com.epickur.api.cron.MongoDBDumpJob.main(MongoDBDumpJob.java:44)
Caused by: com.wordnik.mongo.connection.PersistenceException
at com.wordnik.mongo.connection.MongoDBConnectionManager$.getConnection(MongoDBConnectionManager.scala:147)
at com.wordnik.mongo.connection.MongoDBConnectionManager$.getConnection(MongoDBConnectionManager.scala:93)
at com.wordnik.mongo.connection.MongoDBConnectionManager.getConnection(MongoDBConnectionManager.scala)
at com.wordnik.system.mongodb.SnapshotUtil.getCollections(SnapshotUtil.java:118)
... 5 more

Any suggestions?
Regards,
Carl