thelastpickle / cassandra-reaper Goto Github PK
View Code? Open in Web Editor NEWAutomated Repair Awesomeness for Apache Cassandra
Home Page: http://cassandra-reaper.io/
License: Apache License 2.0
Automated Repair Awesomeness for Apache Cassandra
Home Page: http://cassandra-reaper.io/
License: Apache License 2.0
Hi, we'd like to use your project internally and it's much easier if we can retrieve your build artifacts from Maven Central. Happy to help with a pull request and help get you setup in Maven Central if you need a hand with this.
nodetool repair
successfully.Don't want to swamp you with more details or unnecessary logs, especially as I'd have to sanitize them before sending them out, so please let me know if there is anything else I can provide, and I'll do my best to get it to you.
I was trying to integrate authentication with cassandra and noticed that the version of dropwizard-cassandra being used utilized java.util.Optional rather than Guava's Optional. Because the version of dropwizard being used in the project doesn't automatically add these serializers to the object mapper, we need to explicitly add them during bootstrap.
During repair run we noticed big network spikes on one node in cluster. This was due to communication between monitoring machine and this single node, to get notifications about ongoing repairs.
This node is added as seed to Reaper through UI. There should be possibility to add multiple seeds when defining cluster so network load for communication can be spread across multiple nodes.
It's confusing that this can't be configured in the UI. Let's add this to the UI and make it available per-cluster.
Let's be sure to encrypt the password in the DB, and the config file can specify a path to the encryption key. Any suggestions on this, @zznate ?
The forceRepairRangeAsync and forceRepairAsync methods have been deprecated for a long time now in the Cassandra codebase.
We need to switch to repairAsync(String keyspace, Map<String, String> options) which is the replacement for all of them, in order to have a reliable and simpler way to trigger repair on various Cassandra versions.
This will help us reduce the various checks we make to determine which repair method should be used and remove a few LoC from the JmxProxy.
When the issue in dropwizard cassandra component is fixed, the dependency should be updated.
When repair is started with "Incremental repair : false" the value displayed on UI (and the value stored in back end storage) denotes number of nodes in the cluster instead the number defined in configuration with segmentCount
property.
It appears that the problem is in CommonTools class at line:
RepairRun repairRun = storeNewRepairRun(context, cluster, repairUnit, cause, owner, nodes.keySet().size(), repairParallelism, intensity);
where nodes.keySet().size()
is passed as a value of int segments
parameter. I assume, in case when incrementarRepair
is false
, that tokenSegments.size()
should be passed instead.
If this is, indeed, a bug, I can fix it. I would like to start contributing to the cassandra-reaper.
Also, what is the role of segmentCount
property when incremental repairs are run if that number is just number of nodes? Or I am missing something?
move most of readme to the wiki
I'd like to get a 0.6.0 release to take advantage of the dropwizard version upgrade.
From IRC:
ha ha, yeah been thinking about it. Still deciding. We might fork it and tweak some of the code to basically to hook custom listeners/callbacks on the success/failure cases. Basically trigger off a slack alert if the repair failed
It would be very cool if we shipped support for pluggable classes that can register as observers to handle things like failure, etc.
┆Issue is synchronized with this Jira Task by Unito
┆Issue Number: K8SSAND-267
In the configuration section you are linking to
http://dropwizard.io/manual/configuration.html
This link returns 404.
The UI stores the string description value of the parallelism option when creating a repair, but when Reaper tries to retrieve the repair from the database, it fails with the following stack trace:
ERROR [2016-12-14 20:13:30,824] [dw-22 - GET /repair_run] i.d.j.e.LoggingExceptionMapper - Error handling a request: 18b1eccefcda9f42 java.lang.IllegalArgumentException: No enum constant org.apache.cassandra.repair.RepairParallelism.parallel at java.lang.Enum.valueOf(Enum.java:238) ~[na:1.8.0_102] at org.apache.cassandra.repair.RepairParallelism.valueOf(RepairParallelism.java:26) ~[cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT] at com.spotify.reaper.storage.postgresql.RepairRunMapper.map(RepairRunMapper.java:41) (deleted the rest)
I can (temporarily) get the repair in the database to appear in the UI by changing the value in repair_run.repair_parallelism to the enumeration name (e.g. PARALLEL instead of parallel).
This is reaper 0.3.2 against Cassandra 2.25 using Postgres as a storage backend.
Hi folks. Thanks for your work on reaper!
To track your changes a simple changelog.md would be awesome.
Update documentation to show configuration for reaper with storage type of cassandra.
Logging is a bit insane. We need to go through and make reaper logging useful by decreasing the log levels of certain messages and set up reasonable defaults out of the box.
There's a bug in Cassandra 2.2 (fixed in 3.0) that prevents Reaper from running full repairs.
A null check is missing on the datacenters collection :
We should be able to set up multiple reaper instances for HA purposes. Ultimately, it would be useful to have the ability to run reaper on every Cassandra node.
Hello,
Currently when scheduling or running a repair there is a filed for keyspace, but is it possible to have a wild card * so that all the keyspaces will be repaired.
In my usecase we have multiple keyspaces that gets generated based on the req, it is very much impossible to use keyspace name to repair
After spending some time trying to get this to work, I saved what didn't work for me:
<properties>
+ <!-- Docker related -->
+ <docker-maven-plugin.version>0.4.13</docker-maven-plugin.version>
+ <service.jar>${project.artifactId}-${project.version}-SNAPSHOT.jar</service.jar>
+ <docker.registry>index.docker.io/</docker.registry>
+ <push.image>false</push.image> <!-- set to true to push, normally only on jenkins build on master -->
</properties>
<profile>
<build>
+ <plugin>
+ <groupId>com.spotify</groupId>
+ <artifactId>docker-maven-plugin</artifactId>
+ <version>${docker-maven-plugin.version}</version>
+ <configuration>
+ <imageName>reaperpom</imageName>
+ <baseImage>java:8-jre</baseImage>
+ <entryPoint>["java", "-jar",
+ "/${project.build.finalName}.jar"]
+ </entryPoint>
+ <useConfigFile>true</useConfigFile>
+ <!-- copy the service's jar file from target into the root directory of the image -->
+ <resources>
+ <resource>
+ <targetPath>/</targetPath>
+ <directory>${project.build.directory}
+ </directory>
+ <include>${project.build.finalName}.jar
+ </include>
+ </resource>
+ </resources>
+ </configuration>
+ </plugin>
</plugins>
</build>
</profile>
Perhaps a silly idea, but we should discuss having reaper running constantly, throttling itself back based on metrics like load, dropped messages, etc. Then it could scale back on it's own when load is high and take care of business when load is low
It's important that we prioritize code quality and ensure our test coverage is excellent. I'd like to formalize our requirements in order to hold ourselves accountable as well as anyone submitting a PR to the project.
The list of terminated repair runs can grow quite quickly and make the repair screen barely usable.
The form should move above the list of repairs, and we should use an infinite scrolling technique to display past repair jobs.
We would display all unfinished runs + the last 10 terminated repairs at first, then upon scroll we would display older repair runs.
If infinite scroll is hard to achieve, we can make an history screen to display all past repair runs (paginated or not).
Line 364 in service/RepairRunner.java swallows a throwable that it likely shouldn't be swallowing. If I change that line to
LOG.error("Executing SegmentRunner failed: " + t.getMessage(), t);
I end up getting this:
INFO [2016-11-11 16:05:52,220] [clustername:1:139] c.s.r.s.SegmentRunner - It is ok to repair segment '139' on repair run with id '1' INFO [2016-11-11 16:05:52,222] [clustername:1:139] c.s.r.c.JmxProxy - Triggering repair of range (-8790692742742779625,-8790661161171969027] for keyspace "keyspacename" on host 10.20.5.87, with repair parallelism DATACENTER_AWARE, in cluster with Cassandra version '2.2.5' (can use DATACENTER_AWARE 'true'), for column families: [customers] ERROR [2016-11-11 16:05:52,226] [clustername:1:139] c.s.r.s.RepairRunner - Executing SegmentRunner failed: null java.lang.NullPointerException: null at java.util.AbstractCollection.addAll(AbstractCollection.java:343) ~[na:1.8.0_91] at org.apache.cassandra.service.StorageService.forceRepairRangeAsync(StorageService.java:3005) ~[cassandra-reaper-0.3.1-SNAPSHOT.jar:0.3.1-SNAPSHOT] at sun.reflect.GeneratedMethodAccessor95.invoke(Unknown Source) ~[na:na] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_91] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_91] at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71) ~[na:1.8.0_91] at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) ~[na:na] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_91] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_91] at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:275) ~[na:1.8.0_91] at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112) ~[na:1.8.0_91] at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46) ~[na:1.8.0_91] at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237) ~[na:1.8.0_91] at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138) ~[na:1.8.0_91] at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252) ~[na:1.8.0_91] at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819) ~[na:1.8.0_91] at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801) ~[na:1.8.0_91] at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1468) ~[na:1.8.0_91] at javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:76) ~[na:1.8.0_91] at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1309) ~[na:1.8.0_91] at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1401) ~[na:1.8.0_91] at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:829) ~[na:1.8.0_91] at sun.reflect.GeneratedMethodAccessor29.invoke(Unknown Source) ~[na:na] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_91] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_91] at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:324) ~[na:1.8.0_91] at sun.rmi.transport.Transport$1.run(Transport.java:200) ~[na:1.8.0_91] at sun.rmi.transport.Transport$1.run(Transport.java:197) ~[na:1.8.0_91] at java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0_91] at sun.rmi.transport.Transport.serviceCall(Transport.java:196) ~[na:1.8.0_91] at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:568) ~[na:1.8.0_91] at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826) ~[na:1.8.0_91] at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:683) ~[na:1.8.0_91] at java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0_91] at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682) ~[na:1.8.0_91] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_91] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_91] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
Not real sure where to go from here.
Create a docker image for cassandra-reaper as part of the build process. Somewhat of a precursor #51 .
We will probably want to set up a dedicated mailing list in the near future for Reaper. My personal preference is Google Groups. Thoughts?
Add a field to enter an email address to notify when a repair starts/stops/fails. Currently have a crude ruby script which runs via cron and checks to see if a repair is running and to send an email out
Currently available as spotify/cassandra-reaper#73, needs to be pulled in. There's conflicts, so a PR with the merged conflicts against master is what we need.
JmxProxy
is doing a check if repair is running or not. Between pre 2.2 and post 2.2 version MX bean names have changes and check if repair is running will always issue InstanceNotFoundException
in one of the checks which will get logged and pollute logs even though it is expected.
Version of Cassandra should be checked and only one way of checking should be done, or only exception message should be logged since it is expected to have exception here.
In a multi dc setup, you should be able to configure a reaper per dc. This feature would require using Cassandra storage and would involve coordination between DCs.
when starting a repair (also with scheduled repairs) i always have the problem, that creaper starts the repair with 9 segments, no matter how many segments have been configured with the webapp
currently in PR #2
while creating new run, if the passed value of incremental repair flag is different from the existing value then it should allow to create new repair_unit instead of getting repair_unit based on cluster name/ keyspace /column combination.
so, if the full run was executed for the clustername/keyspace/column was executed earlier, not able to start new incremental run for the same combination. Reverse is also true.
So as per current solution if i delete the repair_unit then due to referential constraints i need to delete repair_segment and repair_run as well which will delete the run history corresponds to the repaid_unit.
Repairs don't intelligently resume. They should.
Tried to use incremental repairs with latest reaper snapshot but sstables didn't get marked with "repaired at" timestamp. And repair has finished too quickly (6min), although previous non incremental repair took ~4 hours.
Reaper log:
INFO [2017-03-31 18:51:20,979] [dw-48 - POST /repair_run?clusterName=test1&keyspace=testks&owner=user&repairParallelism=PARALLEL&incrementalRepair=true] c.s.r.r.RepairRunResource - add repair run called with: clusterName = Optional.of(test1), keyspace = Optional.of(testks), tables = Optional.absent(), owner = Optional.of(user), cause = Optional.absent(), segmentCount = Optional.absent(), repairParallelism = Optional.of(PARALLEL), intensity = Optional.absent(), incrementalRepair = Optional.of(true)
...
INFO [2017-03-31 18:57:11,194] [test1:1] c.s.r.s.RepairRunner - Next segment to run : 12
INFO [2017-03-31 18:57:28,743] [test1:1:12] c.s.r.s.SegmentRunner - It is ok to repair segment '12' on repair run with id '1'
INFO [2017-03-31 18:57:28,745] [test1:1:12] c.s.r.c.JmxProxy - Triggering repair of range (5700662054527062566,5707996255960520728] for keyspace "testks" on host 192.168.0.10, with repair parallelism parallel, in cluster with Cassandra version '3.0.11' (can use DATACENTER_AWARE 'true'), for column families: []
INFO [2017-03-31 18:57:28,746] [test1:1:12] c.s.r.s.SegmentRunner - Repair for segment 12 started, status wait will timeout in 18000000 millis
INFO [2017-03-31 18:57:31,790] [test1:1:12] c.s.r.s.SegmentRunner - Repair command 228 on segment 12 returned with state DONE
INFO [2017-03-31 18:57:41,202] [test1:1] c.s.r.s.RepairRunner - Repair amount done 12
INFO [2017-03-31 18:57:41,202] [test1:1] c.s.r.s.RepairRunner - Repairs for repair run #1 done
Log from Cassandra node:
DEBUG [AntiEntropyStage:1] 2017-03-31 18:55:32,963 RepairMessageVerbHandler.java:144 - Got anticompaction request AnticompactionRequest{parentRepairSession=9d508810-1643-11e7-b728-5bfc5ef6abd0} org.apache.cassandra.repair.messages.AnticompactionRequest@cc41c373
INFO [AntiEntropyStage:1] 2017-03-31 18:55:32,963 ActiveRepairService.java:439 - [repair #9d508810-1643-11e7-b728-5bfc5ef6abd0] Not a global repair, will not do anticompaction
for i in *Data.db; do sstablemetadata $i|grep Repa; done
Repaired at: 0
Repaired at: 0
Repaired at: 0
Repaired at: 0
Repaired at: 0
Repaired at: 0
Repaired at: 0
Repaired at: 0
I've setup my cassandra-reaper config as follows:
segmentCount: 200
repairParallelism: DATACENTER_AWARE
repairIntensity: 0.9
repairRunThreadCount: 15
hangingRepairTimeoutMins: 30
incrementalRepair: false
scheduleDaysBetween: 7
storageType: cassandra
enableCrossOrigin: true
logging:
level: DEBUG
loggers:
io.dropwizard: INFO
org.eclipse.jetty: WARN
appenders:
- type: console
logFormat: "%-6level [%d] [%t] %logger{5} - %msg %n"
server:
type: default
applicationConnectors:
- type: http
port: 8080
bindHost: 0.0.0.0
adminConnectors:
- type: http
port: 8081
bindHost: 0.0.0.0
database:
driverClass: org.postgresql.Driver
user: postgres
password: postgres
url: jdbc:postgresql://127.0.0.1/reaper
cassandra:
clusterName: "StageCassandra"
contactPoints: ["<ip>", "<ip>", "<ip>"]
keyspace: reaper_db
And, after creating the keyspace per https://github.com/thelastpickle/cassandra-reaper/blob/master/src/main/db/reaper_cassandra_db.cql
I get this error after trying to add my first repair.
java.lang.IllegalArgumentException: No enum constant org.apache.cassandra.repair.RepairParallelism.dc_parallel
at java.lang.Enum.valueOf(Enum.java:238) ~[na:1.8.0_111]
at org.apache.cassandra.repair.RepairParallelism.valueOf(RepairParallelism.java:26) ~[cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
at com.spotify.reaper.storage.CassandraStorage.buildRepairRunFromRow(CassandraStorage.java:703) ~[cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
at com.spotify.reaper.storage.CassandraStorage.getRepairRunsAsync(CassandraStorage.java:260) ~[cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
at com.spotify.reaper.storage.CassandraStorage.getRepairRunsForCluster(CassandraStorage.java:223) ~[cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
at com.spotify.reaper.resources.RepairRunResource.listRepairRuns(RepairRunResource.java:447) ~[cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_111]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_111]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_111]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_111]
at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) ~[cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205) ~[cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) ~[cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
at io.dropwizard.jersey.guava.OptionalResourceMethodDispatchAdapter$OptionalRequestDispatcher.dispatch(OptionalResourceMethodDispatchAdapter.java:37) ~[cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302) ~[cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
at com.sun.jersey.server.impl.uri.rules.ResourceObjectRule.accept(ResourceObjectRule.java:100) ~[cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) ~[cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) ~[cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542) [cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473) [cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419) [cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409) [cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409) [cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:540) [cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:715) [cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:848) [cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
at io.dropwizard.jetty.NonblockingServletHolder.handle(NonblockingServletHolder.java:49) [cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1515) [cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83) [cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:348) [cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
at io.dropwizard.jetty.BiDiGzipFilter.doFilter(BiDiGzipFilter.java:127) [cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486) [cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
at io.dropwizard.servlets.ThreadNameFilter.doFilter(ThreadNameFilter.java:29) [cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486) [cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
at io.dropwizard.jersey.filter.AllowedMethodsFilter.handle(AllowedMethodsFilter.java:44) [cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
at io.dropwizard.jersey.filter.AllowedMethodsFilter.doFilter(AllowedMethodsFilter.java:39) [cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486) [cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
at org.eclipse.jetty.servlets.CrossOriginFilter.handle(CrossOriginFilter.java:248) [cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
at org.eclipse.jetty.servlets.CrossOriginFilter.doFilter(CrossOriginFilter.java:211) [cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486) [cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:519) [cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1097) [cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:448) [cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1031) [cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:136) [cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.java:175) [cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
at io.dropwizard.jetty.RoutingHandler.handle(RoutingHandler.java:51) [cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:92) [cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:162) [cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
at org.eclipse.jetty.server.Server.handle(Server.java:446) [cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:271) [cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:246) [cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.run(AbstractConnection.java:358) [cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:601) [cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:532) [cassandra-reaper-0.3.2-SNAPSHOT.jar:0.3.2-SNAPSHOT]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
With memory storage, I don't get said errors.
Here's the respective versions:
Cassandra:
apt-cache policy cassandra
cassandra:
Installed: 3.0.9
Candidate: 3.0.9
Version table:
*** 3.0.9 0
500 http://www.apache.org/dist/cassandra/debian/ 30x/main amd64 Packages
100 /var/lib/dpkg/status
Oracle Java:
apt-cache policy oracle-java8-installer
oracle-java8-installer:
Installed: 8u111+8u111arm-1~webupd8~0
Candidate: 8u111+8u111arm-1~webupd8~0
Version table:
*** 8u111+8u111arm-1~webupd8~0 0
500 http://ppa.launchpad.net/webupd8team/java/ubuntu/ trusty/main amd64 Packages
100 /var/lib/dpkg/status
Cassandra-Reaper:
apt-cache policy reaper
reaper:
Installed: 0.3.2-SNAPSHOT
Candidate: 0.3.2-SNAPSHOT
Version table:
*** 0.3.2-SNAPSHOT 0
100 /var/lib/dpkg/status
The reaper was built from master yesterday using make deb
and oracle Java (plus my two PR's).
If I want to use Cassandra itself as a persistent storage provider of Cassandra-reaper and I have set the different password of default user ‘cassandra’, how can I specify those into the resource/cassandra-reaper.yaml file. Looks like it only works for default Cassandra username/password (cassandra/cassandra).
Right now it just says how to use postgres, we need to add C* storage config information
We've started playing around with your fork of cassandra-reaper over at Reddit and set out to get things all nicely packaged with the included debian definitions. It looks like these are out of date now. Are these going to be maintained long term? If so, are updates in the plans?
This can probably be hashed out pretty quickly, but I want it here as a reminder. I want to make sure we adopt a formal release process starting with the 0.4 release in order to be able to make bug fixes as well as adopt new features.
My proposal is to follow semver closely:
For feature development:
For a release:
1a. For a minor release (0.4, 0.5, 1.0, etc) start by cutting a new branch. We can bikeshed the naming convention but for now I'll just use x.y
. No patch indicated. Any preference on a prefix?
1b. For a bug fix release cutting a new branch is not necessary, as the bug fix should be merged into the branch created in 1a.
2. Create a new tag for the release prefixed with v
plus the semver release version. For example, v1.0.0
is a major release, v0.4.0
is a minor release v0.4.1
is a bug fix release.
3. Major releases are done to indicate backwards incompatibility. The exception to this would be releases prior to 1.0, which may change as things are rapidly stabilized.
For this issue to be closed we must update the readme with these release details.
This is related to #6. Once we have our coding and testing guidelines, it's important that we expose them to anyone submitting a PR.
Reaper should be able to repair a single node or a given set of nodes.
Currently Reaper creates segments for the whole token ring, but should be able to repair only the segments that one node is responsible (or replica) for.
This would allow to run downtime/outage related repairs in Reaper, which is not currently possible.
After building from scratch with mvn clean package -Pbuild-ui
and running with (after switching in the yaml file to the memory backend) java -jar target/cassandra-reaper-X.X.X.jar server resource/cassandra-reaper.yaml
, I browse to localhost:8080 and the webui doesn't work.
It looks like it is incorrectly looking for resources at the root /deps.js
rather than /webui/deps.js
The current UI is so minimal it is frequently not very helpful.
Hi,
I'm trying to schedule repair on my cluster, but with no luck. I get {"message":"There was an error processing your request. It has been logged (ID af6991042de7f785)."}
response in UI and here's the log:
INFO [2017-03-10 08:18:11,384] [dw-172 - POST /repair_schedule?clusterName=productioncluster&keyspace=accounts&owner=rfurmanski&scheduleTriggerTime=2017-03-10T23%3A59&scheduleDaysBetween=3&repairParallelism=SEQUENTIAL&intensity=0.95&incrementalRepair=false] c.s.r.r.RepairScheduleResource - add repair schedule called with: clusterName = Optional.of(productioncluster), keyspace = Optional.of(accounts), tables = Optional.absent(), owner = Optional.of(rfurmanski), segmentCount = Optional.absent(), repairParallelism = Optional.of(SEQUENTIAL), intensity = Optional.of(0.95), incrementalRepair = Optional.of(false), scheduleDaysBetween = Optional.of(3), scheduleTriggerTime = Optional.of(2017-03-10T23:59)
INFO [2017-03-10 08:18:11,431] [dw-172 - POST /repair_schedule?clusterName=productioncluster&keyspace=accounts&owner=rfurmanski&scheduleTriggerTime=2017-03-10T23%3A59&scheduleDaysBetween=3&repairParallelism=SEQUENTIAL&intensity=0.95&incrementalRepair=false] c.s.r.r.RepairScheduleResource - first schedule activation will be: 2017-03-10T23:59:00Z
ERROR [2017-03-10 08:18:11,436] [dw-172 - POST /repair_schedule?clusterName=productioncluster&keyspace=accounts&owner=rfurmanski&scheduleTriggerTime=2017-03-10T23%3A59&scheduleDaysBetween=3&repairParallelism=SEQUENTIAL&intensity=0.95&incrementalRepair=false] i.d.j.e.LoggingExceptionMapper - Error handling a request: c42c96a4a6402f4b
java.lang.NullPointerException: null
at com.spotify.reaper.resources.RepairScheduleResource.addRepairSchedule(RepairScheduleResource.java:152) ~[cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_101]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_101]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_101]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_101]
at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) ~[cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205) ~[cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) ~[cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
at io.dropwizard.jersey.guava.OptionalResourceMethodDispatchAdapter$OptionalRequestDispatcher.dispatch(OptionalResourceMethodDispatchAdapter.java:37) ~[cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302) ~[cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
at com.sun.jersey.server.impl.uri.rules.ResourceObjectRule.accept(ResourceObjectRule.java:100) ~[cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) ~[cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) ~[cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542) [cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473) [cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419) [cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409) [cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409) [cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:540) [cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:715) [cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:848) [cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
at io.dropwizard.jetty.NonblockingServletHolder.handle(NonblockingServletHolder.java:49) [cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1515) [cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83) [cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:295) [cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
at io.dropwizard.jetty.BiDiGzipFilter.doFilter(BiDiGzipFilter.java:127) [cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486) [cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
at io.dropwizard.servlets.ThreadNameFilter.doFilter(ThreadNameFilter.java:29) [cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486) [cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
at io.dropwizard.jersey.filter.AllowedMethodsFilter.handle(AllowedMethodsFilter.java:44) [cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
at io.dropwizard.jersey.filter.AllowedMethodsFilter.doFilter(AllowedMethodsFilter.java:39) [cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486) [cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
at org.eclipse.jetty.servlets.CrossOriginFilter.handle(CrossOriginFilter.java:248) [cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
at org.eclipse.jetty.servlets.CrossOriginFilter.doFilter(CrossOriginFilter.java:211) [cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486) [cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:519) [cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1097) [cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:448) [cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1031) [cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:136) [cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.java:175) [cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
at io.dropwizard.jetty.RoutingHandler.handle(RoutingHandler.java:51) [cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:92) [cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:162) [cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
at org.eclipse.jetty.server.Server.handle(Server.java:446) [cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:271) [cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:246) [cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.run(AbstractConnection.java:358) [cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:601) [cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:532) [cassandra-reaper-0.4.0.jar:0.4.0-SNAPSHOT]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
Can you help me?
At present, whenever you startup cassandra-reaper using cassandra as the persistent storage, it doesn't create the keyspace specified in the configuration file if it didn't already exist in the cluster.
Ideally, this would be done in CassandraStorage.java before acquiring the session using the keyspace.
I noticed the following line commented out in the the initialize_db.cql file.
-- CREATE KEYSPACE IF NOT EXISTS reaper_db WITH REPLICATION={'class':'SimpleStrategy', 'replication_factor':3};
The problem with this approach is if the user wants to use a different replication factor or other options with the keyspace creation. Unfortunately, the configuration of the keyspace is also specified within the dropwizard-cassandra dependency so it will be difficult to extend the cassandra:
section of the yml file as it is based on that dependency.
Some ideas:
cassandra:
section that are specific to the type of of storage in use. e.g. storageOptions: { type: cassandra, replicationStrategy: SimpleStrategy, replicationFactor: 3 }
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.