While working with Cassandra we found that sometimes snapshots get left around after repairs fail or have issues. These can cause 100’s of Gigs of space to just be wasted. This can add up quickly and cause issues. On a node with low disk space we can check for stale snapshots the following way.

1
2
$ cd /cassandaData/cassandra/data/keyspace/
$ du -h

If you see large or multiple directories under columnFamily/snapshots/ that may indicate there are stale snapshots that can be cleaned up. We will want to check when these snapshots where created so inside the snapshots directory of the column family run

1
2
3
4
5
6
7
8
9
$ ls -lah
total 3.3M
drwxr-xr-x 7 cassandra cassandra 4.0K Mar 11 16:14 .
drwxr-xr-x 3 cassandra cassandra 704K Mar 11 16:31 ..
drwxr-xr-x 2 cassandra cassandra 508K Mar 01 14:24 4f37dcd0-c7fa-11e4-b5ae-5f969a9b23c8
drwxr-xr-x 2 cassandra cassandra 504K Mar 11 16:14 ad70a8e0-c809-11e4-9b55-39152d07d3bf
drwxr-xr-x 2 cassandra cassandra 532K Mar 11 16:03 b5cfd3a0-c807-11e4-bcd5-db76d671c3d5
drwxr-xr-x 2 cassandra cassandra 504K Mar 10 22:50 cfe34050-c777-11e4-b5ae-5f969a9b23c8
drwxr-xr-x 2 cassandra cassandra 536K Mar 11 16:09 edc03dd0-c808-11e4-be46-35521ca10087

Each of those are snapshots but you will notice 4f37dcd0-c7fa-11e4-b5ae-5f969a9b23c8 is stale, we can now clean that up with the nodetool clearsnapshot command.

1
nodetool clearsnapshot keyspace -t 4f37dcd0-c7fa-11e4-b5ae-5f969a9b23c8

That will take care of removing the stale snapshot you should see disk space recovered immediately.

You can also look in the logs for failed repairs we see something like the following:

1
2
3
4
5
6
7
8
2015-02-20 23:37:52,220 RepairSession.java (line 288) [repair #72e69720-b959-11e4-9b55-39152d07d3bf] session completed with the following error
system.log.10-java.io.IOException: Failed during snapshot creation.
system.log.10- at org.apache.cassandra.repair.RepairSession.failedSnapshot(RepairSession.java:323)
system.log.10- at org.apache.cassandra.repair.RepairJob$2.onFailure(RepairJob.java:126)
system.log.10- at com.google.common.util.concurrent.Futures$4.run(Futures.java:1160)
system.log.10- at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
system.log.10- at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
system.log.10- at java.lang.Thread.run(Thread.java:745)

In that case the bad snapshot would be 72e69720-b959-11e4-9b55-39152d07d3bf.

Integration testing complex transactions in grails can be tricky due to the default behavior of wrapping integration tests in transactions and rolling them back when complete.

The simple solution is to simply turn off transactions for integration tests, that solution will work but tends to lead to data pollution in downstream tests. Burt Beckwith has a solution in his post An Alternative Approach for Grails Integration Tests. Using this solution we can rebuild the database for each test.

In general most of the time you can use the default transactional testing behavior, therefore we want to only use this method some of the time. To achieve this effect we will modify Burt’s original solution slightly.

  • Rebuild the database after each test
  • Depend on the configured data source
  • Update to Spock

Updated Solution

NonTransactionalIntegrationSpec.groovy
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
import org.codehaus.groovy.grails.orm.hibernate.cfg.DefaultGrailsDomainConfiguration
import org.hibernate.SessionFactory
import org.hibernate.cfg.Configuration
import org.hibernate.tool.hbm2ddl.SchemaExport

class NonTransactionalIntegrationSpec extends company.IntegrationSpec {

    @Shared
    private static Configuration _configuration

    @Shared
    def grailsApplication

    static transactional = false

    def setupSpec() {
        if (!_configuration) {
            // 1-time creation of the configuration
            Properties properties = new Properties()
            properties.setProperty 'hibernate.connection.driver_class', grailsApplication.config.dataSource.driverClassName
            properties.setProperty 'hibernate.connection.username', grailsApplication.config.dataSource.username
            properties.setProperty 'hibernate.connection.password', grailsApplication.config.dataSource.password
            properties.setProperty 'hibernate.connection.url', grailsApplication.config.dataSource.url
            properties.setProperty 'hibernate.dialect', 'org.hibernate.dialect.H2Dialect'

            _configuration = new DefaultGrailsDomainConfiguration(grailsApplication: grailsApplication, properties: properties)
        }
    }

    def cleanupSpec() {
      //After spec nuke and pave the test db
      new SchemaExport(_configuration).create(false, true)

      //Clear the sessions
      SessionFactory sf = grailsApplication.getMainContext().getBean('sessionFactory')
      sf.getCurrentSession().clear()
    }

}

You’ll notice we extend a company.IntegrationSpec there isn’t anything special in that just a base integration spec class that can hold our shared testing code such as bootstrapping methods and security methods. Since our data bootstrap logic is set up there we can share the same initial setup across both our normal integration test and our new non-transactional integration specs.

When to Use

The main time we started to need this new testing method was when doing custom propagation of hibernate transactions.

1
2
3
4
  @Transactional(readOnly = true, propagation = Propagation.REQUIRES_NEW)
  def getSomething(){
    //Do some read only operation.
  }

Since this will require a new transaction we will not be able to see the inserts that have happened in the default transaction. We do many of our reads as read only and force a new transaction when we are more read heavy during an operation.

In general anytime you are working directly with transaction rollbacks or propagation it best to test those things directly without having Grails inject an extra transaction in there for you.

The App Info Plugin for Grails is a great tool. We use it in most of our Grails deployments. One of the great features offered is the ability to view all current sessions and their details. Another tool we are using is Memcached Session Manager which allows shared sessions between our Tomcat servers, great for rolling deploys and high availability.

We found an issue while running both of these in production. Over time the number of sessions reported by App Info would grow extremely large. It turns out that having the App Info plugin track sessions for you when the sessions are shared via memcached can lead to the plugin holding onto references to sessions that have been destroyed on other severs.

So at this time we can’t use the App Info plugin session tracking with the Memcached Session Manager, since it leads to a memory leak as all those sessions have references inside the plugin and can never be garbage collected.

We can simply turn off the session tracking with the following code:

1
grails.plugins.appinfo.useContextListener = false

But note this config change effect will effect how a war is made so you can’t simply override this in production you will need a new war built.

Below is our memory graph before and after the change:

Graph showing memory usage before and after the config change.

Ratpack has a core class that is the center of the great asynchronous support, Promise.

Ratpack Promises are very easy to work with, there are just a few key points:

  • Only attach to a promise one time
  • If dealing with the error case it must be done before the success case
  • They are Lazy
  • In Groovy we depend on Implicit Closure Coercion to change our closures to an Action.

Happy Path

Consuming Value from Promise
1
2
3
4
5
Promise promise = somethingThatReturnsPromise()

promise.then {
  println it
}

What we are doing here is giving a closure to the promise that once the value is ready the closure will be called with the value passed in as a parameter. We can also be very explicit in what we are getting back from the promise.

Explicit Value from Promise
1
2
3
4
5
6
7
def p = httpClient.get {
  it.url.set(new URI("http://example.com"))
}

p.then { ReceivedResponse receivedResponse ->
  println receivedResponse.statusCode
}

If some error occurs while trying to get the value for the then block the exception will be thrown. Which can be picked up by some error handler down the chain.

Error Callback

So for this works great when dealing with the happy path and wanting exceptions. But we also may want to deal with failures to fulfill the promise. So to do this we start with onError instead of then.

Ratpack Promise with Failure Path
1
2
3
4
5
6
7
httpClient.get {
    it.url.set(new URI("http://example.com"))
} onError {
    println "Something when wrong: ${it.message}"
} then {
    render "Got a ${it.statusCode} status with body of:\n\n${it.body.text}"
}

onError will pass in a throwable to the closure that you can log or do whatever work you would like in the case of a failure.

Lazy Promises

Ratpack promises won’t actually try to generate the value until the then block is called at the end of the current execution. This is done to allow for deterministic asynchronous operations.

Deterministic Promise
1
2
3
4
5
def doWork() {
  httpClient.get {  }.then {  }
  sleep 5000
  throw new RuntimeException("bang!")
}

What will happen in Ratpack is we will always get the exception “bang!”, because the get request will not even get started until the doWork block of execution is finished. Once finished having a then{} will trigger a background thread to start generating the value.

What not to do

You shouldn’t try to attach more than once to a Promise, as what ends up happening is two different promise instances will execute in the background and what we want is only to deal with that value once. So don’t do the following:

Don’t do this
1
2
3
4
5
6
7
8
9
10
11
def p = httpClient.get {
  it.url.set(new URI("http://example.com"))
}

p.onError {
  println it
}

p.then {
  println it.statusCode
}

Starting in Ratpack 0.9.9 the above code should actually throw an error.

Cassandra inserts and updates should always be modeled as upserts when possible. Using the query builder in the Java native driver there isn’t a direct upsert called out, but we can do updates instead of inserts for all cases. The update acts as an upsert and it reduces the number of queries you will need to build.

1
2
3
4
Statement upsert = QueryBuilder.update("table")
        .with(QueryBuilder.append("visits", new Date())) //Add to a CQL3 List
        .where(QueryBuilder.eq("id", "MyID"));
session.execute(upsert);

Above you can see how we model our “upsert”. If a value isn’t found for the given where clause it will insert it.

You must use all parts of a Primary Key for an updates where cluase given a CQL Table with a compound key:

1
2
3
4
5
6
create table tablex(
     pk1 varchar,
     pk2 varchar,
     colA varchar,
     PRIMARY KEY(pk1,pk2)
);

We can not do the following query:

1
2
3
Statement upsert = QueryBuilder.update("tablex")
                .with(QueryBuilder.set("colA", "2"))
                .where(QueryBuilder.eq("pk1", "1"));

You will get an InvalidQueryException:

1
2
3
4
com.datastax.driver.core.exceptions.InvalidQueryException: Missing mandatory PRIMARY KEY part pk2
  com.datastax.driver.core.exceptions.InvalidQueryException.copy(InvalidQueryException.java:35)
  com.datastax.driver.core.DefaultResultSetFuture.extractCauseFromExecutionException(DefaultResultSetFuture.java:256)
  com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:172)

But the following will upsert:

1
2
3
4
Statement upsert = QueryBuilder.update("tablex")
        .with(QueryBuilder.set("colA", "2"))
        .where(QueryBuilder.eq("pk1", "1"))
        .and(QueryBuilder.eq("pk2", "2"));

If you are working on a Groovy script with @Grab, you will sometimes get download failures for dependencies. Such as the following:

1
General error during conversion: Error grabbing Grapes -- [download failed: com.google.guava#guava;16.0!guava.jar(bundle), download failed: org.javassist#javassist;3.18.1-GA!javassist.jar(bundle)]

This issues may have nothing to do with the actual dependency but an issue in your local m2 cache. The quick answer is to just delete ~/.groovy/grapes and ~/.m2/repository. But doing this will force you to re-download dependencies.

To only delete the cache for items giving you an issue you just need to delete the correct directories in both m2 and grapes cache. So for our Guava example you would do the following:

1
2
rm -r ~/.groovy/grapes/com.google.guava
rm -r ~/.m2/repository/com/google/guava

After that you should be able to run the groovy script normally.

New Relic with Grails by default will trace most web transactions through the controller but will not trace down into services. While most true work of a request belongs in services or libraries the default tracing leaves something to be desired.

This is easily fixed by adding New Relic annotations to services and libraries.

BuildConfig.groovy Changes

1
2
3
4
dependencies {
  compile 'com.newrelic.agent.java:newrelic-api:3.4.2'

}

Service Changes

1
2
3
4
5
6
7
8
import com.newrelic.api.agent.Trace

class SubscriptionService {

  @Trace
  def save(Subscription subscription) {
    //Work Here
  }

At this point your code is ready to give more detailed transactions, but the agent on the server must also be configured to accept custom tracing. The config option for this is not available from the web so you must update the newrelic.yml file. Set enable_custom_tracing to true.

1
2
  #enable_custom_tracing is used to allow @Trace on methods
  enable_custom_tracing: true

Now you will get any custom tracing added to your application as well as custom tracing from libraries.

If you are running grails 2.3.1 and see the following sequence pop up before you get some odd test failures.

1
2
3
4
5
6
7
$ grails clean
| Application cleaned.

$ grails test-app
| Environment set to test.....
| Warning No config found for the application.
| Warning DataSource.groovy not found, assuming dataSource bean is configured by Spring

Start using package in between and the problem will go away.

1
2
3
4
5
6
7
8
9
10
11
12
13
$ grails clean
| Application cleaned.
$ grails package
| Compiling 10 source files
| Compiling 12 source files.....

$ grails test-app
| Environment set to test.....
| Server running. Browse to http://localhost:8080/api
| Running 6 cucumber tests...
| Completed 6 cucumber tests, 0 failed in 0m 3s
| Server stopped
| Tests PASSED

Using the JMS 1.2 plugin with Grails 2.3.0.RC1 was producing a number of odd results. Mostly with missing JMS files it turns out that the new spring version didn’t have the needed spring jms included. Just add the following to BuildConfig.groovy

1
2
3
4
dependencies {
  compile 'org.springframework:spring-jms:3.2.4.RELEASE'
  ...
}

Using the Grails Spring Security Core Plugin I found the need to customize the UserDetailsService and use a Grails service. (Part of the roles logic depended on an external API that we already had a service for.) This was easy to accomplish by subclassing the UserDetailsService class I wanted as a base in my case it was actually the SpringSamlUserDetailsService class because I was using the SAML plugin but normally you would subclass GormUserDetailsService. A great starting example is given in the documentation here.

The difference in my case was the need to use the Grails service, I went with providing the service in the resources.groovy file. Below is the example file of what I used.

My resources.groovy

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
import com.example.saml.CustomUserDetailsService
import org.codehaus.groovy.grails.plugins.springsecurity.SpringSecurityUtils

beans = {
      userDetailsService(CustomUserDetailsService) {
       grailsApplication = ref('grailsApplication')
       myService = ref('myService')  //Here we give the reference to the service we want available.
       authorityClassName = SpringSecurityUtils.securityConfig.authority.className
       authorityJoinClassName = SpringSecurityUtils.securityConfig.userLookup.authorityJoinClassName
       authorityNameField = SpringSecurityUtils.securityConfig.authority.nameField
       samlAutoCreateActive = SpringSecurityUtils.securityConfig.saml.autoCreate.active
       samlAutoAssignAuthorities = SpringSecurityUtils.securityConfig.saml.autoCreate.assignAuthorities as Boolean
       samlAutoCreateKey = SpringSecurityUtils.securityConfig.saml.autoCreate.key as String
       samlUserAttributeMappings = SpringSecurityUtils.securityConfig.saml.userAttributeMappings
       samlUserGroupAttribute = SpringSecurityUtils.securityConfig.saml.userGroupAttribute as String
       samlUserGroupToRoleMapping = SpringSecurityUtils.securityConfig.saml.userGroupToRoleMapping
       userDomainClassName = SpringSecurityUtils.securityConfig.userLookup.userDomainClassName
       authoritiesPropertyName = SpringSecurityUtils.securityConfig.userLookup.authoritiesPropertyName
   }
}

Snip from CustomUserDetailsService.groovy

1
2
3
4
class CustomUserDetailsService extends SpringSamlUserDetailsService {
  def myService
...
}