JarCastingcassandra-migration-tool-java
Cassandra migration tool for java is a lightweight tool used to execute schema and data migration on Cassandra database. Schema versioning state is saved in schema_version table which stores name and description of migration along with type and timestamp.
There are two types of migrations:
- SCHEMA migrations, which alter database schema (add or remove column, change type of column, add table, etc)
- DATA migrations, which alter data (update, read from one table to another, calculate new fields based on existing data, etc).
Why we did it
The idea behind this project was born while working on a project with live data. Since the development required changes to the data model we had to figure out a way we can update the database schema while keeping the data. We ended up with a lightweight tool that versions database and enables us to change schema and transform data when and how we want (as a part of a build and deploy cycle or application itself). Since these changes are part of the codebase we could also test them before executing on a live cluster.
Versioning
We are going to follow the official DataStax Java driver versions and build our driver for each major, minor and patch version of the driver. We chose this path because of the simplicity to develop, test and deploy for different versions of the driver. This means that if you are using the driver version 2.1.9 you can add this dependency to your project:
<dependency>
<groupId>io.smartcat</groupId>
<artifactId>cassandra-migration-tool</artifactId>
<version>2.1.9.0</version>
</dependency>
or whatever the latest build version of the migration tool is. Check the version at the Maven repository.
Examples
We have two test cases which explain common problems which migration tool can solve. This is only subset of use cases but we think these are most frequent once:
First use case touches problem of adding new field and populating historic data with value. Cassandra does not have DDL default attribute, so you must populate data on application level. MigrationEngineBooksTest can serve as an example of that use case which this tool can cover.
The initial table is simple (can be found in books.cql file) and we use migration classes to do the following:
- Populate data initially with first
datamigration InsertBooksMigration - Add
genrecolumn withschemamigration AddBookGenreFieldMigration - Populate
genrecolumn with seconddatamigration AddGenreMigration
Second use case touches problem of query based modeling. Cassandra has good performance because you model your data as you will query it. Often after initial modeling you have request to read it based on different criteria. In Cassandra you do this with another table which is optimized for new requirements. You need to populate this new table with existing data and you can solve this with migration tool. MigrationEngineItemsTest can serve as an example of that use case which this tool can cover.
The initial table is simple (can be found in items.cql file) and we use migration classes to do the following:
- Populate data initially with first
datamigration InsertInitialItemsMigration - Add
items_by_number_external_idtable withschemamigration CreateItemByNumberAndExternalIdMigration - Populate
items_by_number_external_idtable with seconddatamigration PopulateItemByNumberAndExternalIdMigration
Schema agreement
When executing schema migrations it is necessary to wait for cluster to propagate schema on all nodes. Schema agreement is implemented based on this fix and is exposed through Migration abstract class. To execute a statement with schema agreement you can use executeWithSchemaAgreement method.
