JarCastingA JGit SDK-backed FileSystem driver for Hadoop
This is an experimental FileSystem for Hadoop that uses the JGit SDK. This has not been heavily tested yet. Use at your own risk.
Features:
- Clones each given repo+branch once and uses a background thread to fetch updates
- Proxies through to a read-only local filesystem driver for high speed
- Default packaging uses an uber-jar for easy deployment
- Download prebuilt jar from Maven Central
Import from Maven Central
<dependency>
<groupId>com.simiacryptus</groupId>
<artifactId>hadoop-jgit-fs</artifactId>
<version>0.1</version>
</dependency>
Build Instructions
Build using maven:
$ mvn package
Copy jar and various dependencies to your hadoop libs dir (run 'hadoop classpath' to find appropriate lib dir):
$ cp target/hadoop-jgit-fs-0.1.jar /usr/lib/hadoop/lib/
Add the following keys to your core-site.xml file:
<!-- necessary for Hadoop to load our filesystem driver -->
<property>
<name>fs.git.impl</name>
<value>com.simiacryptus.hadoop_jgit.GitFileSystem</value>
</property>
You should now be able to run commands:
$ hadoop fs -ls git://github.com/SimiaCryptus/hadoop-jgit-fs.git/master/
Tunable parameters
These may or may not improve performance. The defaults were set without much testing.
- fs.jgit.pull.lazy - Frequency (in seconds) of foreground fetches
- fs.jgit.pull.eager - Frequency (in seconds) of background fetches
- fs.jgit.dismount.seconds - Idle time (in seconds) to dismount repo driver
- fs.jgit.dismount.delete - If true, files will be removed when repo driver dismounts
- fs.jgit.datadir - Data directory to use for local storage
- fs.jgit.auth.user - Username for authentication (Optional)
- fs.jgit.auth.pass - Password for authentication (Optional)
Caveats
This is currently implemented as a FileSystem and not a AbstractFileSystem.
Changes
0.1
- Created
