JarCasting
Gradoop: Distributed Graph Analytics on Hadoop
Gradoop is an open source (ALv2) research framework for scalable graph analytics built on top of Apache Flink. It offers a graph data model which extends the widespread property graph model by the concept of logical graphs and further provides operators that can be applied on single logical graphs and collections of logical graphs. The combination of these operators allows the flexible, declarative definition of graph analytical workflows. Gradoop can be easily integrated in a workflow which already uses Flink® operators and Flink® libraries (i.e. Gelly, ML and Table).
Gradoop is work in progress which means APIs may change. It is currently used as a proof of concept implementation and far from production ready.
The project's documentation can be found in our Wiki. The Wiki also contains a tutorial to help getting started using Gradoop.
Further Information (articles and talks)
- Exploration and Analysis of Temporal Property Graphs, EDBT Demo, March 2021
- Graph Sampling with Distributed In-Memory Dataflow Systems, BTW, March 2021
- Evolution Analysis of Large Graphs with Gradoop, ECML PKDD LEG Workshop, September 2019
- Gradoop @Gridka Keynote Distributed Graph Analytics, August 2019
- Temporal Graph Analysis using Gradoop, BTW 2019-Workshopband, March 2019
- Declarative and distributed graph analytics with GRADOOP, VLDB Demo, August 2018
- Cypher-based Graph Pattern Matching in Apache Flink, FlinkForward, September 2017
- Cypher-based Graph Pattern Matching in GRADOOP, SIGMOD GRADES Workshop, May 2017
- DIMSpan - Transactional Frequent Subgraph Mining with Distributed In-Memory Dataflow Systems, arXiv, March 2017
- Distributed Grouping of Property Graphs with GRADOOP, BTW Conf., March 2017
- Graph Mining for Complex Data Analytics, ICDM Demo, December 2016
- [german] Graph Mining für Business Intelligence, data2day, October 2016
- [german] Verteilte Graphanalyse mit Gradoop, JavaSPEKTRUM, October 2016
- Extended Property Graphs with Apache Flink, SIGMOD NDA Workshop, June 2016
- Gradoop @Flink/Neo4j Meetup Berlin, March 2016
- Gradoop @FOSDEM GraphDevroom, January 2016
- Gradoop @FlinkForward, September 2015 (YouTube)
Data Model
In the extended property graph model (EPGM), a database consists of multiple property graphs which are called logical graphs. These graphs describe application-specific subsets of vertices and edges, i.e. a vertex or an edge can be contained in multiple logical graphs. Additionally, not only vertices and edges but also logical graphs have a type label and can have different properties.
Data Model elements (logical graphs, vertices and edges) have a unique identifier, a single label (e.g. User) and a number of key-value properties (e.g. name = Alice). There is no schema involved, meaning each element can have an arbitrary number of properties even if they have the same label.
Graph operators
The EPGM provides operators for both single logical graphs as well as collections of logical graphs; operators may also return single graphs or graph collections. An overview and detailed descriptions of the implemented operators can be found in the Gradoop Wiki.
Setup
Use gradoop via Maven
- Add one of the following dependencies to your maven project
Stable:
<dependency>
<groupId>org.gradoop</groupId>
<artifactId>gradoop-flink</artifactId>
<version>0.6.0</version>
</dependency>
Latest weekly build (additional repository is required):
<repositories>
<repository>
<id>oss.sonatype.org-snapshot</id>
<url>http://oss.sonatype.org/content/repositories/snapshots</url>
<releases><enabled>false</enabled></releases>
<snapshots><enabled>true</enabled></snapshots>
</repository>
</repositories>
<dependency>
<groupId>org.gradoop</groupId>
<artifactId>gradoop-flink</artifactId>
<version>0.7.0-SNAPSHOT</version>
</dependency>
In any case you also need Apache Flink (version 1.9.3):
<dependencies>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-java</artifactId>
<version>1.9.3</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-clients_2.11</artifactId>
<version>1.9.3</version>
</dependency>
</dependencies>
Build gradoop from source
-
Gradoop requires Java 8
-
Clone Gradoop into your local file system
-
Build and execute tests
cd gradoop
mvn clean install
-
You might want to skip tests for faster builds. Also, some tests fail on Windows due to missing test dependencies
mvn clean install -DskipTests
Windows
- Some operators require the Hadoop winutils
Gradoop modules
gradoop-common
The main contents of that module are the EPGM data model and a corresponding POJO implementation which is used in Flink®. The persistent representation of the EPGM is also contained in gradoop-common and together with its mapping to HBase™.
gradoop-data-integration
Provides functionalities to support graph data integration. This includes minimal CSV and JSON importers as well as graph transformation operators (e.g. connect neighbors or conversion of edges to vertices and vice versa).
gradoop-accumulo
Input and output formats for reading and writing graph collections from Apache Accumulo®.
gradoop-hbase
Input and output formats for reading and writing graph collections from Apache HBase™.
gradoop-flink
This module contains reference implementations of the EPGM operators. The EPGM is mapped to Flink® DataSets while the operators are implemented using DataSet transformations. The module also contains implementations of general graph algorithms (e.g. Label Propagation, Frequent Subgraph Mining) adapted to be used with the EPGM model.
gradoop-temporal
This module contains a reference implementation of the Temporal Property Graph Model (TPGM) and it's operators used to perform graph analysis with respect to the additional time dimension in real-world graphs.
gradoop-examples
Contains example pipelines showing use cases for Gradoop.
- Graph grouping example (build structural aggregates of property graphs)
- Social network examples (composition of multiple operators to analyze social networks graphs)
- Input/Output examples (usage of DataSource and DataSink implementations)
gradoop-checkstyle
Used to maintain the code style for the whole project.
Related Repositories
Gradoop Tutorial
Gradoop Tutorial which has been shown in BOSS20' Workshop of VLDB 2020 international conference.
Gradoop Benchmarks
This repository contains sets of Gradoop operator benchmarks designed to run on a cluster to measure scalability and speedup of the operators.
Gradoop Demo
Demo application to show the functionalities of the grouping and query operator in an interactive web UI.
Temporal Graph Explorer
Gradoop Temporal Graph Explorer Demo which showcases some operators of the Temporal Property Graph Model.
Gradoop GDL
This repository contains the definition of our Temporal Graph Definition Language (Temporal-GDL).
Version History
See the Changelog at the Wiki pages.
Disclaimer
Apache®, Apache Accumulo®, Apache Flink, Flink®, Apache HBase™ and HBase™ are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.
