Apache YARN Data Flow Server

Data Flow Server for Apache Yarn

License

License

Categories

Categories

Data
GroupId

GroupId

org.springframework.cloud
ArtifactId

ArtifactId

spring-cloud-dataflow-server-yarn
Last Version

Last Version

1.2.0.RELEASE
Release Date

Release Date

Type

Type

jar
Description

Description

Apache YARN Data Flow Server
Data Flow Server for Apache Yarn
Project Organization

Project Organization

Pivotal Software, Inc.

Download spring-cloud-dataflow-server-yarn

How to add to project

<!-- https://jarcasting.com/artifacts/org.springframework.cloud/spring-cloud-dataflow-server-yarn/ -->
<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-dataflow-server-yarn</artifactId>
    <version>1.2.0.RELEASE</version>
</dependency>
// https://jarcasting.com/artifacts/org.springframework.cloud/spring-cloud-dataflow-server-yarn/
implementation 'org.springframework.cloud:spring-cloud-dataflow-server-yarn:1.2.0.RELEASE'
// https://jarcasting.com/artifacts/org.springframework.cloud/spring-cloud-dataflow-server-yarn/
implementation ("org.springframework.cloud:spring-cloud-dataflow-server-yarn:1.2.0.RELEASE")
'org.springframework.cloud:spring-cloud-dataflow-server-yarn:jar:1.2.0.RELEASE'
<dependency org="org.springframework.cloud" name="spring-cloud-dataflow-server-yarn" rev="1.2.0.RELEASE">
  <artifact name="spring-cloud-dataflow-server-yarn" type="jar" />
</dependency>
@Grapes(
@Grab(group='org.springframework.cloud', module='spring-cloud-dataflow-server-yarn', version='1.2.0.RELEASE')
)
libraryDependencies += "org.springframework.cloud" % "spring-cloud-dataflow-server-yarn" % "1.2.0.RELEASE"
[org.springframework.cloud/spring-cloud-dataflow-server-yarn "1.2.0.RELEASE"]

Dependencies

compile (4)

Group / Artifact Type Version
org.springframework.cloud : spring-cloud-dataflow-server-core jar 1.2.0.RELEASE
org.springframework.cloud : spring-cloud-dataflow-server-yarn-autoconfig jar 1.2.0.RELEASE
org.springframework.cloud : spring-cloud-starter-deployer-yarn jar 1.2.0.RELEASE
org.springframework.cloud : spring-cloud-deployer-yarn-autoconfig jar 1.2.0.RELEASE

Project Modules

There are no modules declared in this project.

Spring Data Flow Dashboard

Latest Release Version Latest Snapshot Version
Build Status

Spring Cloud Data Flow is a microservices-based toolkit for building streaming and batch data processing pipelines in Cloud Foundry and Kubernetes.

Data processing pipelines consist of Spring Boot apps, built using the Spring Cloud Stream or Spring Cloud Task microservice frameworks.

This makes Spring Cloud Data Flow ideal for a range of data processing use cases, from import/export to event streaming and predictive analytics.


Components

Architecture: The Spring Cloud Data Flow Server is a Spring Boot application that provides RESTful API and REST clients (Shell, Dashboard, Java DSL). A single Spring Cloud Data Flow installation can support orchestrating the deployment of streams and tasks to Local, Cloud Foundry, and Kubernetes.

Familiarize yourself with the Spring Cloud Data Flow architecture and feature capabilities.

Deployer SPI: A Service Provider Interface (SPI) is defined in the Spring Cloud Deployer project. The Deployer SPI provides an abstraction layer for deploying the apps for a given streaming or batch data pipeline, and managing the application lifecycle.

Spring Cloud Deployer Implementations:

Domain Model: The Spring Cloud Data Flow domain module includes the concept of a stream that is a composition of Spring Cloud Stream applications in a linear data pipeline from a source to a sink, optionally including processor application(s) in between. The domain also includes the concept of a task, which may be any process that does not run indefinitely, including Spring Batch jobs.

Application Registry: The App Registry maintains the metadata of the catalog of reusable applications. For example, if relying on Maven coordinates, an application URI would be of the format: maven://<groupId>:<artifactId>:<version>.

Shell/CLI: The Shell connects to the Spring Cloud Data Flow Server's REST API and supports a DSL that simplifies the process of defining a stream or task and managing its lifecycle.

Community Implementations: There are also community maintained Spring Cloud Data Flow implementations that are currently based on the 1.7.x series of Spring Cloud Data Flow.

The Apache YARN implementation has reached end-of-line status. Let us know at Gitter if you are interested in forking the project to continue developing and maintaining it.


Building

Clone the repo and type

$ ./mvnw clean install 

Looking for more information? Follow this link.

Building on Windows

When using Git on Windows to check out the project, it is important to handle line-endings correctly during checkouts. By default Git will change the line-endings during checkout to CRLF. This is, however, not desired for Spring Cloud Data Flow as this may lead to test failures under Windows.

Therefore, please ensure that you set Git property core.autocrlf to false, e.g. using: $ git config core.autocrlf false. Fore more information please refer to the Git documentation, Formatting and Whitespace.


Contributing

We welcome contributions! Follow this link for more information on how to contribute.


Code formatting guidelines

  • The directory ./src/eclipse has two files for use with code formatting, eclipse-code-formatter.xml for the majority of the code formatting rules and eclipse.importorder to order the import statements.

  • In eclipse you import these files by navigating Windows -> Preferences and then the menu items Preferences > Java > Code Style > Formatter and Preferences > Java > Code Style > Organize Imports respectfully.

  • In IntelliJ, install the plugin Eclipse Code Formatter. You can find it by searching the "Browse Repositories" under the plugin option within IntelliJ (Once installed you will need to reboot Intellij for it to take effect). Then navigate to Intellij IDEA > Preferences and select the Eclipse Code Formatter. Select the eclipse-code-formatter.xml file for the field Eclipse Java Formatter config file and the file eclipse.importorder for the field Import order. Enable the Eclipse code formatter by clicking Use the Eclipse code formatter then click the OK button. ** NOTE: If you configure the Eclipse Code Formatter from File > Other Settings > Default Settings it will set this policy across all of your Intellij projects.

org.springframework.cloud

Spring Cloud

Tools for building common patterns in distributed systems with Spring

Versions

Version
1.2.0.RELEASE
1.1.0.RELEASE
1.0.2.RELEASE
1.0.1.RELEASE
1.0.0.RELEASE