Cascading

Thanks for using Cascading.

General Information:

For project documentation and community support, visit: cascading.org

To download a pre-built distribution, visit http://cascading.org/downloads/, or use Maven (described below).

The project includes five Cascading jar files:

• cascading-core-x.y.z.jar - all Cascading Core class files
• cascading-xml-x.y.z.jar - all Cascading XML operations class files
• cascading-local-x.y.z.jar - all Cascading Local in-memory mode class files
• cascading-hadoop-x.y.z.jar - all Cascading Hadoop 1.x MapReduce mode class files
• cascading-hadoop2-mr1-x.y.z.jar - all Cascading Hadoop 2.x MapReduce mode class files

These class jars, along with, tests, source and javadoc jars, are all available via the Conjars.org Maven repository.

Hadoop 1.x mode is where the Cascading application should run on a Hadoop cluster. Hadoop 2.x is the same but for Hadoop 2.x releases.

Local mode is where the Cascading application will run locally in memory without any Hadoop dependencies.

Extensions, the SDK, and DSLs

There are a number of projects based on Cascading available. Visit the Cascading Extensions page for a current list.

Or download the Cascading SDK which includes pre-built binaries.

Of note are two top level projects:

• Lingual - ANSI SQL and JDBC with Cascading
• Pattern - Machine Learning and PMML support with Cascading

And new languages:

• Scalding - A Scala based DSL
• Cascalog - A Clojure based DSL

Versioning

Cascading stable releases are always of the form x.y.z, where z is the current maintenance release.

The source and tags for all stable releases can be found here: https://github.com/Cascading/cascading

WIP (work in progress) releases are fully tested builds of code not yet deemed fully stable. On every build by our continuous integration servers, the WIP build number is increased. Successful builds are then tagged and published.

The WIP releases are always of the form x.y.z-wip-n, where x.y.z will be the next stable release version the WIP releases are leading up to. n is the current successfully tested build.

The source, working branches, and tags for all WIP releases can be found here: https://github.com/cwensel/cascading

Or downloaded from here: http://concurrentinc.com/downloads/

When a WIP is deemed stable and ready for production use, it will be published as a x.y.z release, and made available from the cascading.org site.

Reporting issues

To report an issue, first ping the Cascading User mailing list with any questions.

If unresolved, look for a comparable test in the cascading-platform sub-project, reproduce your issue, then issue a pull request with the failing test added to one of the existing suites.

If you wish to allow your test code to be added to the Cascading test suite, please sign and return this contributor agreement.

Using with Maven/Ivy

It is strongly recommended developers pull Cascading from our Maven compatible jar repository Conjars.org.

You can find the latest public and WIP (work in progress) releases here:

• http://conjars.org/cascading/cascading-core
• http://conjars.org/cascading/cascading-local
• http://conjars.org/cascading/cascading-hadoop
• http://conjars.org/cascading/cascading-hadoop2-mr1
• http://conjars.org/cascading/cascading-xml

When creating tests, make sure to add any of the relevant above dependencies to your test scope or equivalent configuration along with the cascading-platform dependency.

• http://conjars.org/cascading/cascading-platform

Note the cascading-platform compile dependency has no classes, you must pull the tests dependency with the tests classifier.

See http://cascading.org/downloads/#maven for example Maven pom dependency settings.

Source and Javadoc artifacts (using the appropriate classifier) are also available through Conjars.

Note that cascading-hadoop and cascading-hadoop2-mr1 has a provided dependency on the Hadoop jars so that it won't get sucked into any application packaging as a dependency, typically.

Building and IDE Integration

For most cases, building Cascading is unnecessary as it has been pre-built, tested, and published to our Maven repository (above).

To build Cascading, run the following in the shell:

> git clone https://github.com/cascading/cascading.git
> cd cascading
> gradle build

Cascading requires Gradle to build.

To use an IDE like IntelliJ, run the following to create IntelliJ project files:

> gradle idea

Similarly for Eclipse:

> gradle eclipse

Using with Apache Hadoop

First confirm you are using a supported version of Apache Hadoop by checking the Compatibility page.

To use Cascading with Hadoop, we suggest stuffing cascading-core, cascading-hadoop, (optionally) cascading-xml jar files and all third-party libs into the lib folder of your job jar and executing your job via $HADOOP_HOME/bin/hadoop jar your.jar <your args>.

For example, your job jar would look like this (via: jar -t your.jar)

/<all your class and resource files>
/lib/cascading-core-x.y.z.jar
/lib/cascading-hadoop-x.y.z.jar
/lib/cascading-xml-x.y.z.jar
/lib/<cascading third-party jar files>

Hadoop will unpack the jar locally and remotely (in the cluster) and add any libraries in lib to the classpath. This is a feature specific to Hadoop.