Best Practices for Reproducible Research

Session 6

Ingmar Steiner

2017-06-14

Assignment Review

Build Projects and Tasks

Project anatomy

  • Dependencies
  • Tasks
  • Properties
  • Publications
Dependencies
project “inputs”
Publications
project “outputs”

Task anatomy

Inputs
  • Files
  • Properties
  • Other tasks
  • Determine whether task is up-to-date
Action
  • What the task actually does
  • Creates outputs
Outputs
  • Files
  • Properties

Gradle syntax

Java

project.task("foo");

Groovy

project.task(foo)

Gradle DSL

task foo

Task example

task foo {
  // task configuration
  println "configuring foo"
  doLast {
    // task execution
    println "executing foo"
  }
}

Task dependency example

task foo {
  println "configuring foo"
  doLast {
    println "executing foo"
  }
}

task bar {
  dependsOn foo
  println "configuring bar"
  doLast {
    println "executing bar"
  }
}

Task output example

task foo {
  // declare output file
  outputs.file 'foo'
  doLast {
    // create the file
    println "creating file 'foo'"
    file('foo').createNewFile() // we can always mix in java!
  }
}

Ad-hoc tasks

  • These ad-hoc task definitions are DefaultTasks
  • They have an empty configuration…
  • …and do nothing — until custom code is added.

Of course, there are many common task types provided by the Gradle DSL

Common Task Types

Copy

task copyFoo(type: Copy) {
  from 'foo'
  into 'fooCopy'
  include '**/*.txt' // just text files, at any depth
  expand properties
}
  • Any number of sources
  • Only one destination (will be created if needed!)
  • Easily include or exclude using patterns
  • Filter text files using properties or Ant filters

Exec

task wordCountFoo(type: Exec) {
  commandLine 'wc', 'foo.txt'
  def outputFile = file('wc.txt')
  outputs.file outputFile
  workingDir 'fooCopy'
  standardOutput = new ByteArrayOutputStream()
  doLast {
    outputFile.text = standardOutput.toString()
  }
}
  • Execute any shell command in a subprocess
  • Declare output files as appropriate
  • Capture output streams via POJOs
  • Extra actions can be appended to task execution

Custom Build Logic

Defining custom task types

Mix Groovy classes into DSL build script

class CreateFooFile extends DefaultTask {
  @Input
  String foo
  
  @OutputFile
  File outputFile
  
  @TaskAction
  void doStuff() {
    outputFile.text = foo
  }
}

task foo(type: CreateFooFile) {
  foo = 'bar'
  outputFile = file('baz/foo.txt')
}
  • @Input/@Output properties and files are “autowired” as task inputs/outputs
  • @OutputFile additionally creates any missing parent directories

buildSrc project

  • A special buildSrc subdirectory will automatically build a Groovy project and provide its output to the main build classpath!
  • buildSrc/src/main/groovy/CreateFooFile.groovy, content as before:

Custom plugins

class FooPlugin implements Plugin<Project> {
  void apply(Project project) {
    project.task('foo', type: CreateFooFile) {
      foo = 'bar'
      outputFile = project.file('baz/foo.txt')
    }
  }
}

class CreateFooFile extends DefaultTask {
  @Input
  String foo

  @OutputFile
  File outputFile

  @TaskAction
  void doStuff() {
    outputFile.text = foo
  }
}

apply plugin: FooPlugin

foo {
  // override default configuration
  foo = 'baz'
}
  • Plugin preconfigures build logic (adds tasks, sets up dependencies, etc.)
  • Optional custom configuration after plugin application

buildSrc plugins

  • Put plugin code under buildSrc for preconfigured custom build logic:
buildSrc
└── src
    └── main
        └── groovy
            ├── CreateFooFile.groovy
            └── FooPlugin.groovy

build.gradle

apply plugin: FooPlugin

External plugins

  • Complex plugins can be created externally (including functional tests!) and published.
  • Conventional plugin application: build.gradle

    buildscript {
      repositories {
        mavenLocal()
      }
      dependencies {
        classpath group: 'my.group', name: 'my-plugin', version: '0.1-SNAPSHOT'
      }
    }
    apply plugin: 'my-plugin-id'
  • New application: settings.gradle

    pluginRepositories {
      maven {
        url "${System.properties['user.home']}/.m2/repository"
      }
    }

    build.gradle

    plugins {
      id 'my-plugin-id' version '0.1'
    }

Build Modularization

Dynamic task creation

build.gradle

task sayAll

['foo', 'bar', 'baz'].each { thingie ->
  task("say_$thingie") {
    sayAll.dependsOn it
    doLast {
      println thingie
    }
  }
}

Subprojects

  • Projects can have nested subprojects
  • Defined at init time in settings.gradle
  • Can have their own build.gradle files in subdirectories…
  • …or common build logic applied via parent or shared buildscripts
  • Task dependencies can cross subproject boundaries for complex builds

Subprojects (cont’d)

settings.gradle

include 'foo', 'bar', 'baz'

build.gradle

subprojects {
  task printName {
    doLast {
      println project.name
    }
  }
}

Dynamic subprojects

projects.txt

foo
bar
baz

settings.gradle

new File('projects.txt').eachLine { projName ->
  include projName
}
  • Scales badly – initialization takes longer for more subprojects!

Assembling subproject output

build.gradle

subprojects {
  task foo {
    def outputFile = file("${project.name}.txt")
    outputs.file outputFile
    doLast {
      outputFile.text = project.name
    }
  }
}

task dist(type: Zip) {
  from subprojects.collect { project ->
    project.foo
  }
  baseName 'dist'
}

Parallel Processing

Decoupled projects

  • Subprojects can be decoupled and built in parallel

build.gradle

class Sleep extends DefaultTask {
  @TaskAction
  void sleep() {
    project.exec {
      commandLine 'sleep', 2
    }
  }
}

subprojects {
  task sleep(type: Sleep)
}

Parallelizable tasks

build.gradle

@ParallelizableTask
class Sleep extends DefaultTask {
  @TaskAction
  void sleep() {
    project.exec {
      commandLine 'sleep', 2
    }
  }
}

task sleepAll

(1..3).each { i ->
  task("sleep_$i", type: Sleep) {
    sleepAll.dependsOn it
  }
}

Next

Upcoming topics

  • Dependencies
  • Publications
  • Resource-based configuration
  • Composite builds

Questions?