Controller groups

Each controller object only supports only one type of worker configuration which you set in advance. However, different controllers may have different types of workers, and crew supports controller groups to coordinate among these different worker types. With third-party launcher subclasses from other packages, this mechanism will allow you to e.g. send some tasks to GPU-capable or high-memory workers while other tasks go to low-spec workers.

We demonstrate with a controller of fully persistent workers which always stay running and a controller of semi-persistent workers which terminate after completing four tasks. We create controller objects with names.

library(crew)
persistent <- crew_controller_local(name = "persistent")
transient <- crew_controller_local(name = "semi-persistent", tasks_max = 4L)

crew uses a different TCP port for each controller you run, so please do not create hundreds of controllers. Please see the subsection on ports in the README.

We put these controller objects into a new controller group object.

group <- crew_controller_group(persistent, transient)

This controller group has a global connect() method to initialize both controllers.

group$start()

You can choose which worker pool to receive tasks.

group$push(name = "my task", command = sqrt(4), controller = "semi-persistent")

The controller group also supports global methods for wait(), pop(), and terminate(). These methods operate on all controllers at once by default, but the controllers argument allows you to select a subset of controllers to act on. Below in pop() the launcher column of the output indicates which controller ran the task.

group$wait(controllers = "semi-persistent")
group$pop()
#> # A tibble: 1 × 11
#>   name    command result seconds   seed error trace warni…¹ launc…² worker
#>   <chr>   <chr>   <list>   <dbl>  <int> <chr> <chr> <chr>   <chr>    <int>
#> 1 my task sqrt(4) <dbl>        0 5.11e8 NA    NA    NA      semi-p…      1
#> # … with 1 more variable: instance <chr>, and abbreviated variable names
#> #   ¹​warnings, ²​launcher

The controller group has a summary() method which aggregates all the information from the selected controllers.

group$summary(columns = starts_with(c("controller", "tasks")))
#> # A tibble: 2 × 3
#>   controller     tasks_assigned tasks_complete
#>   <chr>                   <int>          <int>
#> 1 persistent                  0              0
#> 2 semi-persistent             1              1

When you are finished, please call terminate() with no arguments to terminate all controllers in the controller group.

group$terminate()