Manual

Workflows

Workflows are designed by defining the states and the allowed transitions. Using these states and transitions a graph is build. The study governor software will keep track of the states each experiment is in and has passed. It will also enforce valid transistions. At each state/node a Callback can be executed. These callbacks can be used to start custom script/commands, run pipelines, or send emails.

Workflows are defined using a YAML file, which has one main field “states”. Transitions and callbacks are defined as part of the state.

There’s two additinal options “external_systems” and “scantypes” which will be discussed later on.

States

States are defined in the states field of the workflow YAML. Each workflow needs an “untracked” state. This is the state each of the experiments start in. This state is limited to only one output transition and is transitioned by the study governor. This state is also the most basic state we can define. Each state needs at least the following fields:

fields	Description
label	The name of this state, must be unique.
callbacks	A list of callback definitions.
freetext	A description of this state.
transitions	A list of possible transitions from this state

When a state is entered all callbacks will be executed (or skipped if the condition is unmet). Then when all callback executions are finished, transitions will be checked in order to find the first transition whose condition is met. This transition will be used to update the state of the experiment.

Example of a state:

callbacks:
  - <callback definition>
  - <callback definition>
freetext: "The pre-initial state for untracked data"
label: untracked
transitions:
  - <transition definition>
  - <transition definition>

Transitions

Besides states we need to define the possible tranistions between those states. We define a source a destination and a contition:

fields	Description
destination	Label of the destination state.
condition	Extra condition for the transition. Should be one-liner Python code that results in a bool value

Example of a transition:

destination: step3
condition: "experiment.scandate > datetime.datetime(2019, 1, 1)"

Callbacks

fields	type	Description
label	string	Name to identify the callback
description	string	Just some text to describe the callback
function	string	Label of the destination state.
callback_arguments	mapping	The arguments to pass to the callback function, should be a mapping for keyword arguments
run_timeout	int	Timeout (in minutes) before a callback execution run is considered failed
wait_timeout	int	Timeout (in minutes) before a callback execution waiting for manual response is considered failed
initial_delay	int	Time (in seconds) before the callback is allowed to start running
condition	string	A condition for the callback. Should be a one-line Python code that results in a bool
variable_map	mapping	Dict that indicates what variables from the callback should be stored back into the experiment. Has the form {“target”: “source”}.

The callback function and arguments will be used to perform the actual execution of this callback. The callback_arguments should be a mapping where the keys match the arguments of the callback function.

The variable_map is specified as {"target": "source"}, this means that {"key_a": "key_b"} will result in an assignment similar to: experiment.variables['key_a'] = result_data['key_b'].

The target can be a JSON pointer as implemented by the jsonpointer Python packge. This makes it possible to only take parts of results to be a variable. For example:

# Assume result data
result_data = {
     "names": ["John", "Eric", "Terry", "Graham", "Terry"],
}

# The following variable map:
# {"name": "/names/1"}
#  Would result in the following:
experiment.variables['name'] = result_data['names'][1]  # That'd be Eric

Note

If the wait_timeout is set to 0, there won’t be a wait step and the callback will be considered finished as soon as the step finished.

Note

By default variables in the result value of the callback are not store in the experiment, to enable storing result variables back into the experiment use the variable_map attribute.

Example of a callback:

label: step2_callback1
function: test_callback
callback_arguments:
  foo: bar
  answer: 42
run_timeout: 30
wait_timeout: 0
initial_delay: 1
description: "Test callback that just sets some vars"
condition: "experiment.variables.get('test') == 42"

Conditions

Conditions for callbacks and transitions are following the same syntax. A condition string must be a line of Python code that evaluates to a True or False when run.

Examples of two conditions:

condition: "experiment.variables.get('test') == 42"
condition: "experiment.scandate > datetime.datetime(2019, 1, 1)"

For the condition evaluation only a limited set of variables are available:

• experiment: An object with the properties: label, scandate, variables and state

• datetime: The python datetime module for time calculations/comparisons.

• callbacks: A dict with the callback label as the key and the callback exection as a the value.

Note

Only if a transition condition is evaluated there are callbacks set. For conditions on starting a callback the callbacks dictionary will be empty.

The Experiment has the following properties:

Property	Type	Description
label	str	The experiment label
scandate	datetime	Datetime object with the scan time of the experiment
state	str	Label of the current state of the experiment
variables	dict	Dict containing all variables set on this experiment, the keys are the variable names, the values are variable value and the value type depends on the variable type.

The CallbackExecution has the following properties:

Property	Type	Description
status	CallbackExecutionStatus	The status of the callback, and Enum that can be used as a string too
result	CallbackExecutionResult	The result of the callback, and Enum that can be used as a string too
result_values	dict	Dictionary with the result values of the callback, the contents of this are what is returned by a callback and therefore depended on the callback function that was called.

Permissions

The permissions are defined as follows:

Permission	Description
sample_state_update	Allows updating a sample. (super-user)
sample_add	Allows adding a sample. (super-user)
sample_update	Allows updating a sample. (admin)
sample_delete	Allows deleting a sample. (admin)
roles_manage	Allows adding and removing roles from users
user_read	Allows seeing your user information.
user_read_all	Allows seeing all users.
user_add	Allows adding users.
user_update_all	Allows updating all users.
user_delete	Allows deleting users.
upload_data	Allows the uploading of data to the StudyGovernor (super-user)

Callbacks

This is a description of the various callbacks that are available in the StudyGovernor. The first two arguments of each allback are always callback_execution_data and config. These are automatically supplied by the StudyGovernor and do not need to be defined. All callbacks should always accept these two arguments as the first two arguments.

studygovernor.callbacks.command.command(callback_execution_data, config, binary, args=None, kwargs=None, xnat_external_system_name='XNAT', expected_return_code=0, **ignore)

Calls a command. This can be any command that is on the PATH of the StudyGovernor (worker) environment.

Warning

It is strongly recommened to use the external_program callback instead of possible, as its behaviour is a lot more predicable and reproducible.

The binary gets the command in the form:

binary $ARGS $KWARGS

Parameters:

• callback_execution_data – callback execution data

• config (Dict[str, Any]) – flask app configuration dict

• binary (str) – binary that gets executed

• args (Sequence[str]) – list of args [val1 val2 …]

• kwargs (Mapping[str, str]) – list of [key1 val1 key2 val2 …]

• xnat_external_system_name (Any) – name of the external xnat [XNAT]

• expected_return_code (Union[int, Sequence[int], None]) – An int or list of ints that are considered successful return codes

The items in args and values in kwargs that contain certain VARS will be replaced. Accepted VARS:

The items in args and values in kwargs that contain certain VARS will be replaced. Accepted VARS:

• $EXPERIMENT: will be substituted with the experiment URL.

• $SUBJECT: will be substituted with the subject URL.

• $XNAT: will be substituted with the XNAT URL.

• $CALLBACK_EXECUTION: Will be substituted with the callback execution URL

• $CALLBACK_SECRET: Will be substituted with the callback execution secret

Example:

function: command
callback_arguments:
  binary: check.py
  args:
    - $CALLBACK_EXECUTION
    - $CALLBACK_SECRET
  kwargs:
    -x: "$XNAT"

Result of the callback is:

Variable name	Type	Description
command	list	The command represented as a list of strings
stdout	str	The stdout of the called process
stderr	str	The stderr of the called process
return_code	int	The return code of the called process

Return type:

Dict[str, Any]

studygovernor.callbacks.create_task.create_task(callback_execution_data, config, task_base, task_info, extra_tags_var=None, xnat_external_system='XNAT', taskmanager_external_system_name='TASKMANAGER', **ignore)

Create taskmanager task

Parameters:

• callback_execution_data – callback execution data

• config (Dict[str, Any]) – flask app configuration dict

• task_base – task_base is a Template that contains info for the task

• task_info – Additional info for the task as a list of [key1 val1 key2 val2 …]

• extra_tags_var (Optional[str]) – Name of variable (in experiment) from which to extract extra tags

• xnat_external_system (str) – name of the external xnat [XNAT]

• taskmanager_external_system_name (str) – Taskmanager external ID [TASKMANAGER]

Example:

function: create_task
callback_arguments:
  task_base: manual_qa.json,
  task_info:
    project: sandbox
    application_name: ViewR
    application_version: 5.1.4
    template: manual_qa
    tags: ["QA", "Quality Assurance"]
    distribute_in_group: quality_assurance
  extra_tags_var: tag_variable
  xnat_external_system: XNAT
  taskmanager_external_system_name: TASKMANAGER

If extra_tags_var is tag_variable, the value of experiment.variable_map[tag_variable] is used to define extra tags. If the variable is not set for the experiment it is ignore. The resulting value should be a string or list of string (in which case multiple tags are added).

Return values of this callback is:

Variable name	Type	Description
response_status_code	int	Response code for the request to create the task on the taskmanager
response_text	str	Response text for the request to create the task on the taskmanager
task_info	dict	Task info sent to the taskmanager to create the task
task_uri	str	URI of the newly created task

Return type:

Dict[str, Any]

studygovernor.callbacks.create_taskgroup.create_taskgroup(callback_execution_data, config, label, tasks, distribute_in_group=None, distribute_method=None, tags=None, project=None, xnat_external_system='XNAT', taskmanager_external_system_name='TASKMANAGER', extra_tags_var=None, **ignore)

Create taskmanager task

Parameters:

• callback_execution_data – All data needed about the callback execution

• config (Dict[str, Any]) – App configuration

• label – label of the created task group

• tasks – a list of tasks to create, each should be a dict with base, template and tags defined

• distribute_in_group – group the tasks should be distributed in

• distribute_method – the method of distributed to use

• tags – list of tags to add to each tasks in the taskgroup

• project – the project for the tasks in the taskgroup

• xnat_external_system (str) – name of the external xnat [XNAT]

• taskmanager_external_system_name (str) – Taskmanager external ID [TASKMANAGER]

Example:

function: create_taskgroup
callback_arguments:
    label: 123_inspect
    distribute_in_group: raters
    tags: [rss, inspect]
    project: RSS
    tasks:
        - base: base_tissue.json
          template: tissuewml
          tags: [rss, tissue]
        - base: base_mask.json
          template: mask
          tags: [rss, mask]
          project: OVERWRITE
        - base: base_lobes.json
          template: lobes
          tags: [rss, lobes]
    extra_tags_var: tag_variable

Return values of this callback is:

Variable name	Type	Description
response_status_code	int	Response code for the request to the taskmanager
response_text	str	Response text for the request to the taskmanager
task_info	dict	Task info sent to the taskmanager to create the task group
task_uri	str	URI of the newly created task

Return type:

Dict[str, Any]

studygovernor.callbacks.external_program.external_program(callback_execution_data, config, binary, args=None, kwargs=None, xnat_external_system_name='XNAT', expected_return_code=0, **ignore)

Calls a binary from the StudyGovernor binaries directory. This is configured using the STUDYGOV_PROJECT_BIN configuration field.

The binary gets the command in the form:

binary $ARGS $KWARGS

Parameters:

• callback_execution_data – callback execution data

• config (Dict[str, Any]) – flask app configuration dict

• binary (str) – binary that gets executed

• args (Optional[Sequence[str]]) – list of args [val1 val2 …]

• kwargs (Optional[Mapping[str, str]]) – list of [key1 val1 key2 val2 …]

• xnat_external_system_name (str) – name of the external xnat [XNAT]

• expected_return_code (Union[int, Sequence[int], None]) – An int or list of ints that are considered successful return codes

The items in args and values in kwargs that contain certain VARS will be replaced. Accepted VARS:

• $EXPERIMENT: will be substituted with the experiment URL.

• $SUBJECT: will be substituted with the subject URL.

• $XNAT: will be substituted with the XNAT URL.

• $CALLBACK_EXECUTION: Will be substituted with the callback execution URL

• $CALLBACK_SECRET: Will be substituted with the callback execution secret

Example:

function: external_program
callback_arguments:
  binary: check.py
  args:
    - $CALLBACK_EXECUTION
    - $CALLBACK_SECRET
  kwargs:
    -x: "$XNAT"

Result of the callback is:

Variable name	Type	Description
command	list	The command represented as a list of strings
stdout	str	The stdout of the called process
stderr	str	The stderr of the called process
return_code	int	The return code of the called process

Return type:

Dict[str, Any]

studygovernor.callbacks.fastr.fastr(callback_execution_data, config, network_id, source_mapping, sink_mapping, xnat_external_system_name='XNAT', fastr_home=None, **ignore)

Execute Fastr pipeline

Parameters:

• callback_execution_data – All data needed about the callback execution

• config (Dict[str, Any]) – App configuration

• network_id (str) – network that gets executed

• source_mapping (Mapping[str, str]) – Mapping[str, str],

• sink_mapping (Mapping[str, str]) – Mapping[str, str],

• fastr_home (Optional[str]) – optionally, set the FASTRHOME variable passed to fastr

Example:

function: fastr
callback_arguments:
  network_id: preprocessing
  source_mapping:
    t1: /scans/T1W*/resources/DICOM
    flair: /scans/*FLAIR*/resources/DICOM
  sink_mapping:
    t1_nii: /scans/T1W*/resources/NIFTI/files/image{ext}
    flair_nii: /scans/*FLAIR*/resources/NIFTI/files/image{ext}
    flair_coregistered: /scans/T1W*/resources/NIFTI/files/flair_to_t1{ext}
  xnat_external_system_name: XNAT
  fastr_home: /path/to/fastr_home

Note

If setting fastr_home this needs to be available on the worker nodes, as the callback will be executed on workers and not the main StuyGovernor process.

Variable name	Type	Description
xnat_uri	str	URL of the XNAT server used to create the source and sink data
project	str	The XNAT project used to create the source and sink data
subject	str	The XNAT subject id used to create source and sink data
experiment	str	The XNAT experiment id used to create source and sink data
source_mapping	dict	The source_mapping used
sink_mapping	dict	The sink_mapping used
label	str	The experiment label use as the sample id for fastr

Return type:

Dict[str, Any]

studygovernor.callbacks.python_function.python_function(callback_execution_data, config, package, module, function, pass_callback_data=True, args=None, kwargs=None, **ignore)

Calls a python funciton. The function should return a dictionairy.

Parameters:

• callback_execution_data – callback execution data

• config (Dict[str, Any]) – flask app configuration dict

• package (str) – package that gets imported

• module (str) – module that gets loaded

• function (str) – function that gets called

• pass_callback_data (bool) – flag that indicates whether or not the callback_execution_data and config have to be given to the function as the first arguments

• args (Sequence[str]) – list of args [val1 val2 …]

• kwargs (Mapping[str, str]) – mapping of keyword arguments {key1: val1, key2: val2}

The items in args and values in kwargs that contain certain VARS will be replaced. Accepted VARS:

The items in args and values in kwargs that contain certain VARS will be replaced. Accepted VARS:

• $EXPERIMENT: will be substituted with the experiment URL.

• $SUBJECT: will be substituted with the subject URL.

• $XNAT: will be substituted with the XNAT URL.

• $CALLBACK_EXECUTION: Will be substituted with the callback execution URL

• $CALLBACK_SECRET: Will be substituted with the callback execution secret

Example:

function: external_program
callback_arguments:
  package: os
  module: path
  function: join
  args:
    - $SUBJECT
    - $EXPERIMENT

Result of the callback is:

Variable name	Type	Description
package	str	package where function resides in
module	str	name of module where function resides in
function	str	name of the function to run
args	list	list of arguments used to call function
kwargs	dict	list of keyword arguments used to call the function
return_value	any	return value of the function that was called
__traceback__	str	in case of an error, the traceback as a string
__exception__	str	in case of an error, the error message
__success__	bool	a boolean that indicates if the function was successful

Return type:

Dict[str, Any]

studygovernor.callbacks.send_mail.send_mail(callback_execution_data, config, subject, body, **ignore)

Send an email with SUBJECT and BODY.

Parameters:

• callback_execution_data (Dict) – All data needed about the callback execution,

• config (Dict[str, Any]) – App configuration,

• subject (str) – str,

• body (str) – str):

Substitution fields for SUBJECT and BODY are:

• {experiment}: experiment id

• {experiment_url}: full url for an experiment

• {action_url}: full url for an action.

Example:

function: send_mail
callback_arguments:
  subject: Test mail for {experiment}
  body: The email body, this is about {experiment} located at {experiment_url}

Result of the callback is:

Variable name	Type	Description
sender	str	sender used when sending email
recipient	str	recipient(s) used when sending the email
subject	str	subject used when sending email, with the fields substituted
body	str	the body used when sending email, with the fields substituted

Return type:

Dict[str, Any]

studygovernor.callbacks.sleep.sleep(callback_execution_data, config, seconds, **kwargs)

Simple callback that just sleeps for a number of seconds

Parameters:

• callback_execution_data (Dict) –

• config (Dict[str, Any]) –

• seconds (int) – How many seconds to sleep

• kwargs – Extra arguments are added in the return value

Return type:

Dict[str, Any]

Returns:

Example:

function: sleep
callback_arguments:
  seconds: 10
  kwargs:
    extra_random_argument: 42

kwargs can contain any nested data. It will simply be return by the callback. This is meant for either debug purposes or to inject extra variables.

Result of the callback are:

Variable name	Type	Description
sleep_duration	int	How long the callback slept for
extra_kwargs	dict	The values supplied to kwargs are returned here