# User notifications in LAVA

Users can receive notifications containing information about submitted test
jobs and the accompanying results by using settings in job definition yaml.
Users can also compare the job results within notifications against previous
results which satisfy certain conditions.

The basic setup of the notify block in job definitions will have
**criteria**, **verbosity**, **recipients** and **compare** blocks.

+ **Criteria** tells the system when the notifications should be sent.

+ **verbosity** will tell the system how detailed the email notification should
  be.

+ **Criteria** can be set by job state (running, complete, incomplete,
  canceled) and the type (regression, progression). There is a special status of
  'finished', which will match any of (complete, incomplete, canceled). You
  could also use "all" to receive a notification for both "running" and
  "finished" states.

The basic **email** template includes job details, device details and results,
based on the level of **verbosity** (verbose, quiet, status-only).
verbose level job information will only be included if the job finished as
complete or incomplete, not when the job was canceled.

## Notification recipients

The method and destination of each notification can be set for each recipient.
Currently, two notification methods are supported, **email** and **IRC**.

Recipients can be specified using the LAVA username or in full.

If the **recipients** section is omitted in the notify block, the system will
send an email to the job submitter only, provided the **criteria** is satisfied,
and there is no **callbacks** section.

## Notification callbacks

In addition to sending email and IRC messages to **recipients**, the system can
send multiple URL callback actions. This will do a GET or POST request to the
specified URL in the **callbacks** subsection. This can be used to trigger some
action remotely. If a callback uses the POST request, the system will attach job data as described below.
The **callbacks** section supports list of the following options:

+ **url** The URL for the request. This also supports field value
  substitution, i.e. in http://example.com/{ID}/{STATUS} **id** and
  **status** will be replaced with corresponding values from the job.

+ **method** GET or POST

+ **token** This option is used to supply the API token of the
  authenticated user, appended as the POST request parameter. If the submitting
  user has an XMLRPC auth token with a description that matches this field, that
  token is returned instead. The token is included in the POST data, and also in
  an Authorization header.

+ **dataset** This option specifies style of data that the system
  will provide in the callback. It applies only for the POST request. The format
  of the data and possible options are as following:

    + **minimal** This will provide basic job info such as job id, status,
    submit_time, start_time, end_time, submitter_username, submitter_email,
    failure_comment, priority, description, actual_device_id, definition,
    health_check and metadata.
    + **logs** In addition to minimal data this will also attach the job log
    output in the url encoded format
    + **results** In addition to minimal data this will also attach the job
    results as a list of test suites exported in yaml format.
    + **all** In addition to minimal data this will include both the **logs**
    and **results** datasets as described above.

+ **content-type** This option is used to determine how the POST data is submitted:

    + **urlencoded** (Default) Will return a standard HTTP POST request, with an
    application/x-www-form-urlencoded Content-Type header and data sent as an
    urlencoded query string.
    + **json** The data is dumped into JSON and returned with a ``Content-Type:
    application/json`` header.

Example callback usage:

```yaml
notify:
  criteria:
    status: finished
  callbacks:
    - url: https://my-url/callback
      method: GET
    - url: https://remote-site/callback/lava?status={STATUS}&status_string={STATUS_STRING}
      method: POST
      dataset: all
      token: ci-callback
      content-type: json

```

### Restricting callback URLs

By default, notification callbacks can target any URL. Administrators can
restrict which hosts are permitted by setting `CALLBACK_ALLOWED_HOSTS` in
`/etc/lava-server/settings.yaml` or a file in `/etc/lava-server/settings.d/`:

```yaml
CALLBACK_ALLOWED_HOSTS:
  - "example.com"
  - "*.example.com"
  - "*.ci.example.com"
```

When this setting is defined, only callback URLs whose hostname matches one of
the patterns will be accepted. All other callback URLs will be rejected and a
warning logged. Shell-style wildcards are supported:

- `example.com` — matches `example.com` only
- `*.example.com` — matches `ci.example.com`, `api.example.com` but not
  the bare `example.com`
- `*.ci.example.com` — matches `build.ci.example.com`,
  `test.ci.example.com`

When `CALLBACK_ALLOWED_HOSTS` is not set, all callback URLs are permitted.

!!! warning "Security recommendation"

    It is recommended to configure `CALLBACK_ALLOWED_HOSTS` in production
    deployments to prevent the LAVA server from being used to reach internal
    network services via crafted callback URLs.

### Debugging notification callbacks

The job data can also be retrieved using the REST API (which supports
authentication). For example::

```
wget -O job_data.gz http://localhost/scheduler/job/2126/job_data
```

```
wget -O job_data.gz "http://localhost/scheduler/job/2127/job_data?user=joe&token=22yj3ls...."
```

Returns a gzip file containing the job data as JSON.

!!! info "callback data retrieval"

    Only test jobs which are configured to use the notification callback
    will create notification callback data for later retrieval. Other jobs will
    generate a 404 error.

### Using profile settings

LAVA users can configure the ``IRC settings`` or email address in their own
profile data in the instance. This allows the recipient to be specified using
only the LAVA username.

![profile menu](profile-menu.png)

### Direct listing of recipients

If the user has not configured their own profile data, the recipient details
must be specified in full.

Examples for user vs manual addressing:

```yaml
notify:
  recipients:
  - to:
     method: irc
     user: neil.williams
  - to:
     method: irc
     handle: codehelp
     server: irc.freenode.net
  - to:
     method: email
     user: neil.williams
  - to:
     method: email
     email: neil.williams@linaro.org
```

IRC and email notifications use different templates since emails allow for more
verbosity, so some options which are present in the ``notify`` block for email
recipients do not apply for IRC recipients, like **comparing** options and
**verbosity**.


## Result comparison in notifications

Users can use notifications to compare the job status against the last few jobs
and also compare job results, like test case/suites difference. The
notification will report if there are new test cases compared to previous test
jobs, if there are some missing and how the said results differ.

Which test jobs the current one is compared to is determined by the ``queries``
system, where those previous jobs will be taken from the query specified in the
notification block. ``Custom queries`` can also be specified.

**Blacklisting** test cases is also an option. Ignoring some of the test cases
in the comparing section of the notification is possible through an option in
the notification block.

Here are some comparing setup examples from test definition excerpts:

```yaml
notify:
  recipients:
  - to:
     method: email
     user: neil.williams
  criteria:
    status: finished
  verbosity: verbose
  compare:
    query:
      name: query_name
      username: query_owner
    blacklist: [lava, singlenode-advanced]
```
```yaml
notify:
  recipients:
  - to:
     method: email
     user: neil.williams
  criteria:
    status: finished
  verbosity: verbose
  compare:
    query:
      entity: testjob
      conditions:
        submitter: stevan.radakovic
        requested_device_type: qemu
    blacklist: [singlenode-advanced]
```
