Set up the notification service
Being notified about key events of the training process may save hours of compute resources and researcher’s time. Aim notification service sends notifications to the enabled channels (slack, workplace), when a new message is logged or a stuck run is detected.
Quick start
These are the required steps in order to set up the notifications service.
Configure notifiers. The notifier configuration is described in detail in Configuring Notifiers section.
Start notifier service:
aim-watcher --repo <AIM_REPO_PATH> start
Aim notifications service is comprised of the following components:
Aim watcher service. A service responsible for sending notifications.
Notifier configuration CLI. Command line utility for configuring where and how you want to receive notifications about training run status.
Below are the components described in more detail.
Configuring notifiers
Use aim-watcher
CLI to interactively choose and setup the notifiers.
Currently, following types of notifiers are supported:
Slack: to receive notifications as a message on a slack channel.
Workplace: to receive notifications as a post in a workplace group.
Logger: to echo notification text to watcher standard output.
The base command is:
aim-watcher notifiers add
Once notifier type is selected, the watcher will ask to fill-in the required inputs for that particular notifier.
aim-watcher
allows configuring multiple notifiers in case you’d like to receive them in
multiple channels at the same time. Once notifier configuration added, aim-watcher
should be re-run, so
new configuration can take effect.
Next sections will describe the setup flow for each of them.
Configuring notifier for slack
In order to execute the steps described below you must have admin permissions for Slack workspace.
Here are the steps to configure Aim notifier for Slack.
Go to Slack API web page and push “Create an app” button.
Select “From scratch” and follow the steps.
Make sure Incoming Webhooks are active. This is the Slack API Aim will use to send notifications.
Create a new Webhook URL and copy to clipboard.
Run
aim-watcher notifiers add slack
and once prompted for Webhook url, paste the one copied in step 4.Optional Change the training run failure message template.
Confirm setup.
Configuring notifier for workplace
In order to execute steps described below, please make sure you have access to Workplace Admin panel
Here are the steps to configure Aim notifier for Workplace.
#. Go to “Your workplace” Admin Panel #.
Select “Integrations” tab and press “Create custom integration”
Follow the setup steps.
In the “Permissions” tab make sure “Manage group content” checkbox checked.
Select the group(s) where you want to receive notifications.
In the “Details” tab press “Create access token” and copy the generated access token.
Run
aim-watcher notifiers add workplace
and provide group ID and copied access token.Optional Change the training run failure message template.
Confirm setup.
Enabling/disabling notifier configuration
Aim watcher CLI allows to temporarily disable or completely remove notifiers. Each notifier configuration has unique auto-generated ID which can be used to manipulate the notifier. In order to list available notifiers for the aim Repo:
aim-watcher --repo . notifiers list
> NOTIFIER ID TYPE STATUS
> 88bbacb3-159d-4ee2-8f33-180addd162f1 slack enabled
> 75fe529d-c060-4816-9cbb-246770cb8592 slack disabled
> 248821e4-2bf3-4aa9-857e-194bf3ab2f8a logger enabled
You can disable notifier by given ID:
aim-watcher --repo . notifiers disable 88bbacb3-159d-4ee2-8f33-180addd162f1
and enable it back:
aim-watcher --repo . notifiers enable 88bbacb3-159d-4ee2-8f33-180addd162f1
In order to remove notifier configuration completely:
aim-watcher --repo . notifiers remove 88bbacb3-159d-4ee2-8f33-180addd162f1
Complete reference for aim-watcher CLI is available here.
Configuring notification levels
The notification levels to notify on logged messages can be configured.
Available options are DEBUG
, INFO
, WARNING
, ERROR
.
Run aim-watcher notifiers set-log-level [LVL]
command to set the level.
Note
WARNING is set as a default level.