Query language basics
Introduction
Aim enables a powerful query language(Aim QL) to filter through all the stored metadata.
AimQL filters the tracked metadata using python expression. Think of it as a python if statement over everything you have tracked. Hence, nearly any python compatible expression is available with some restrictions in place.
The data is saved as different types of entities (e.g. run
, metric
). The search queries are written against these entities.
When iterating over entities the python expression is evaluated in a Boolean context. When the value is “truthy”, then the current entity is yielded. Otherwise the entity is skipped over.
Currently, AimQL is only used for filtering data, and has no role in sorting or aggregating the data.
Searching
Let’s track several Runs via Aim SDK:
from aim import Run
# Initialize run_1
# Define its params and track loss metric within test and train contexts
run_1 = Run()
run_1['learning_rate'] = 0.001
run_1['batch_size'] = 32
for i in range(10):
run_1.track(i, name='loss', context={ 'subset':'train' })
run_1.track(i, name='loss', context={ 'subset':'test' })
# Initialize run_2
run_2 = Run()
run_2['learning_rate'] = 0.0007
run_2['batch_size'] = 64
for i in range(20):
run_2.track(i, name='loss', context={ 'subset':'train' })
run_2.track(i, name='loss', context={ 'subset':'test' })
run_2.track(i/100, name='accuracy')
# Initialize run_3
run_3 = Run()
run_3['learning_rate'] = 0.005
run_3['batch_size'] = 16
for i in range(30):
run_3.track(i, name='loss', context={ 'subset':'train' })
run_3.track(i, name='loss', context={ 'subset':'test' })
run_3.track(i/100, name='accuracy')
Aim SDK will collect and store the above metadata in .aim
repo.
Run |
Parameters |
Metrics |
||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|
|
|
|
||||||||||
|
|
|
||||||||||
|
|
|
When searching runs, use the run
keyword which represents the Run
object. It has the following properties:
Property |
Description |
---|---|
|
Run name |
|
Run hash |
|
Experiment name |
|
List of run tags |
|
|
|
|
|
Run duration in seconds |
|
Run creation datetime |
|
Run end datetime |
|
Set of run metrics |
Run parameters could be accessed both via chained properties and attributes.
The two following examples are equal:
run.hparams.learning_rate == 32
run[“hparams”, “learning_rate”] == 32
AimQL has been designed to be highly performant. Only the params that are used in the query will be loaded into memory.
If you use the [‘hparams’][‘learning_rate’] syntax Aim will load the whole dictionary into memory. The search performance will be impacted.
We recommend to use either [‘hparams’, ‘learning_rate’] or hparams.learning_rate syntax which are equivalent to each other in terms of the performance.
Query examples:
Get runs where
learning_rate
is greater than0.0001
andbatch_size
is greater than32
.
run.learning_rate > 0.0001 and run.batch_size > 32
Result:
Run |
Parameters |
||||
---|---|---|---|---|---|
|
|
Get runs where
learning_rate
is either0.0001
or0.005
.
run.learning_rate in [0.0001, 0.005]
Result:
Run |
Parameters |
||||
---|---|---|---|---|---|
|
|
||||
|
|
Search runs based on metrics’ last values.
metrics
property takes 2 arguments: metric name and context (name is required, context is empty({}) by default).
Get runs where
accuracy, {}
last value is greater than 0.25
run.metrics['accuracy'].last > 0.25
Result:
Run |
Parameters |
Metrics |
||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|
|
|
|
Get runs where
loss, {'subset': 'train'}
last value is smaller than 25
run.metrics['accuracy', {'subset': 'train'}].last < 25
Result:
Run |
Parameters |
Metrics |
||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|
|
|
|
||||||||||
|
|
|
Search runs where
duration
is smaller than 10s (to filter out failed runs for example) .. code-block:: pythonrun.duration < 10
This will return all the runs on our test set, but will have a use case on real life experiments to filter the runs that finished under 10 seconds. Then those can be archived/deleted as they’ve probably crashed or were stopped.
Search runs based on creation or end times
created_at
and finalized_at
properties are represented as python datetime
objects. So any of these queries are a competent way to filter runs:
Get runs for a specific date
run.created_at.strftime("%d/%m/%Y") == "24/12/2021"
Get runs after a specific date
run.finalized_at > datetime(2022, 2, 2)
Get runs of a specific month
run.created_at.month == 2 and run.created_at.year == 2022
Searching metrics and images
Searching metrics
When iterating over metrics, use the metric
keyword which represents the tracked metric. While searching metrics, you can also refer to the related runs via the run
keyword.
metric
has the following default properties.
Property |
Description |
---|---|
|
Metric name |
|
Metric context dictionary |
|
Last tracked value of the metric |
|
Last tracked step number of the metric |
|
First tracked step number of the metric |
Query examples
Query metrics by name:
metric.name == "loss"
Result:
Metric |
Related run |
---|---|
|
|
|
|
|
|
|
|
|
|
|
|
Query metrics by name and context
metric.name == "loss" and metric.context.subset == "train"
Result:
Metric |
Related run |
---|---|
|
|
|
|
|
|
Query metrics by name and run parameters
metric.name == "loss" and run.learning_rate >= 0.001
Result:
Metric |
Related run |
---|---|
|
|
|
|
|
|
|
|
Query metrics by name and last value
metric.name == "loss" and metric.last >= 15
Result:
Metric |
Related run |
---|---|
|
|
|
|
|
|
|
|
Searching images
Images search works in the same way as metrics.
When iterating over images, use the images
keyword which represents the tracked images sequence.
While searching images, you can also refer to the related runs via the run
keyword.
images
keyword has the following default properties.
Property |
Description |
---|---|
|
Image sequence name |
|
Image sequence context dictionary |
Query examples:
images.name == “generated” and run.learning_rate >= 0.001
images.name == “generated” and images.context.ema == 0
Security restrictions
AimQL expression is evaluated with RestrictedPython.
RestrictedPython is a tool that helps to define a subset of the Python language which allows to provide a program input into a trusted environment.
We have followed these restrictions to avoid security risks such as executing a non-safe function via AimQL.