LION.evaluation.evaluation

Evaluating a reconstruction method on a test set.

Evaluation is the standard evaluation of one problem, like an Experiment: the test data (an experiment’s split, or a dataset), the operator, the metrics (PSNR, SSIM, HaarPSI) and how their range is set. Its run(method) scores every test sample and gives a test record:

• name, method (what was run, e.g. a model’s spec and weights hash);

• evaluation (this evaluation’s spec), data (the test data’s record, LION.data_loaders.data_record), metrics (their specs; and whether higher_is_better), data_range;

• scores: the sample ids and, per metric, the score of each sample;

• summary: per metric mean, std, median, q25, q75, min, max; seconds_per_sample; environment.

Records are LIONParameter``s, saved as plain JSON in ``save_folder (by default $LION_RESULTS_PATH/evaluations). Running the same method on the same evaluation again reads the saved record instead of recomputing it.

The functions work on their own too: scores (one method on some data), summary, table (records as markdown, LaTeX or CSV).

Classes

Evaluation([experiment, dataset, operator, ...])

The standard evaluation of one problem (see the module docstring).

Functions

default_metrics()	PSNR, SSIM and HaarPSI.
is_test_record(value)	Whether value (a dict or a LIONParameter) is a test record (Evaluation's): it has the record's key.
read_test_record(source)	A test record: a record, or the path of its .json.
sample_ids(data)	The IDs of a dataset's samples (their indices if it has none).
scores(run, data, metrics[, data_range, ...])	Every sample of data (a dataset of (measurement, target)) through run, in order, scored by every metric.
summary(record)	The summary of a test record, {metric: {mean, std, ...}}.
summary_of(values)	Mean, std, median, quartiles, min and max (NaNs left out).
table(records[, format, digits])	Records as a table: one row per method (its name), one column per metric, mean ± std, the best of each column in bold.

---

class LION.evaluation.evaluation.Evaluation(experiment=None, dataset=None, operator=None, split='test', metrics=None, normalisation='dataset', samples=None, batch_size=1, save_folder=None, device=None)

Bases: object

The standard evaluation of one problem (see the module docstring).

experiment: its split is the test data, and its geometry gives the operator; or dataset (and operator) for data set up without an experiment. metrics: {name: metric}, by default PSNR, SSIM and HaarPSI. normalisation: the range the metrics use: "dataset" (the test samples’ minimum and maximum, the same for every method; default), "sample" (each target’s own range), (min, max), or a LION Normalisation. samples: all (None), n (evenly spaced over the test set) or a list of sample IDs. save_folder: where records are saved and reused (default $LION_RESULTS_PATH/evaluations; None without it). device: where methods run (default LION’s).

data()

The test samples (the dataset, or the chosen ones of it).

get_operator()

The operator: the one given, or the experiment’s.

run(x, name=None, *, save=True, allow=(), verbose=1)

Score x (a model, a model’s file, a solver, a reconstructor or a function f(y, operator); see LION.evaluation.method) on every test sample. Returns its test record, saved in save_folder; a record saved before for the same method and evaluation is read instead. allow: modules outside LION a model file may name (e.g. your own solver). verbose: 1 (default) shows a progress bar over the test samples; 0 nothing.

data_range(evaluation_key=None)

(min, max) the metrics use, or None (each target’s own).

spec()

This evaluation’s spec (its arguments; not where it saves or runs).

data_record()

The test data’s record (made once).

classmethod instantiate(source, *, allow=(), **arguments)

The evaluation a test record (or its spec) describes; arguments replace recorded ones (e.g. save_folder).

LION.evaluation.evaluation.default_metrics()

PSNR, SSIM and HaarPSI.

LION.evaluation.evaluation.is_test_record(value)

Whether value (a dict or a LIONParameter) is a test record (Evaluation’s): it has the record’s key.

LION.evaluation.evaluation.read_test_record(source)

A test record: a record, or the path of its .json.

LION.evaluation.evaluation.sample_ids(data)

The IDs of a dataset’s samples (their indices if it has none).

LION.evaluation.evaluation.scores(run, data, metrics, data_range=None, batch_size=1, device=None, verbose=0, desc=None)

Every sample of data (a dataset of (measurement, target)) through run, in order, scored by every metric. data_range: (min, max) (outputs and targets are shifted by min and the metrics get max - min), or None for each target’s own range.

Returns (ids, {name: numpy array of per-sample scores}, seconds per sample). verbose >= 1: a progress bar over the samples (desc).

LION.evaluation.evaluation.summary(record)

The summary of a test record, {metric: {mean, std, ...}}.

LION.evaluation.evaluation.summary_of(values)

Mean, std, median, quartiles, min and max (NaNs left out).

LION.evaluation.evaluation.table(records, format='markdown', digits=None)

Records as a table: one row per method (its name), one column per metric, mean ± std, the best of each column in bold. format: "markdown", "latex" (booktabs) or "csv" (mean and std columns). digits: decimals per metric (default: 2 for values of 10 or more, else 4).