LION.evaluation.comparison

Comparing reconstruction methods on one evaluation.

Comparison(evaluation, methods) runs each method, and a baseline, on the same Evaluation, after checking from the models’ records that the comparison is fair:

• no leakage: a model trained or validated on test data is an error. The check is by group where the data have groups (LIDC-IDRI’s patients, whose slices are near-copies; 2DeteCT’s physical samples), otherwise by sample;

• the same problem: a model trained on other data or noise (another experiment, dose, geometry) gives a warning naming the differences;

• the same code: a model trained with other LION code is listed.

Methods without a record (a model or reconstructor made in the script) can’t be checked, which is said; classical methods (functions) need no check. The baseline is the experiment’s (Experiment.baseline()) or the modality’s (LION.evaluation.baseline: the operator’s default reconstruction, FDK for CT).

Classes

Comparison(evaluation, methods[, baseline, ...])

Several methods on one Evaluation, against a baseline (see the module docstring).

Functions

problem_differences(test, training)	How the problem of training differs from that of test (data records): the experiment, if both have one; otherwise the dataset (but its split) and its noise.
shared_samples(test, training)	The test data's samples (or groups) that are also in training, both data records.

---

class LION.evaluation.comparison.Comparison(evaluation, methods, baseline=None, allow=())

Bases: object

Several methods on one Evaluation, against a baseline (see the module docstring).

methods: {name: method}, or a list (a file is named by its file name, anything else as Evaluation names it); a method is anything LION.evaluation.method takes. baseline: a method; None: the experiment’s, else the modality’s; False: no baseline. allow: modules outside LION that model files may name.

named_methods()

{name: method}, the baseline first.

get_baseline()

The baseline: the one given, the experiment’s, or the modality’s.

check()

Checks the methods against the test data (see the module docstring): raises if a model was trained or validated on test data; returns what else differs, as sentences.

run(*, save=True, verbose=1)

Checks the methods (warning what differs), then runs each on the evaluation (reusing saved results). Returns {name: test record}, the baseline first, also kept as self.records. verbose: 1 (default) shows a progress bar per method; 0 nothing.

table(format='markdown', digits=None)

The records as a table (LION.evaluation.table), the baseline first.

baseline_name()

The baseline’s name (its row), or None without a baseline.

figures()

The score plots, {file name: Figure}: per metric, a box plot of every method and, with a baseline, each method’s per-sample difference to it.

chosen_samples(ids=None, by=None)

{row label: sample id}: the given ids, or the samples where the baseline (or the first method) scores best, median and worst on by (default: the first metric).

images(ids=None, by=None, zoom=None, differences=False)

Example images (LION.evaluation.figures.example_images): the targets and every method’s images of the chosen samples (see chosen_samples), each with its by score; zoom: [(row, col, height, width), ...]; differences: to the target.

save(folder, ids=None, by=None, zoom=None, differences=False)

Writes the comparison to folder: comparison.json (its spec, to make it again with Comparison.instantiate(folder), and what was drawn), records/ (the test records), table.md/.tex/ .csv, scores.csv (per-sample scores: one row per sample, one column per method and metric), and each figure as .pdf and .png (figures(), and images.* of images(ids, by, zoom, differences)). Returns the folder.

spec()

This comparison’s spec: its evaluation’s, its methods and baseline (files by path; methods made in the script only named).

classmethod instantiate(source, *, allow=(), **arguments)

The comparison a saved one describes: source is the folder save wrote, its comparison.json, or a spec. arguments replace recorded ones; evaluation_arguments (a dict) are given to the evaluation (e.g. {"save_folder": ...}). allow: modules outside LION the file may name (the recorded allow isn’t used: a file can’t widen what it may build).

LION.evaluation.comparison.problem_differences(test, training)

How the problem of training differs from that of test (data records): the experiment, if both have one; otherwise the dataset (but its split) and its noise. Where the data were read from isn’t compared.

LION.evaluation.comparison.shared_samples(test, training)

The test data’s samples (or groups) that are also in training, both data records. Returns (shared, what they are, a note or None): by group when both records have them, otherwise by sample; data from another dataset share none.