Evaluation and comparison

A result in a paper is a method scored on test data, against others. LION makes that one standard procedure per problem:

• Evaluation: the test data, the operator and the metrics of one problem (an experiment). evaluation.run(method) scores any method on every test sample and returns a test record, saved and reused;

• Comparison: several methods on one evaluation, against a baseline. It first checks, from the methods’ records, that the comparison is fair, then writes the tables and figures a paper needs.

Previous: Reconstructors. Next: Publishing and reproducing.

Needs a GPU and the processed LIDC-IDRI data in LION_DATA_PATH.

Note

The models trained in these tutorials are toys: a few epochs on a few slices, to show how LION is used, so their results are poor (often worse than FDK). Trained as their papers train them (many epochs, all the data), they do what the papers report.

Methods to compare

A method is a model, a model’s file, a solver, a reconstructor (or its file), or a function f(measurements, operator). Here two FBPConvNets trained briefly (one more than the other), and SIRT:

import functools
import tempfile
from pathlib import Path

import torch
from torch.utils.data import DataLoader, Subset

from LION.experiments import ct_experiments
from LION.models.post_processing.FBPConvNet import FBPConvNet
from LION.operators.ct import sirt
from LION.optimizers.SupervisedSolver import SupervisedSolver

experiment = ct_experiments.LowDoseCTRecon()
training = Subset(experiment.get_dataset("train"), range(40))
training_loader = DataLoader(training, batch_size=4, shuffle=True)
folder = Path(tempfile.mkdtemp())

for epochs in (1, 3):
    model = FBPConvNet(experiment.operator_params)
    solver = SupervisedSolver(
        model, torch.optim.Adam(model.parameters()), torch.nn.MSELoss(), verbose=0
    )
    solver.set_training(training_loader)
    solver.set_saving(folder, f"FBPConvNet_{epochs}")
    solver.train(epochs)
    solver.save_final_results()

/mhome/damtp/r/ab2860/AItomotools/LION/data_loaders/LIDC_IDRI.py:487: UserWarning: LIDC-IDRI: left out 2 damaged or unusable files (re-run the pre-processing for these patients): ['LIDC-IDRI-0238 (not resampled)', 'LIDC-IDRI-0585 (not resampled)']
  warnings.warn(

An evaluation

The experiment’s test split, scored with PSNR, SSIM and HaarPSI. The metrics’ range is the test images’ (the same for every method). samples takes evenly spaced test slices (all of them by default); results are saved in save_folder.

from LION.evaluation import Evaluation, summary

evaluation = Evaluation(experiment, samples=20, save_folder=folder / "evaluations")
record = evaluation.run(folder / "FBPConvNet_3", "FBPConvNet, 3 epochs")

for metric, values in summary(record).items():
    print(f"{metric}: {values['mean']:.3f} ± {values['std']:.3f}")

PSNR: 16.081 ± 5.456
SSIM: 0.165 ± 0.045
HaarPSI: 0.347 ± 0.104

The test record holds the score of each test sample (by patient and slice), what was evaluated (the model’s spec and weights hash), the test data’s record and the environment. Running the same method on the same evaluation again reads it instead of computing it:

print(record.scores.ids[:3], record.scores.PSNR[:3])
again = evaluation.run(folder / "FBPConvNet_3", "FBPConvNet, 3 epochs")

['LIDC-IDRI-0910/slice_0', 'LIDC-IDRI-0915/slice_114', 'LIDC-IDRI-0920/slice_331'] [17.426639556884766, 17.62146759033203, 16.989444732666016]

A comparison

Methods by name, against the problem’s baseline (FDK, for CT):

from LION.evaluation import Comparison

comparison = Comparison(
    evaluation,
    {
        "FBPConvNet, 1 epoch": folder / "FBPConvNet_1",
        "FBPConvNet, 3 epochs": folder / "FBPConvNet_3",
        # a partial, so it can be recorded
        "SIRT": functools.partial(sirt, num_iterations=100),
    },
)
records = comparison.run()
print(comparison.table())
print(comparison.table("latex"))

| Method | PSNR | SSIM | HaarPSI |
|---|---|---|---|
| fdk | 21.87 ± 5.60 | 0.1541 ± 0.0933 | 0.3945 ± 0.2007 |
| FBPConvNet, 1 epoch | 22.69 ± 6.63 | 0.2599 ± 0.1811 | 0.4309 ± 0.2143 |
| FBPConvNet, 3 epochs | 16.08 ± 5.46 | 0.1652 ± 0.0445 | 0.3473 ± 0.1035 |
| SIRT | **33.03 ± 2.19** | **0.7872 ± 0.1425** | **0.6543 ± 0.0618** |

\begin{tabular}{lccc}
\toprule
Method & PSNR & SSIM & HaarPSI \\
\midrule
fdk & 21.87 $\pm$ 5.60 & 0.1541 $\pm$ 0.0933 & 0.3945 $\pm$ 0.2007 \\
FBPConvNet, 1 epoch & 22.69 $\pm$ 6.63 & 0.2599 $\pm$ 0.1811 & 0.4309 $\pm$ 0.2143 \\
FBPConvNet, 3 epochs & 16.08 $\pm$ 5.46 & 0.1652 $\pm$ 0.0445 & 0.3473 $\pm$ 0.1035 \\
SIRT & \textbf{33.03 $\pm$ 2.19} & \textbf{0.7872 $\pm$ 0.1425} & \textbf{0.6543 $\pm$ 0.0618} \\
\bottomrule
\end{tabular}

Before running, the comparison checks each method’s records and stops on what makes it unfair:

• a model trained or validated on a test patient (by patient, not only by slice);

and warns about what makes it doubtful:

• models trained for another problem (another geometry or noise);

• results made with other LION code (another commit, uncommitted changes).

Saving it

save writes everything: the table (markdown, LaTeX, CSV), the score of every sample, the test records, box plots of the scores, and images of chosen samples (by default the best, median and worst by the baseline’s PSNR), with zoomed crops and the differences to the target.

saved = comparison.save(
    folder / "comparison", zoom=[(200, 200, 100, 100)], differences=True
)
print(sorted(p.name for p in saved.iterdir()))

['box_HaarPSI.pdf', 'box_HaarPSI.png', 'box_PSNR.pdf', 'box_PSNR.png', 'box_SSIM.pdf', 'box_SSIM.png', 'comparison.json', 'difference_HaarPSI.pdf', 'difference_HaarPSI.png', 'difference_PSNR.pdf', 'difference_PSNR.png', 'difference_SSIM.pdf', 'difference_SSIM.png', 'images.pdf', 'images.png', 'records', 'scores.csv', 'table.csv', 'table.md', 'table.tex']

The figures are also there to show or change:

figures = comparison.figures()
images = comparison.images(zoom=[(200, 200, 100, 100)])

The saved comparison makes itself again, e.g. to add a method later:

comparison = Comparison.instantiate(folder / "comparison")

Data without an experiment

An evaluation can also take a dataset and an operator directly, e.g. for data you load yourself:

evaluation = Evaluation(dataset=my_test_data, operator=my_operator)