LION.data_loaders.data_record

The record of the data a model is trained or tested on.

data_record(dataset) describes a LION dataset so it can be made again and checked, from the record alone:

• dataset: its spec (class, mode, geometry, parameters; LION.utils.spec);

• sinogram_transform and its seed: how the measurements are simulated, e.g. functools.partial(sinogram_add_noise, I0=...). A function that can’t be written as a spec (a lambda) is recorded by name only, with a warning;

• experiment and split: for a dataset made by an experiment, the experiment’s spec and which split this is;

• samples: what each sample is (e.g. "LIDC-IDRI-0912/slice_37"), in order, their number and a hash of the list; for datasets whose samples come in groups (sample_groups(): LIDC-IDRI’s patients, 2DeteCT’s physical samples), the groups they are from, sorted. Evaluations check by group that no test patient was trained on;

• subset: for a torch.utils.data.Subset, its indices into the dataset it is taken from (whose record this is); samples lists the subset’s samples;

• files (hash_files=True): a hash of the contents of the files the samples are read from, to check that two copies of the data are the same. It reads the data, so it is only computed when asked.

A record is a LIONParameter (record.samples.ids, record.split) in which the specs (record.dataset, record.experiment) stay dicts: they describe objects, which are only made on request. It is saved with a model as plain JSON (model.save(f, dataset=data_record(dataset))) and read back as a LIONParameter. The data are made again from the record or the model’s file:

• LIONDataset.instantiate(source): the dataset alone, without noise (the noise is part of the experiment), checked against the record’s samples;

• Experiment.instantiate(source).get_dataset(record.split): the data as the experiment made them, noise included.

Functions

data_record(dataset[, hash_files])	The record of a LION dataset (see the module docstring).
file_contents_hash(paths)	A hash of the contents of the files, in order (not of their names or locations, which differ between machines).
is_data_record(value)	A data record, as a LIONParameter or as the dict read from JSON.
read_record(source)	The data record in source: a record, a model's metadata holding one (as its dataset_params), or the path of a .json with either.
recorded_spec(record, key)	The spec of record.<key> ("dataset", "experiment"), or an error if the record could only name it.

---

LION.data_loaders.data_record.data_record(dataset, hash_files=False)

The record of a LION dataset (see the module docstring).

LION.data_loaders.data_record.file_contents_hash(paths)

A hash of the contents of the files, in order (not of their names or locations, which differ between machines).

LION.data_loaders.data_record.is_data_record(value)

A data record, as a LIONParameter or as the dict read from JSON.

LION.data_loaders.data_record.read_record(source)

The data record in source: a record, a model’s metadata holding one (as its dataset_params), or the path of a .json with either.

LION.data_loaders.data_record.recorded_spec(record, key)

The spec of record.<key> (“dataset”, “experiment”), or an error if the record could only name it.