LION.package

Publishing a trained model: everything needed to make it, train it again and test it, in one folder (a package), from the records LION saved:

from LION.package import publish
publish("results/lpd/final", "lpd_package", evaluation="results/evaluations/lpd-1234.json")

publish(model_file, folder, evaluation=None): model_file is a model a LION solver saved ("final", .json or .pt), or a reconstructor’s file (reconstructor.save: its model with it); evaluation is optional: its test record (Evaluation.run), the record’s file, or an Evaluation, which is then run on the model. The package:

model.json, model.pt   the model file, exactly as LION saved it: LIONmodel.load("model")
<name>_initial.pt      the weights training started from, when the solver saved them
experiment.json        the experiment: geometry, noise, dataset (Experiment.instantiate)
data/training.json     the training data's record: split, samples, noise seed
data/validation.json   the validation data's record, if it was validated
solver.json            the training record: solver, optimizer, seed, settings, history
test.json              the test record, if an evaluation was given
reconstructor.json     the reconstructor, if one is published (its model: model)
origin/model.json      for a fine-tuned model, its original's metadata (and
                       origin/origin/model.json, ... for the original's original)
environment.txt        the packages installed where it was published (conda's and Python's)
package.json           the contents and their sha256, the citations, LION's version and commit, notes
README.md              a model card, generated from the records (edit it: it isn't checksummed)

Each file is made again by its own instantiate (LIONmodel, Experiment, LIONDataset, LIONsolver, Evaluation). A package is used whole with:

package = load(“lpd_package”) # the recorded arguments, to change; nothing made package.model.model_parameters.n_iters = 5 package.experiment.experiment_params.noise_params.I0 = 5000 built = build(package) # the model, experiment, data, solver, evaluation built = load_and_build(“lpd_package”) # as published model = train_again(package, “retrained”) # trained again, changed or not

load warns about files changed since publishing and LION code other than the model’s. Things run where the model was trained unless device= says otherwise. The model card shows the same, and the code step by step.

publish refuses, with an error and before writing anything:

• LION code that isn’t committed and pushed (environment.publish_problems): the code running now, the code that saved the model and the code that tested it;

• a model file without a training record or a data record (one not saved by a LION solver from a LION dataset);

• what training again would miss: initial weights that weren’t saved, an unseeded training (seed=None), data not made by an Experiment, data outside LION_DATA_PATH (another machine can’t find them);

• records naming code outside LION (other than torch’s layers, losses and optimizers): a package is made again with LION alone, without allow=;

• records that could only name something (e.g. a lambda loss), which can’t be made again;

• a test record of other weights than the model’s, or test data the model was trained or validated on;

• for a fine-tuned model, an original (and its original, …) that is missing, has other weights than training started from, or would be refused itself;

• a folder that isn’t empty.

What a reader should know of a complete package (it was published in another environment than it was trained in, or tested on another problem) is a note, in package.json and the model card, and a warning. citations are each component’s cite(): where it comes from, not the work being published.

Functions

build(package, *[, device, initial_weights])	The components a package (load) describes, made, changed or not: model, experiment, training_data, validation_data, solver (with model, its training and validation data set; not its saving), evaluation (None without a test), reconstructor (with the model; None without one) and device.
citations(saved[, test, reconstructor])	Where the model and its components come from: each one's cite() (BibTeX), once each, as {"of": ..., "class": ..., "bibtex": ...}.
environment_list()	The packages installed where LION runs: conda's (name=version=build, as conda list --export writes them) when it is a conda environment, and Python's (name==version).
file_sha256(path)	The SHA-256 of the file at path, as a hex string (read in 1 MB blocks).
load(folder)	A package's components as their recorded arguments, to change before build makes them (nothing is made here):
load_and_build(folder, *[, device])	build(load(folder)): the package's components, as published.
model_card(saved, package, model[, test, ...])	The README of a package: what it is, from its records.
publish(model_file, folder[, evaluation])	Write the package of model_file in folder (see the module docstring); returns the folder.
train_again(package, save_folder[, name, device])	Train the model of package (a load-ed package, changed or not, or its folder) again, from its initial weights (new ones if its arguments were changed), for its epochs; saved as save_folder/name, as a solver saves (name and, if validated, its best validation).

---

LION.package.build(package, *, device=None, initial_weights=False)

The components a package (load) describes, made, changed or not: model, experiment, training_data, validation_data, solver (with model, its training and validation data set; not its saving), evaluation (None without a test), reconstructor (with the model; None without one) and device.

The model has its trained weights, or, with initial_weights=True, the weights its training started from; a model whose arguments were changed has new, random ones (it is to be trained). device: where the solver and the evaluation run; by default where it was trained (the CPU if that was a GPU and there is none).

LION.package.citations(saved, test=None, reconstructor=None)

Where the model and its components come from: each one’s cite() (BibTeX), once each, as {"of": ..., "class": ..., "bibtex": ...}.

LION.package.environment_list()

The packages installed where LION runs: conda’s (name=version=build, as conda list --export writes them) when it is a conda environment, and Python’s (name==version). Without paths.

LION.package.file_sha256(path)

The SHA-256 of the file at path, as a hex string (read in 1 MB blocks).

LION.package.load(folder)

A package’s components as their recorded arguments, to change before build makes them (nothing is made here):

• model: the model’s arguments (model_parameters, and e.g. operator_params);

• experiment: the experiment’s (experiment_params, dataset);

• training_data, validation_data: which data: split and subset (indices, or None); validation_data is None if it wasn’t validated;

• solver: the solver’s arguments (loss_fn, seed, solver_params, …), and optimizer: its hyperparameters (lr); parameter_groups: its parameter groups (each one’s parameters, by name, and settings), or None for one group with those hyperparameters; scheduler: the learning-rate scheduler’s arguments, or None;

• training: batch_size, shuffle, drop_last and epochs (those done);

• validation: batch_size, every, function and reconstructor (its arguments, or None); None if not validated;

• evaluation: the test’s evaluation’s arguments; None without a test;

• reconstructor: the reconstructor’s arguments (operator, parameters); None without one;

• device: where it was trained; folder; records: the training, data, test and reconstructor records, the origins of a fine-tuned model, and package.json, as read.

The classes are the recorded ones (another class’s parameters come from its parameters_from_file). Files changed since the package was published, and LION code other than the model was saved with, give a warning.

LION.package.load_and_build(folder, *, device=None)

build(load(folder)): the package’s components, as published.

LION.package.model_card(saved, package, model, test=None, reconstructor=None, origins=())

The README of a package: what it is, from its records.

LION.package.publish(model_file, folder, evaluation=None)

Write the package of model_file in folder (see the module docstring); returns the folder. model_file can be a reconstructor’s file (reconstructor.save): its model is published, with the reconstructor.

LION.package.train_again(package, save_folder, name='model', *, device=None)

Train the model of package (a load-ed package, changed or not, or its folder) again, from its initial weights (new ones if its arguments were changed), for its epochs; saved as save_folder/name, as a solver saves (name and, if validated, its best validation). Returns the trained model. device: as in build.