🎙️ Discussion

Container Metadata

What are the challenges that we are trying to solve?

Containers are not able to provide enough metadata about binaries / libraries inside to assess compatibility with a host.
Although there is a limited ability to parse high level container metadata, details about the contents of containers does not come in any standard format that can be easily parsed. HPC developers often must parse build files, scripts, image layers, etc. (when available) to determine the contents.
Configuration of software and environment variables can affect reproducibility but are rarely captured in metadata.

Containers can seem like black boxes. But for running a container on HPC, we need to know about:

attributes of the applications and binaries that are contained inside
what requirements might be for running (e.g. hardware, memory, disk space)
potentially other information that could be used by a resource manager or scheduler to know if it can run a container using this image.

TODO add ideas from papers below?

Existing technologies that address parts of these ideas (incompletely):

Docker provides Object labels that are optional for images, containers, volumes, and more (such as the LABEL command within the Dockerfile).
Singularity definition files provide a few relevant sections, such as label and help. More Singularity metadata discussion here.

The following works are relevant: