Interacting with compute clusters
Asimov is designed to form a layer between you, a data analyst, and the computing cluster which performs your analysis.
An important part of its job is therefore interacting with that cluster, including submitting jobs to it, checking on how they’re running, and collecting results once jobs have finished.
Currently Asimov is designed to use the htcondor scheduling system, though at some point in the future we may implement additional systems to improve the overall flexibility of Asimov.
HTCondor, which we’ll sometimes call condor for short, can be run both on a full-scale computing cluster and on a single machine (their minicondor distribution is ideal for this) and allows jobs to be queued, and run in the correct order to ensure that required inputs are produced for each stage of an analysis.
Interacting with the scheduler
Asimov handles all interactions which are required with the htcondor scheduler.
If you’re using the command line interface to Asimov then the olivaw manage submit and olivaw monitor commands will initiate interaction between Asimov and the cluster.
The olivaw manage submit command will submit a job to the cluster, and each pipeline which Asimov supports will provide sufficient logic to set a job up on the cluster based on the configuration files created by running olivaw manage build.
The olivaw monitor command will check the status of a job in a cluster, and update the ledger as appropriate, to indicate if a job has completed or become stuck.
In the case of a completed job the pipeline’s post-completion hook will be run, allowing post-processing to be started for a finished job.
Caching
In order to improve the performance of Asimov’s interactions with clusters, and to reduce the strain placed on the schedulers’ databases by default asimov will cache job information for 15 minutes.
This can be adjusted in the main configuration file for asimov.
Configuration settings
Configuration options for HTCondor are stored in the htcondor namespace of the main configuration file for asimov.
cache_time
[htcondor]
cache_time = 900
The time for which job status information should be cached, in seconds.
The default setting is 900 seconds (15 minutes).
Please take care when reducing this setting, as excessively frequent querying of busy schedulers can result in reduced performance.