Running TXPipe at NERSC
On NERSC, follow the setup command described in NERSC Shifter Installation to set up the TXPipe dependencies.
NERSC Interactive Sessions
You should then only run small test commands on the login node, with only a small number of cores. To run larger test jobs you can start an interactive session. This example would give you one node for 1 hour:
salloc --nodes 1 --qos interactive --time 01:00:00 --constraint haswell
Run the setup command again after starting your job.
Running pipelines at NERSC
The tx
command runs programs in a TXPipe environment. To run whole pipelines, you can do this:
tx ceci examples/metacal/pipeline.yml
Running individual stages at NERSC
You can also use the TX command to run individual stages, to test or develop them.
The easiest way to find the correct command to run is to run a pipeline with ceci (as above)
using the --dry-run
flag to print out a list of commands.
They will normally have a form something like this at NERSC:
srun -u --ntasks=4 --cpus-per-task=2 --nodes=1 tx python -m txpipe TXSourceSelector ...