Post by Val KremI am trying to develop a pipeline procedure in SLURM
(https://hpc.nih.gov/docs/job_dependencies.html).
I have two jobs job1.sh and job2.sh. Job2.sh deepened on job1.sh and
give job dependency syntax when I am submitting job2.sh
Post by Val KremIn SLURM, when I submit the job the syntax is as follow
JJ1=$(sbatch job1.sh)
This is not a background process. This is a FOREGROUND process. The
sbatch command runs in the foreground, and the command substitution and
variable assignment (and therefore the entire bash script) are "paused",
waiting until the sbatch command's standard output is closed.
Once sbatch closes stdout (usually by terminating), the entire contents
that were written to stdout are grabbed, trailing newlines are trimmed,
NUL bytes are removed (with or without a warning, depending on the version
of bash), and then the result is stored in the JJ1 variable.
Calling this a "job" is incredibly misleading. The use of the English
word "job" in a shell programming context implies something running
in the background.
Post by Val KremJJ1 will give the job id or PID (numeric value) for the job1.sh.
echo ${JJ1} will produce number like this 11254323
Wrong wrong wrong wrong wrong.
...
Unless you're claiming that your "sbatch" command is self-backgrounding.
If "sbatch" is an evil self-backgrounding abomination, which forks and
abandons a child process, and then writes the child's process ID to
stdout, then... what you say may be literally true, but I would run
away, screaming. This is a horror show. It's the worst mistakes of
the 1980s, risen anew.
Post by Val Kremwhen I submit job2.sh
JJ2=$(sbatch --dependency=afterok:${JJ1} job2.sh)
In this case Job2.sh runs only if job1.sh finishes.
For single job submission it is ok.
Oh dear god, no.
I'm out. Sorry. I can't even ... no. Just no.
You would be better off throwing this entire infrastructure away, and
building your own from scratch.
Did you know that you can create dependencies among systemd units?
Maybe that would be a better framework for doing this thing of yours.
Set up, say, one systemd service that provides the things your second
service needs. Let's call this first service "dirt", because that is a
BETTER and MORE USEFUL name than "job1.sh". Let's call the other service
"grass", because grass depends on dirt. This is a BETTER and MORE USEFUL
name than "job2.sh".
So, then you ask yourself: "What does dirt do?" Well, we don't
know, because you haven't SAID a single useful thing ANYWHERE. All
we know is that you need it to run. To be running? To have executed
at least one time since boot? Let's assume it has to be continuously
running, like a database engine or something.
To do that, you would figure out whatever command you have to run to
make the dirt service "go". To be up and running, listening on its
unix domain socket, or whatever it does, so that the grass service can
use it.
IF AT ALL POSSIBLE, the command you write to invoke the dirt service
should be a FOREGROUND process. It should not self-background. It should
not fork and abandon a child, like some 1980s BSD abomination. Then you
set up a systemd service that will run this foreground process, as the
correct user, with the correct environment variables, in the correct
directory, etc.
I won't try to show this here.
Then you set up the grass service, and tell systemd that it depdends
on dirt. If you also run grass as a foreground process, then it may
be something as simple as this:
==== example /etc/systemd/system/grass.service
[Unit]
Description=Grass service, grows in the dirt
Requires=dirt.service
After=dirt.service
[Service]
ExecStart=/usr/local/bin/grass ...
KillMode=process
Restart=on-failure
[Install]
WantedBy=multi-user.target
====
I'm not saying systemd is great, and I'm not yet ready to attempt to
write real documentation for systemd (although god knows it NEEDS it,
BADLY). But it sounds like it could be a match for your poorly defined,
obfuscated task.
This is quite a bit outside the scope of "help-bash". Setting up system
services with dependencies on each other is a system administration task,
not a bash programming task. Bash isn't even INVOLVED.
If your target system doesn't use systemd, then... well. Go to your
operating system's mailing list, explain what you are actually trying to
do, with *real names*, and actually explain what each of your processes
*does*, and the actual nature of their relationship, and maybe someone
can come up with a solution.
My curiosity got the better of me so I took a quick glance at the
documentation of the system the OP is using. It seems to be an hpc cluster.
It's likely that the sbatch program submits what they refer to as "jobs" to
run on a different system altogether and returns immediately. The output
might be a pid on that system or perhaps some sort of task ID. So there's
no issue with backgrounding on the local system.
Words have contextual meaning, but it's the responsibility of a good
communicator to set up the context. The OP failed in that regard.