[Help-bash] Dealing with "Broken pipe: 13" error

[Peng Yu]

Hi,

When I run the following code, it will generate the "Broken pipe: 13"
error. And the for-loop hangs there at the iteration 1859. How to make
the awk process persistent so that `read` still can get new data
beyond iteration 1859?

$ cat ./main.sh
#!/usr/bin/env bash
# vim: set noexpandtab tabstop=2:

set -v
myfifo=$(mktemp -u)
guard=$(mktemp -u)
mkfifo "$myfifo" "$guard"

"$myfifo" < "$guard" &

awk -e 'BEGIN { for(i=1;;++i) print i }' > "$myfifo" &
jobs
sleep 5
for i in {1..10000}
do
read -r x < "$myfifo"
echo "$x"
jobs
done
$ ./main.sh
myfifo=$(mktemp -u)
guard=$(mktemp -u)
mkfifo "$myfifo" "$guard"

"$myfifo" < "$guard" &

awk -e 'BEGIN { for(i=1;;++i) print i }' > "$myfifo" &
jobs
[1]- Running > "$myfifo" < "$guard" &
[2]+ Running awk -e 'BEGIN { for(i=1;;++i) print i }'

"$myfifo" &

sleep 5
for i in {1..10000}
do
read -r x < "$myfifo"
echo "$x"
jobs
done
1
[1]- Running > "$myfifo" < "$guard" &
[2]+ Running awk -e 'BEGIN { for(i=1;;++i) print i }'

"$myfifo" &

2
[1]- Running > "$myfifo" < "$guard" &
[2]+ Running awk -e 'BEGIN { for(i=1;;++i) print i }'

"$myfifo" &

3
[1]- Running > "$myfifo" < "$guard" &
[2]+ Running awk -e 'BEGIN { for(i=1;;++i) print i }'

"$myfifo" &

4
[1]- Running > "$myfifo" < "$guard" &
[2]+ Broken pipe: 13 awk -e 'BEGIN { for(i=1;;++i) print i }'

"$myfifo"

5
[1]+ Running > "$myfifo" < "$guard" &
6
[1]+ Running > "$myfifo" < "$guard" &
...
1858
[1]+ Running > "$myfifo" < "$guard" &
1859
[1]+ Running > "$myfifo" < "$guard" &

[Pierre Gaston]

Post by Peng Yu
Hi,
When I run the following code, it will generate the "Broken pipe: 13"
error. And the for-loop hangs there at the iteration 1859. How to make
the awk process persistent so that `read` still can get new data
beyond iteration 1859?
$ cat ./main.sh
#!/usr/bin/env bash
set -v
myfifo=$(mktemp -u)
guard=$(mktemp -u)
mkfifo "$myfifo" "$guard"

"$myfifo" < "$guard" &

awk -e 'BEGIN { for(i=1;;++i) print i }' > "$myfifo" &
jobs
sleep 5
for i in {1..10000}
do
read -r x < "$myfifo"
echo "$x"
jobs
done

You are using a guarding process to keep a file descriptor open, but you
keep the writing end opened.
this end doesn't need to be guarded because awk will keep it open anyway.

On the other hand the reading end is not kept opened, read is closing it.
So awk is buffering all it can, then when the first read happens it writes
a full buffer to the pipe, then read close the pipe
and when awk tries to write again it dies with broken pipe.
The guarding process doesn't die because it never writes to the pipe

The loop continues to repoen and read the remaining of the data from the
pipe
(it can do that because you keep the pipe open for writing on the other
end) and block when there is no more data.

opening the pipe the other way round in the guard process fixes your
problem

#!/usr/bin/env bash
# vim: set noexpandtab tabstop=2:

set -v
myfifo=$(mktemp -u)
guard=$(mktemp -u)
mkfifo "$myfifo" "$guard"
< "$myfifo" < "$guard" &
p=$!
awk 'BEGIN { for(i=1;;++i) print i }' > "$myfifo" &

for i in {1..10000}
do
read -r x < "$myfifo"
echo "$x"
done
kill $!
rm "$myfifo" "$guard"

PS: in this case i would rather use another fd, it avoids a process,
reopening the pipe each time
and if you use read -u it probably even avoids calling dup()

#!/usr/bin/env bash
# vim: set noexpandtab tabstop=2:

set -v
myfifo=$(mktemp -u)
guard=$(mktemp -u)
mkfifo "$myfifo" "$guard"
awk 'BEGIN { for(i=1;;++i) print i }' > "$myfifo" &
exec 3< "$myfifo"
for i in {1..10000}
do
read -r -u 3 x
echo "$x"
done
rm "$myfifo" "$guard"