[Help-bash] Array Expansion

[J.B.]

Expanding an array for use in a for-loop behaves differently depending
on how the array was created. How can I get bash to expand using a
different delimiter? I tried `find -print0' after setting IFS= but bash
just ignores it (and says so). I then tried `find -exec ls
--quoting-style=shell-escape-always', but that didn't work either.

$ !find
find ./path/ -type f -name \*.txt
./path/filename with spaces.txt
./path/good_filename.txt

$ listoffiles=($(find ./path/ -type f -exec ls
--quoting-style=shell-escape-always '{}' \;))


$ echo ${listoffiles[@]}
'./path/filename with spaces.txt' './path/good_filename.txt'

$ for file in ${listoffiles[@]}; do echo $file; done
'./path/filename
with
spaces.txt'
'./path/good_filename.txt'

$ for file in "${listoffiles[@]}"; do echo $file; done
'./path/filename
with
spaces.txt'
'./path/good_filename.txt'

$ for file in "${listoffiles[*]}"; do echo $file; done
'./path/filename with spaces.txt' './path/good_filename.txt'

$ !unse
unset -v listoffiles

$ listoffiles=(

'./path/filename with spaces.txt'
'./path/good_filename.txt'
)

$ for file in ${listoffiles[@]}; do echo $file; done
./path/filename
with
spaces.txt
./path/good_filename.txt

$ for file in "${listoffiles[@]}"; do echo $file; done
./path/filename with spaces.txt
./path/good_filename.txt

[Andy Chu]

Two ideas:

1) Use something like:

find -print0 | while read -d '' line; do echo $line; done

read -d '' splits on NUL terminators (this was discussed a few months ago).

'shopt -s lastpipe' or using a subshell might be useful if you want the
while loop state to persist after the pipeline.

If you really need to populate an array rather than just doing things in
the loop body, you'll need one of those.

Example of subshell:

find -print0 | (i=0; while read -d '' line; do echo $line; ((i++)); done;
echo $i )

Hm I can't get lastpipe to work for 'while loop', maybe it only works for
simple builtins like 'read' ?

$ shopt -s lastpipe; i=0; find -print0 | while read -d '' line; do echo
$line; ((i++)); done; echo
$i;
0 # was expecting non-zero


2) If your filenames contain spaces, but not newlines, set IFS=$'\n', and
then $(find ...) should split the paths correctly.


Andy

Expanding an array for use in a for-loop behaves differently depending on
how the array was created. How can I get bash to expand using a different
delimiter? I tried `find -print0' after setting IFS= but bash just ignores
it (and says so). I then tried `find -exec ls --quoting-style=shell-escape-always',
but that didn't work either.
$ !find
find ./path/ -type f -name \*.txt
./path/filename with spaces.txt
./path/good_filename.txt
$ listoffiles=($(find ./path/ -type f -exec ls
--quoting-style=shell-escape-always '{}' \;))
'./path/filename with spaces.txt' './path/good_filename.txt'
'./path/filename
with
spaces.txt'
'./path/good_filename.txt'
'./path/filename
with
spaces.txt'
'./path/good_filename.txt'
$ for file in "${listoffiles[*]}"; do echo $file; done
'./path/filename with spaces.txt' './path/good_filename.txt'
$ !unse
unset -v listoffiles
$ listoffiles=(

Post by J.B.
'./path/filename with spaces.txt'
'./path/good_filename.txt'
)

./path/filename
with
spaces.txt
./path/good_filename.txt
./path/filename with spaces.txt
./path/good_filename.txt

[João Eiras]

Expanding an array for use in a for-loop behaves differently depending on
how the array was created. How can I get bash to expand using a different
delimiter? I tried `find -print0' after setting IFS= but bash just ignores

Your array is not being expanded differently in the for-loop. You're
creating it incorrectly.

What's happening ?
First bash evaluates the command $() and substitutes that expression
with the value from stdout.
Secondly, it breaks apart that string into tokens given the values in IFS.
Thirdly it assigns each token to a different position in the array.

This is most likely what you want:

IFS=$'\x0a'
listoffiles=( $(find ...) )

Or if you're picky about wanting not to treat the newline as a
separator, this helps you further
https://stackoverflow.com/questions/1116992/capturing-output-of-find-print0-into-a-bash-array

find -print0 | (i=0; while read -d '' line; do echo $line; ((i++)); done;
echo $i )

You're reading the values into a sub-shell, which are not inherited by
the main shell.

[J.B.]

Post by João Eiras

Expanding an array for use in a for-loop behaves differently depending on
how the array was created. How can I get bash to expand using a different
delimiter? I tried `find -print0' after setting IFS= but bash just ignores

Your array is not being expanded differently in the for-loop. You're
creating it incorrectly.
What's happening ?
First bash evaluates the command $() and substitutes that expression
with the value from stdout.
Secondly, it breaks apart that string into tokens given the values in IFS.
Thirdly it assigns each token to a different position in the array.
IFS=$'\x0a'
listoffiles=( $(find ...) )
Or if you're picky about wanting not to treat the newline as a
separator, this helps you further
https://stackoverflow.com/questions/1116992/capturing-output-of-find-print0-into-a-bash-array

find -print0 | (i=0; while read -d '' line; do echo $line; ((i++)); done;
echo $i )

You're reading the values into a sub-shell, which are not inherited by
the main shell.

Thanks. Setting IFS to $'\x0a' fixed the way my script handles filenames
with spaces.

[Andy Chu]

Post by João Eiras
IFS=$'\x0a'
listoffiles=( $(find ...) )

This seems like an obscure way to write $'\n' :) (my second suggestion)

Post by João Eiras

Post by Andy Chu
find -print0 | (i=0; while read -d '' line; do echo $line; ((i++)); done;
echo $i )

You're reading the values into a sub-shell, which are not inherited by
the main shell.

That's true, but I think you might not have gotten the point of the
snippet. The point is that you can use $i after the while loop, but
without the subshell you can't. The subshell is part of the pipeline.

That might not be convenient for all scripts, but it's sufficient for many
(especially in the absense of lastpipe).

Andy

[J.B.]

Post by Andy Chu
find -print0 | while read -d '' line; do echo $line; done
read -d '' splits on NUL terminators (this was discussed a few months ago).
'shopt -s lastpipe' or using a subshell might be useful if you want
the while loop state to persist after the pipeline.
If you really need to populate an array rather than just doing things
in the loop body, you'll need one of those.
find -print0 | (i=0; while read -d '' line; do echo $line; ((i++));
done; echo $i )
Hm I can't get lastpipe to work for 'while loop', maybe it only works
for simple builtins like 'read' ?
$ shopt -s lastpipe; i=0; find -print0 | while read -d '' line; do
echo $line; ((i++)); done; echo
$i;
0 # was expecting non-zero
2) If your filenames contain spaces, but not newlines, set IFS=$'\n',
and then $(find ...) should split the paths correctly.
Andy
Expanding an array for use in a for-loop behaves differently
depending on how the array was created. How can I get bash to
expand using a different delimiter? I tried `find -print0' after
setting IFS= but bash just ignores it (and says so). I then tried
`find -exec ls --quoting-style=shell-escape-always', but that
didn't work either.
$ !find
find ./path/ -type f -name \*.txt
./path/filename with spaces.txt
./path/good_filename.txt
$ listoffiles=($(find ./path/ -type f -exec ls
--quoting-style=shell-escape-always '{}' \;))
'./path/filename with spaces.txt' './path/good_filename.txt'
'./path/filename
with
spaces.txt'
'./path/good_filename.txt'
'./path/filename
with
spaces.txt'
'./path/good_filename.txt'
$ for file in "${listoffiles[*]}"; do echo $file; done
'./path/filename with spaces.txt' './path/good_filename.txt'
$ !unse
unset -v listoffiles
$ listoffiles=(

Post by J.B.
'./path/filename with spaces.txt'
'./path/good_filename.txt'
)

./path/filename
with
spaces.txt
./path/good_filename.txt
./path/filename with spaces.txt
./path/good_filename.txt

Thanks for your reply, Andy. I noticed `shopt -s lastpipe' doesn't seem
to work, too, in a previous script I wrote.

[Greg Wooledge]

Expanding an array for use in a for-loop behaves differently depending on
how the array was created.

No, it doesn't. You are simply not creating the array correctly.

There are only two ways to iterate over an array in bash: by value, or
by index.

$ listoffiles=($(find ./path/ -type f -exec ls
--quoting-style=shell-escape-always '{}' \;))

This is wrong. You are word-splitting the output of $(find) which is
never correct.

IFS=$'\x0a'
listoffiles=( $(find ...) )

No, this also fails when filenames contain newlines.

Anything that relies on word-splitting is fragile.


There are a few correct ways to read output from find into a bash array.
Here are some of them:

mapfile -t -d '' files < <(find ... -print0) # requires bash 4.4

files=(); while IFS= read -r -d '' f; do
files+=("$f")
done < <(find ... -print0)

files=() i=0; while IFS= read -r -d '' f; do
files[i++]="$f"
done < <(find ... -print0)

[J.B.]

Post by Greg Wooledge

Expanding an array for use in a for-loop behaves differently depending on
how the array was created.

No, it doesn't. You are simply not creating the array correctly.
There are only two ways to iterate over an array in bash: by value, or
by index.

$ listoffiles=($(find ./path/ -type f -exec ls
--quoting-style=shell-escape-always '{}' \;))

This is wrong. You are word-splitting the output of $(find) which is
never correct.

IFS=$'\x0a'
listoffiles=( $(find ...) )

No, this also fails when filenames contain newlines.
Anything that relies on word-splitting is fragile.
There are a few correct ways to read output from find into a bash array.
mapfile -t -d '' files < <(find ... -print0) # requires bash 4.4
files=(); while IFS= read -r -d '' f; do
files+=("$f")
done < <(find ... -print0)
files=() i=0; while IFS= read -r -d '' f; do
files[i++]="$f"
done < <(find ... -print0)

I see. Can you rewrite bash to work the way I'm using it in my script?
Also, how long will it take you?

Seriously though, thanks for your bash wiki. I learned a lot from it and
still use it as a reference guide. So, thanks to everyone that's
contributed to it.

[Peter West]

Thank you for this; and seconded.

I see. Can you rewrite bash to work the way I'm using it in my script? Also, how long will it take you?
Seriously though, thanks for your bash wiki. I learned a lot from it and still use it as a reference guide. So, thanks to everyone that's contributed to it.

--
Peter West
***@pbw.id.au
“Come to me, all who labor and are heavy laden, and I will give you rest.”