Discussion:
bash variable interpolation
(too old to reply)
Peng Yu
2015-03-19 22:04:04 UTC
Permalink
Hi,

I want to interpolate variable in a bash string variable. But the
following code shows the spaces are not maintained. There is some
built-in features in perl to do string interpolation. Does anyone know
the best way to do string interpolation in bash? Thanks.

~$ cat main.sh
#!/usr/bin/env bash

x=ABC
y=IJK
z=XYZ
str='$x $y $z'
eval echo "$str"
~$ ./main.sh
ABC IJK XYZ
--
Regards,
Peng
Eduardo A. Bustamante López
2015-03-19 22:09:12 UTC
Permalink
On Thu, Mar 19, 2015 at 05:04:04PM -0500, Peng Yu wrote:
[...]
Post by Peng Yu
I want to interpolate variable in a bash string variable. But the
following code shows the spaces are not maintained. There is some
built-in features in perl to do string interpolation. Does anyone know
the best way to do string interpolation in bash? Thanks.
You can do that with printf:

***@hp:~$ x=ABC y=IJK z=XYX; printf -v str '%s %s %s' "$x" "$y" "$z"; echo "$str"
ABC IJK XYX
Chet Ramey
2015-03-19 22:19:40 UTC
Permalink
Post by Peng Yu
Hi,
I want to interpolate variable in a bash string variable. But the
following code shows the spaces are not maintained. There is some
built-in features in perl to do string interpolation. Does anyone know
the best way to do string interpolation in bash? Thanks.
This seems much more complex than necessary. You're using eval, which is
going to require a second set of escaped quotes to account for the second
set of word expansions.

What's wrong with something like

str="$x $y $z"

and then using

echo "$str"?

Do you require the delayed evaluation?
--
``The lyf so short, the craft so long to lerne.'' - Chaucer
``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, ITS, CWRU ***@case.edu http://cnswww.cns.cwru.edu/~chet/
Eric Blake
2015-03-19 22:42:29 UTC
Permalink
Post by Peng Yu
Hi,
I want to interpolate variable in a bash string variable. But the
following code shows the spaces are not maintained.
Insufficient quoting on your part.
Post by Peng Yu
There is some
built-in features in perl to do string interpolation. Does anyone know
the best way to do string interpolation in bash? Thanks.
eval does string interpolation. But it is a very heavy hammer, and
should be avoided if there is any other way to do what you really need
done, because it is so easily misused (in particular, use of eval on
unvetted user-supplied is a gaping security hole).
Post by Peng Yu
~$ cat main.sh
#!/usr/bin/env bash
x=ABC
y=IJK
z=XYZ
str='$x $y $z'
eval echo "$str"
Remember, eval basically causes a second pass of quote removal. You
executed the command:

"eval" "echo" "$str"

which is the same as:

"eval" "echo" "$x $y $z"

then after quote removal, it gets interpolated as if you had written:

echo $x $y $z

at which point the extra spaces are eaten. You WANT the interpolated
string to have proper quoting, as if you had written:

echo "$x $y $z"

which means your original eval has to add one more layer of quoting
beyond that:

eval echo \""$str"\"

That way, $str is expanded as part of the command line to be eval'd, but
there are still double quotes available for the string that eval is
parsing to preserve the spacing.

And once you understand that, you will see why eval should be avoided if
there is any other way to accomplish what you really need, because it is
far too easy to get quoting wrong when you have to think about multiple
layers of quoting being stripped, not to mention user-controlled input
cause security breaches by executing arbitrary code when you fail to
sanitize the input string.
--
Eric Blake eblake redhat com +1-919-301-3266
Libvirt virtualization library http://libvirt.org
Peng Yu
2015-03-19 23:37:30 UTC
Permalink
Post by Eric Blake
Post by Peng Yu
Hi,
I want to interpolate variable in a bash string variable. But the
following code shows the spaces are not maintained.
Insufficient quoting on your part.
Post by Peng Yu
There is some
built-in features in perl to do string interpolation. Does anyone know
the best way to do string interpolation in bash? Thanks.
eval does string interpolation. But it is a very heavy hammer, and
should be avoided if there is any other way to do what you really need
done, because it is so easily misused (in particular, use of eval on
unvetted user-supplied is a gaping security hole).
Post by Peng Yu
~$ cat main.sh
#!/usr/bin/env bash
x=ABC
y=IJK
z=XYZ
str='$x $y $z'
eval echo "$str"
Remember, eval basically causes a second pass of quote removal. You
"eval" "echo" "$str"
"eval" "echo" "$x $y $z"
echo $x $y $z
at which point the extra spaces are eaten. You WANT the interpolated
echo "$x $y $z"
which means your original eval has to add one more layer of quoting
eval echo \""$str"\"
The following example shows that above code is not robust with
respective to arbitrary 'str'.

#!/usr/bin/env bash

x="'"
y=IJK
z=XYZ
str='"$x" $y $z'
eval echo \""$str"\"
Post by Eric Blake
That way, $str is expanded as part of the command line to be eval'd, but
there are still double quotes available for the string that eval is
parsing to preserve the spacing.
And once you understand that, you will see why eval should be avoided if
there is any other way to accomplish what you really need, because it is
far too easy to get quoting wrong when you have to think about multiple
layers of quoting being stripped, not to mention user-controlled input
cause security breaches by executing arbitrary code when you fail to
sanitize the input string.
The real problem is that I want to replace some bash variables in a
file and then print the output.

sed has the problem of not robust with respect to special characters.
Please let me know if there is a convenient solution in bash.

~$ cat main_from_file.sh
#!/usr/bin/env bash

x="'"
y=IJK
z=XYZ
str=$(cat myfile.txt)
eval echo "$str"
~$ cat myfile.txt
"$x" $y $z
--
Regards,
Peng
Dave Rutherford
2015-03-20 00:03:01 UTC
Permalink
Post by Peng Yu
Post by Eric Blake
Post by Peng Yu
I want to interpolate variable in a bash string variable. But the
following code shows the spaces are not maintained.
Insufficient quoting on your part.
Post by Peng Yu
There is some
built-in features in perl to do string interpolation. Does anyone know
the best way to do string interpolation in bash? Thanks.
I feel dumb. What is meant by `string interpolation'?
Post by Peng Yu
The real problem is that I want to replace some bash variables in a
file and then print the output.
Can you arrange things so that the file is given as a here-document?
Here-documents are treated to variable expansion, and the results
ought to be safe regardless of the variable contents.
Peng Yu
2015-03-20 00:06:56 UTC
Permalink
On Thu, Mar 19, 2015 at 7:03 PM, Dave Rutherford
Post by Dave Rutherford
Post by Peng Yu
Post by Eric Blake
Post by Peng Yu
I want to interpolate variable in a bash string variable. But the
following code shows the spaces are not maintained.
Insufficient quoting on your part.
Post by Peng Yu
There is some
built-in features in perl to do string interpolation. Does anyone know
the best way to do string interpolation in bash? Thanks.
I feel dumb. What is meant by `string interpolation'?
Post by Peng Yu
The real problem is that I want to replace some bash variables in a
file and then print the output.
Can you arrange things so that the file is given as a here-document?
Here-documents are treated to variable expansion, and the results
ought to be safe regardless of the variable contents.
Heredoc is good when the doc is short. But when the doc is long, it is
better to put it in a file.

Therefore, could there be a new feature in bash called, say, theredoc
be added for this case?
--
Regards,
Peng
Matthew Cengia
2015-03-20 00:14:17 UTC
Permalink
On 2015-03-19 19:06, Peng Yu wrote:
[...]
Post by Peng Yu
Heredoc is good when the doc is short. But when the doc is long, it is
better to put it in a file.
Therefore, could there be a new feature in bash called, say, theredoc
be added for this case?
This is not what bash is for. There are tools specifically designed for
this purpose, namely M4 (http://www.gnu.org/software/m4/manual/m4.html).
--
Regards,
Matthew Cengia
Ken Irving
2015-03-20 03:53:43 UTC
Permalink
Post by Dave Rutherford
Post by Peng Yu
Post by Eric Blake
Post by Peng Yu
I want to interpolate variable in a bash string variable. But the
following code shows the spaces are not maintained.
Insufficient quoting on your part.
Post by Peng Yu
There is some
built-in features in perl to do string interpolation. Does anyone know
the best way to do string interpolation in bash? Thanks.
I feel dumb. What is meant by `string interpolation'?
I think I first saw that term used in Perl, meaning to expand (perl)
variables in place in a string, but I didn't remember it or recognize
it in this context until a few posts in.
Post by Dave Rutherford
Post by Peng Yu
The real problem is that I want to replace some bash variables in a
file and then print the output.
A common term for expanding variables or tokens in a file is 'template
expansion'. I shy away from eval, so would probably treat the variables
as string tokens and subsitute the values in a loop.

Ken
John Kearney
2015-03-20 05:02:28 UTC
Permalink
Here is a simple example of how to do this manually.

ks_IntrStr_ExpandVars(){
local CLine Key
while IFS= read -r CLine ; do
while Key=${CLine##*\$\{} && [ "${Key}" != "${CLine}" ] ; do
Key=${Key%%\}*}
if [ -v "${Key}" ]; then
KeyValue=${!Key}
else
KeyValue="<unset-${Key}>"
fi
CLine=${CLine//\$\{${Key}\}/${KeyValue}}
done
echo "${CLine}"
done <<EOF
${1:-}
EOF
}

TestFormat='
Hello ${a} ${b} Hello
Hello ${${a}} Hello
'
ks_IntrStr_ExpandVars "${TestFormat}"
a=TestVarA ks_IntrStr_ExpandVars "${TestFormat}"
a=TestVarA b=TestVarB ks_IntrStr_ExpandVars "${TestFormat}"
TestVarA=TestTestVarA a=TestVarA b=TestVarB ks_IntrStr_ExpandVars
"${TestFormat}"

Note it does recursive expansion.
so the output of the above would be.


Hello <unset-a> <unset-b> Hello
Hello <unset-<unset-a>> Hello


Hello TestVarA <unset-b> Hello
Hello <unset-TestVarA> Hello


Hello TestVarA TestVarB Hello
Hello <unset-TestVarA> Hello


Hello TestVarA TestVarB Hello
Hello TestTestVarA Hello
Post by Ken Irving
Post by Dave Rutherford
Post by Peng Yu
Post by Eric Blake
Post by Peng Yu
I want to interpolate variable in a bash string variable. But the
following code shows the spaces are not maintained.
Insufficient quoting on your part.
Post by Peng Yu
There is some
built-in features in perl to do string interpolation. Does anyone
know
Post by Dave Rutherford
Post by Peng Yu
Post by Eric Blake
Post by Peng Yu
the best way to do string interpolation in bash? Thanks.
I feel dumb. What is meant by `string interpolation'?
I think I first saw that term used in Perl, meaning to expand (perl)
variables in place in a string, but I didn't remember it or recognize
it in this context until a few posts in.
Post by Dave Rutherford
Post by Peng Yu
The real problem is that I want to replace some bash variables in a
file and then print the output.
A common term for expanding variables or tokens in a file is 'template
expansion'. I shy away from eval, so would probably treat the variables
as string tokens and subsitute the values in a loop.
Ken
Eduardo A. Bustamante López
2015-03-20 00:04:30 UTC
Permalink
Post by Peng Yu
The real problem is that I want to replace some bash variables in a
file and then print the output.
This is hard to do safely. The simplest approaches (eval, sed) will lead
to arbitrary code execution, or to an escaping mess.
Post by Peng Yu
Please let me know if there is a convenient solution in bash.
I wrote: https://raw.githubusercontent.com/dualbus/myscripts/master/bin/stencil

Some time ago, precisely to deal with this issue. It's safe to use, because it
doesn't do any evaluation. It just replaces stuff that looks like bash
parameter expansions.

***@hp ~ % x="'" y=IJK z=XYZ ./stencil <<< '"$x" $y $z'
"'" IJK XYZ

It's POSIX awk, so that should work for all awks.
Peng Yu
2015-03-20 00:14:25 UTC
Permalink
On Thu, Mar 19, 2015 at 7:04 PM, Eduardo A. Bustamante López
Post by Eduardo A. Bustamante López
Post by Peng Yu
The real problem is that I want to replace some bash variables in a
file and then print the output.
This is hard to do safely. The simplest approaches (eval, sed) will lead
to arbitrary code execution, or to an escaping mess.
Post by Peng Yu
Please let me know if there is a convenient solution in bash.
I wrote: https://raw.githubusercontent.com/dualbus/myscripts/master/bin/stencil
This is essentially a parser, which definitely could be a solution.

But since bash has the parsing capability, it might be better to reuse
whatever already implemented in bash.
Post by Eduardo A. Bustamante López
Some time ago, precisely to deal with this issue. It's safe to use, because it
doesn't do any evaluation. It just replaces stuff that looks like bash
parameter expansions.
There are very complex bash variables (array, subarray, etc). I'm not
sure this code would be robust against all possible usages (for
example, ":" seems to be missing).
Post by Eduardo A. Bustamante López
"'" IJK XYZ
It's POSIX awk, so that should work for all awks.
--
Regards,
Peng
Eduardo A. Bustamante López
2015-03-20 01:34:55 UTC
Permalink
Post by Peng Yu
There are very complex bash variables (array, subarray, etc). I'm not
sure this code would be robust against all possible usages (for
example, ":" seems to be missing).
Well, I don't know about your use case. With every reply, it seems to be
getting more complex.

Then, I'd suggest to just quote properly and throw an eval there, with all the
risks involved.
Greg Wooledge
2015-03-20 12:20:08 UTC
Permalink
Post by Peng Yu
The real problem is that I want to replace some bash variables in a
file and then print the output.
http://mywiki.wooledge.org/TemplateFiles
Peng Yu
2015-03-21 02:39:24 UTC
Permalink
Post by Greg Wooledge
Post by Peng Yu
The real problem is that I want to replace some bash variables in a
file and then print the output.
http://mywiki.wooledge.org/TemplateFiles
The problem is that the variable must be exported to be usable. Is
there a way to somehow use the current shell?

~$ cat template.txt
$x $y
~$ cat main.sh
#!/usr/bin/env bash

export x=abc
export y=xyz

{
echo "cat <<EOF"
cat template.txt
echo "EOF"
} | bash
~$ ./main.sh
abc xyz
--
Regards,
Peng
Dan Douglas
2015-03-21 23:05:02 UTC
Permalink
Post by Peng Yu
The real problem is that I want to replace some bash variables in a
file and then print the output.
So you want to take the content of a file and expand it as though it were
the content of a heredoc? So in this example FD 3 is your template file and
4 expands the template into the code that generates result.

#!/usr/bin/env bash

x=abc y=$'def\n ghi ' z=\$x

{ eval "$(</dev/fd/4)"; } 3<<\TEMPLATE 4<<-CODE
blah blah $x
blah $y $z
TEMPLATE
{ result=\$(</dev/fd/5); } 5<<EOF
$(</dev/fd/3)
EOF
CODE

printf '%s\n' "$result"

# output:
# blah blah abc
# blah def
# ghi $x

...Provided the template is correct, the result will be correct for any
input variables. I didn't bother fixing trailing newline stripping. You
can fix that if you want.

The only reason I've ever had to do this is for generating complex
tests for testing shell code itself. I can't think of any other good
reason. I wouldn't use this technique merely to abuse a shell
as a macro expander.

--
Dan Douglas

Loading...