Discussion:
[Help-bash] bash way of variable substitution
Peng Yu
2018-02-07 21:15:58 UTC
Permalink
Hi,

I'd like to do variable substitution using the bash syntax (only allow
things like ${x} or $x without allowing anything else in bash) instead
of using other tools like m4. In order to not to have a conflict with
environmental variables, I did not use bash for this implementation.
Instead, I come up with the following implement in python. But I am
not how variable substitution is implemented internally in bash and
whether there are any other better and cleverer implementations.

Does anybody have any other suggestions about a better implementation?

$ cat varbashsub0.py
#!/usr/bin/python
# vim: set noexpandtab tabstop=2 shiftwidth=2 softtabstop=-1 fileencoding=utf-8:

import sys
import re

rpl_dict = {}

for s, d in zip(sys.argv[1::2], sys.argv[2::2]):
rpl_dict[s] = d

regex=r'(\\\\)|(\\\$)' +
'|\$(([A-Za-z_][A-Za-z0-9_]*)|{([A-Za-z_][A-Za-z0-9_]*)})'
r = re.compile(regex)

def rpl_str(r, s, rpl_dict):
result = []
prev_end = 0
for m in r.finditer(s):
if m.group(1):
varval = '\\'
elif m.group(2):
varval = '$'
else:
varname = m.group(4)
if varname is None:
varname = m.group(5)

if varname in rpl_dict:
varval = rpl_dict[varname]
else:
varval = ''

result.append(s[prev_end:m.start()])
result.append(varval)
prev_end = m.end()

result.append(s[prev_end:])

return result

for line in sys.stdin:
line = line.rstrip('\n')
print ''.join(rpl_str(r, line, rpl_dict))

$ ./varbashsub0.py <<< '\\\$a'
\$a
$ ./varbashsub0.py <<< '\$'
$
$ ./varbashsub0.py <<< 'x${ab}y'
xy
$ ./varbashsub0.py <<< 'x$ab y'
x y
$ ./varbashsub0.py a 1 b 2 <<< '${a}$b'
12
$ ./varbashsub0.py a 1 b 2 <<< 'x${a}${b}y'
x12y
--
Regards,
Peng
Tim Visher
2018-02-08 14:43:51 UTC
Permalink
It's 10,000 lines of C but this is how substitution is implemented in Bash.

https://git.savannah.gnu.org/cgit/bash.git/tree/subst.c

Given the generality of your question I'm not sure what else to point you
to.
Post by Peng Yu
Hi,
I'd like to do variable substitution using the bash syntax (only allow
things like ${x} or $x without allowing anything else in bash) instead
of using other tools like m4. In order to not to have a conflict with
environmental variables, I did not use bash for this implementation.
Instead, I come up with the following implement in python. But I am
not how variable substitution is implemented internally in bash and
whether there are any other better and cleverer implementations.
Does anybody have any other suggestions about a better implementation?
$ cat varbashsub0.py
#!/usr/bin/python
# vim: set noexpandtab tabstop=2 shiftwidth=2 softtabstop=-1
import sys
import re
rpl_dict = {}
rpl_dict[s] = d
regex=r'(\\\\)|(\\\$)' +
'|\$(([A-Za-z_][A-Za-z0-9_]*)|{([A-Za-z_][A-Za-z0-9_]*)})'
r = re.compile(regex)
result = []
prev_end = 0
varval = '\\'
varval = '$'
varname = m.group(4)
varname = m.group(5)
varval = rpl_dict[varname]
varval = ''
result.append(s[prev_end:m.start()])
result.append(varval)
prev_end = m.end()
result.append(s[prev_end:])
return result
line = line.rstrip('\n')
print ''.join(rpl_str(r, line, rpl_dict))
$ ./varbashsub0.py <<< '\\\$a'
\$a
$ ./varbashsub0.py <<< '\$'
$
$ ./varbashsub0.py <<< 'x${ab}y'
xy
$ ./varbashsub0.py <<< 'x$ab y'
x y
$ ./varbashsub0.py a 1 b 2 <<< '${a}$b'
12
$ ./varbashsub0.py a 1 b 2 <<< 'x${a}${b}y'
x12y
--
Regards,
Peng
Chet Ramey
2018-02-08 15:04:36 UTC
Permalink
Post by Tim Visher
It's 10,000 lines of C but this is how substitution is implemented in Bash.
The Springsteen quote at the top of the file still applies.
--
``The lyf so short, the craft so long to lerne.'' - Chaucer
``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRU ***@case.edu http://tiswww.cwru.edu/~chet/
Peng Yu
2018-02-08 15:28:16 UTC
Permalink
Post by Chet Ramey
Post by Tim Visher
It's 10,000 lines of C but this is how substitution is implemented in Bash.
It is too long for me to understand. And because it does more than my
specification in the original email, most of the code probably is not
relevant to my spec.

For my spec, is my implementation a reasonable replication of what is
implemented in bash?
Post by Chet Ramey
The Springsteen quote at the top of the file still applies.
I am not sure what you refer to.
--
Regards,
Peng
Phil Dobbin
2018-02-08 16:17:40 UTC
Permalink
Post by Peng Yu
Post by Chet Ramey
Post by Tim Visher
It's 10,000 lines of C but this is how substitution is implemented in Bash.
It is too long for me to understand. And because it does more than my
specification in the original email, most of the code probably is not
relevant to my spec.
For my spec, is my implementation a reasonable replication of what is
implemented in bash?
Post by Chet Ramey
The Springsteen quote at the top of the file still applies.
I am not sure what you refer to.
at the top of the file is a line from one of Bruce Springsteen's songs
[1]. It's nestled in the comments.

Cheers,

Phil.

[1] the song is 'Thunder Road' I believe...
--
The sun shone, having no alternative, on the nothing new
Murphy by Samuel Beckett

Loading Image...
Chet Ramey
2018-02-08 16:22:03 UTC
Permalink
Post by Phil Dobbin
Post by Peng Yu
I am not sure what you refer to.
at the top of the file is a line from one of Bruce Springsteen's songs
[1]. It's nestled in the comments.
Cheers,
Phil.
[1] the song is 'Thunder Road' I believe...
It is, in fact, "Thunder Road". Springsteen's best song, hands down.
--
``The lyf so short, the craft so long to lerne.'' - Chaucer
``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRU ***@case.edu http://tiswww.cwru.edu/~chet/
Pierre Gaston
2018-02-08 14:55:55 UTC
Permalink
Post by Peng Yu
Hi,
I'd like to do variable substitution using the bash syntax (only allow
things like ${x} or $x without allowing anything else in bash) instead
of using other tools like m4. In order to not to have a conflict with
environmental variables, I did not use bash for this implementation.
Instead, I come up with the following implement in python. But I am
not how variable substitution is implemented internally in bash and
whether there are any other better and cleverer implementations.
Does anybody have any other suggestions about a better implementation?
check http://mywiki.wooledge.org/TemplateFiles
Peng Yu
2018-02-08 15:24:12 UTC
Permalink
Post by Pierre Gaston
check http://mywiki.wooledge.org/TemplateFiles
Neither of the solutions is what I need.

- the one based on sed: I want the variable not specified to be
replaced be replaced with an empty string. sed does not do so.
- envsubst will replace all environment variables. I only want to
replace variables that I specified to replace. For ones I do not
specify, replace with empty strings.
--
Regards,
Peng
Greg Wooledge
2018-02-08 15:35:38 UTC
Permalink
Post by Peng Yu
- envsubst will replace all environment variables. I only want to
replace variables that I specified to replace. For ones I do not
specify, replace with empty strings.
help unset
man env
Peng Yu
2018-02-11 18:03:46 UTC
Permalink
Post by Pierre Gaston
check http://mywiki.wooledge.org/TemplateFiles
It seems that envsubst does not serve the purpose.

$ envsubst <<< '\\\$a'
\\\
$ echo "\\\$a"
\$a
--
Regards,
Peng
Greg Wooledge
2018-02-12 13:50:16 UTC
Permalink
Post by Peng Yu
Post by Pierre Gaston
check http://mywiki.wooledge.org/TemplateFiles
It seems that envsubst does not serve the purpose.
WHAT purpose? How are we supposed to guess what your desired output is?
Post by Peng Yu
$ envsubst <<< '\\\$a'
\\\
It replaced $a with the value of that environment variable. Is that
not what you wanted?

Did you intend for the preceding backslashes to be significant? In
what way? What did you expect the output to be, and why?

Are you trying to write a bash syntax parser? There is only one bash
syntax parser in the entire world, and that is bash itself. Every
other thing that tries to parse bash syntax (text editors doing colors,
etc.) inevitably fails to meet some weird corner case.

Loading...