[Help-bash] bash 4.3 escaping : (breaking script), unlike 4.2. Huh?

[B. S.]

In kubuntu 12.04, bash (4.2.25) line of:

tmpstr="{ md5sum - ; echo \$TAR_REALNAME ; } | sed -e
':a;N;\$!ba;s/-\\n//g'"

when used in a tar --to-command call:

tar blah --to-command="${tmpstr}" --extract

worked fine.


In kubuntu 14.04, bash (4.3.11), same code, causes tar (--to-command
subshell) to report:

tar: { md5sum - ; echo $TAR_REALNAME ; } | sed -e
'\:a;N;$!ba;s/-\\n//g': Cannot exec: No such file or directory
tar: Error is not recoverable: exiting now


i.e. the colon is now getting escaped.

- what has changed 4.2 to 4.3 (i.e. what magic search term will take me
directly to an explanation of what's going on / changed)?


- how do I stop the : from being escaped?

TIA!

[Eduardo Bustamante]

OK, so the assumption here is that your interactive shell is Bash.
Nothing has changed in bash that would cause the escaping that you're
seeing. Bash passes the arguments unmodified to GNU tar (another
assumption is that you're running system's tar, which in Ubuntu 14.04
is GNU tar 1.27.1-1 [1]).

Reading through the commit logs in GNU tar code repository, you can
find the following commit:

hp% PAGER=cat git show 61cd3f
commit 61cd3fd26855d40ee3a6491b2aa3611cd2aae46c
Author: Sergey Poznyakoff <***@gnu.org.ua>
Date: Thu Apr 14 11:51:38 2016 +0300

Fix argument handling when running external commands.

* src/system.c (xexec): Use sh -c to run the command. This fixed
bug introduced by 7b5e80396 (tar 1.27)
* doc/tar.texi: Fix checkpoint examples: (1) $TAR_FILENAME
is not available when creating archive and (2) --checkpoint
can't be used as abbreviation of --checkpoint-action
[...]

If you inspect the patch for src/system.c [2], you'll realize that
this was fixed after version 1.27. I'm not sure what the best option
is here, but you need to upgrade your tar command.

More links to read:

- Re: [Bug-tar] How one can do untar a file and gzip
its files with one command line?
http://osdir.com/ml/bug-tar-gnu/2016-04/msg00011.html


[1] http://packages.ubuntu.com/trusty/tar
[2] http://git.savannah.gnu.org/cgit/tar.git/commit/?id=61cd3fd26855d40ee3a6491b2aa3611cd2aae46c

[B. S.]

Thanks!

Would never have found this, let alone with such efficiency!

Worst case, current code of if later bash (which I guess I should change
to if later tar), of:

mytocommands=blah

# inserted code
if later bash {
myshfile=mktemp
echo $mytocommands > $myshfile
mytocommands=myshfile
}

Letting me leave 'tar -to-command=$mytocommands' alone, works.

I expect I'll try mytocommands="bash blah", but otherwise won't waste
further eyespace on this - I've been hunting this wumpus for many hours
and several days now. (Saying to myself "It shouldn't be THIS hard!"
several times along the way.)


* Thanks to everyone for having this list, and it being so responsive! A
few minutes here, some patience == a faster path to resolution than all
the time I spent hunting prior.

Post by Eduardo Bustamante
OK, so the assumption here is that your interactive shell is Bash.
Nothing has changed in bash that would cause the escaping that you're
seeing. Bash passes the arguments unmodified to GNU tar (another
assumption is that you're running system's tar, which in Ubuntu 14.04
is GNU tar 1.27.1-1 [1]).
Reading through the commit logs in GNU tar code repository, you can
hp% PAGER=cat git show 61cd3f
commit 61cd3fd26855d40ee3a6491b2aa3611cd2aae46c
Date: Thu Apr 14 11:51:38 2016 +0300
Fix argument handling when running external commands.
* src/system.c (xexec): Use sh -c to run the command. This fixed
bug introduced by 7b5e80396 (tar 1.27)
* doc/tar.texi: Fix checkpoint examples: (1) $TAR_FILENAME
is not available when creating archive and (2) --checkpoint
can't be used as abbreviation of --checkpoint-action
[...]
If you inspect the patch for src/system.c [2], you'll realize that
this was fixed after version 1.27. I'm not sure what the best option
is here, but you need to upgrade your tar command.
- Re: [Bug-tar] How one can do untar a file and gzip
its files with one command line?
http://osdir.com/ml/bug-tar-gnu/2016-04/msg00011.html
[1] http://packages.ubuntu.com/trusty/tar
[2] http://git.savannah.gnu.org/cgit/tar.git/commit/?id=61cd3fd26855d40ee3a6491b2aa3611cd2aae46c

[B. S.]

Oh for pity's sake ... thank you! It never occurred to me to blame tar.
Silly me.

tar has indeed changed between the two, e.g. 14.04 understands --acls
--xattr --selinux


Kubuntu 12.04, GNU bash, version 4.2.25(1)-release
(x86_64-pc-linux-gnu), tar (GNU tar) 1.26


Kubuntu 14.04, GNU bash, version 4.3.11(1)-release (i686-pc-linux-gnu),
tar (GNU tar) 1.27.1


FWIW ...

uname -a ; lsb_release -a

12.04:
Linux testwmedia 3.13.0-93-generic #140~precise1-Ubuntu SMP Tue Jul 19
19:55:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

LSB Version:
core-2.0-amd64:core-2.0-noarch:core-3.0-amd64:core-3.0-noarch:core-3.1-amd64:core-3.1-noarch:core-3.2-amd64:core-3.2-noarch:core-4.0-amd64:core-4.0-noarch
Distributor ID: Ubuntu
Description: Ubuntu 12.04.5 LTS
Release: 12.04
Codename: precise


14.04:

Linux teststore5 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:47
UTC 2016 i686 i686 i686 GNU/Linux

LSB Version:
core-2.0-ia32:core-2.0-noarch:core-3.0-ia32:core-3.0-noarch:core-3.1-ia32:core-3.1-noarch:core-3.2-ia32:core-3.2-noarch:core-4.0-ia32:core-4.0-noarch:core-4.1-ia32:core-4.1-noarch:cxx-3.0-ia32:cxx-3.0-noarch:cxx-3.1-ia32:cxx-3.1-noarch:cxx-3.2-ia32:cxx-3.2-noarch:cxx-4.0-ia32:cxx-4.0-noarch:cxx-4.1-ia32:cxx-4.1-noarch:desktop-3.1-ia32:desktop-3.1-noarch:desktop-3.2-ia32:desktop-3.2-noarch:desktop-4.0-ia32:desktop-4.0-noarch:desktop-4.1-ia32:desktop-4.1-noarch:graphics-2.0-ia32:graphics-2.0-noarch:graphics-3.0-ia32:graphics-3.0-noarch:graphics-3.1-ia32:graphics-3.1-noarch:graphics-3.2-ia32:graphics-3.2-noarch:graphics-4.0-ia32:graphics-4.0-noarch:graphics-4.1-ia32:graphics-4.1-noarch:languages-3.2-ia32:languages-3.2-noarch:languages-4.0-ia32:languages-4.0-noarch:languages-4.1-ia32:languages-4.1-noarch:multimedia-3.2-ia32:multimedia-3.2-noarch:multimedia-4.0-ia32:multimedia-4.0-noarch:multimedia-4.1-ia32:multimedia-4.1-noarch:printing-3.2-ia32:printing-3.2-noarch:printing-4.0-ia32:print
ing-4.0-noarch:printing-4.1-ia32:printing-4.1-noarch:qt4-3.1-ia32:qt4-3.1-noarch:security-4.0-ia32:security-4.0-noarch:security-4.1-ia32:security-4.1-noarch
Distributor ID: Ubuntu
Description: Ubuntu 14.04.5 LTS
Release: 14.04
Codename: trusty

What's tar --version on both systems? This smells like a tar issue and not
a bash issue. Tar's error message looks like it's treating the entire
string as a command to execute, not a shell string. In other words, it
looks like it's calling exec*() (no shell) and not system() (uses /bin/sh).
Andy

Post by B. S.
tmpstr="{ md5sum - ; echo \$TAR_REALNAME ; } | sed -e
':a;N;\$!ba;s/-\\n//g'"
tar blah --to-command="${tmpstr}" --extract
worked fine.
In kubuntu 14.04, bash (4.3.11), same code, causes tar (--to-command
Cannot exec: No such file or directory
tar: Error is not recoverable: exiting now
i.e. the colon is now getting escaped.
- what has changed 4.2 to 4.3 (i.e. what magic search term will take me
directly to an explanation of what's going on / changed)?
- how do I stop the : from being escaped?
TIA!