Re: Piping commands to a shell but keeping interactivity

James Harris <james.harris.1@gmail.com> wrote:

On 08/03/2024 03:39, Rich wrote:

James Harris <james.harris.1@gmail.com> wrote:

That's right.  What this is for is code to list files in a folder
which are duplicates of those in another folder (same name, same
relative place in the folder hierarchy, etc).  For example, say there
are two folders, c and d, and one is potentially a copy of the other.
The command
>
$ ./lsdup.py -r c d
>
lists files in d (and subdirectories due to the -r recurse option) which
are duplicates of those in c.
>
In answer to your other point about using xargs, I would use it if it
would do what's required, and do so consistently,

 It will, provided you output file names with ASCII null terminators
instead of ASCII newlines, and use the -0 option to tell xargs the
filenames are null separated.
 

but I am not sure whether I can trust it or not.

 You can.  Provided you feed it ASCII null terminated filenames, it will
work properly, no matter what other weird characters might be in the
filenames.

 You are right: unusual characters in file names is the kind of issue I am wary about with xargs.

And this is exactly why xargs was modified to consume null terminated names, so that no otherwise legal filename character would create a problem.  And ASCII newline is also a legal filename character.

Running a command to do automatic deletion requires a lot of trust in the command's operation.

Agreed.

I /have/ been thinking about adding a -0 option but it has issues such as:
 (1) With -0 one cannot easily postprocess the list of file names, if required.

That depends upon what you mean by 'postprocessing'.
GNU sort has -z, --zero-terminated options to inform it to process "lines" with ASCII null terminators.
GNU grep has -z, --null-data options to also inform it to process "lines" that are ASCII null terminated.
If you want to "preview" in something like less, you can add "tr '\0' '\n'" to the pipeline before less to conver the nulls into newlines so that any filename other than one with an actual newline in it would be one line in less.

(2) With xargs interactivity (e.g. the -i in rm -i) is lost.

This is true, so if you value using -i to acknowledge each file individually then yes, using xargs to launch rm is not in the cards.  Although a small workaround would be to use xargs (GNU xargs at least) option -p, --interactive combined with -n 1 to run one rm per filename and have xargs prompt (instead of rm) before each.  This loses the efficiency gains of one rm and a maximum number of filenames, but it does allow 'interactivity' again.

Yes, there are ways round such issues but they may require altering a command line /after/ it has been found to be correct. That was the issue which led to this thread. If I I generate a list of delete commands with a series of commands such as
   A | B
 then once I am happy with them I would prefer simply to append | sh as in
   A | B | sh
 rather than changing the form to
   sh <(A | B)

With xargs and nulls, the change becomes:
A | B | tr '\0' '\n' | less to
A | B | xargs -0 rm or
A | B | xargs -0 -n 1 -p rm (for interactive deletes)

This is not so much about convenience as about making sure less can go wrong - always a good idea when deleting files by means of a command.

Yes, if you have no backup with which to recover then agreed, verifying the deletion before deleting is important.

(A separate command, lsempty.py, is is used to delete the resultant
empty folders.)

 A separate python command is completely unnecessary.  Removing empty leaf directories is already easy using the tools provided by the system.:
 $ find c d -type d -empty -print0 | xargs -0 rmdir
 If you also want to remove parent empty directories should all their
children go away, change to:
 $ find c d -type d -empty -print0 | xargs -0 rmdir -p

 Thanks. That looks as though it would work, albeit that it would print a few spurious error messages which add to the work of the person running the code. For example,

If you dislike the spurious erros, then add the --ignore-fail-on-non-empty option, which suppresses the errors for 'not empty' as rmdir walks back up the tree.

In answer to the point about filenames with spaces and odd characters, I currently output names in single quotes, as above.  This will complain until the parent is empty, but the complaints can be ignored.

 Single quote is also a possible filename character, so if, by chance, you end up with a file containing one ' somewhere your wrapping in single quotes will result in a fail at that point.

 I deal with that (at present) by juxtaposing single-quoted strings.  For example, if there is a file called
   won't scan.txt
 with an apostrophe and a space then I get the following results. First, the command:
 $ ./lsdup.py c d -r | grep won | sed 's/^/rm -v /'
rm -v 'd/won'\''t scan.txt'

Ok, you had not yet revealed you were doing so, and we can't read your mind remotely over Usenet to know you are handling single quotes properly (at least for one single quote in a filename).

That said, I am not sure that this will work in all cases and expect

Provided you replace every input instance of ' with '\'' it would work (at least for ').
The minefield aspect into which you are traversing is that each shell has a different set of meta-characters that are 'special' to it and so the proper "escaping" is dependent upon /which/ shell the output is being piped into.
And by piping into the shell, if your escaping is not 100% perfect, you open yourself up to someone crafting a filename that results in your shell doing one of:
   "rm -fr /" or    "rm -fr ~"
depending upon whether you run the deletion as root or a non-root user.