Re: Files tree

James Harris <james.harris.1@gmail.com> wrote:

export LC_ALL=C
sudo find /\
 -path "/proc/*" -prune -o\
 -path "/run/*" -prune -o\
 -path "/sys/*" -prune -o\
 -path "/tmp/*/*" -prune -o\
 -print0 | sort -z | tr '\0' '\n' > /tmp/f1

If you are going to output null terminated filenames (-print0) then
don't almost immediately throw out the nulls by converting them to
newlines.  The purpose of -print0 and the nulls is to avoid *any*
problems with any filename character (i.e., a filename /can/ contain a
newline).  If, by any chance, you have even one filename with  a
newline in the name, converting the nulls to newlines for storage will
break the storage file (i.e., you can't differentate the "newlines
ending filenames" from the "newlines that belong inside a filename").

Convert the  nulls to newlines only when you want to view with less,
then your "files of records" are not corrupt from the start.:

   tr '\0' '\n' < /tmp/f1 | less   ; or

   < /tmp/f1 tr '\0' '\n' | less  ; if you prefer the input file on the
                                    left

You'll see I made some choices such as to omit files from /proc but not
from /dev, for example, to record any lost+found contents, to record
mounted filesystems, to show just one level of /tmp, etc.

You get to decide which 'files' are important to track for whatever it
is you are wanting to do.

I am not sure I coded the command right albeit that it seems to work on
test cases.

It will work, until that day you end up with a file name containing a
newline, then it will break for that filename.  Now, if you never
encounter a filename with a newline it won't break.  But the change to
make sure it never breaks is trivial.

The output from that starts with lines such as
 
/
/bin
/boot
/boot/System.map-5.15.0-101-generic
/boot/System.map-5.15.0-102-generic
...etc...
 
Such a form would be ideal for input to grep and diff to look for
relevant files that have been added or removed between any two runs.

grep supports "null terminated lines" using the "--null-data" so you
could 'grep' null terminated lines files without translating them.

The second, and less important, part is to store (in a separate file)
info about each of the file names as that may be relevant in some cases.
That takes the first file as input and has the following form.
 
cat /tmp/f1 |\
 tr '\n' '\0' |\
 xargs -0 sudo ls -ld > /tmp/f2

Here's where you will /break/ things if you ever encounter a file with
a newline in its filename, converting the newlines back to nulls
results in that filename with an embedded newline becoming "two lines"
(two files) -- and neither will be found.

If you kept the nulls in f1, your above can be:

  xargs -0 ls -ld < /tmp/f2 > /tmp/f2

However, if you want 'data' on the files, you'd be better off using the
'stat' command, as it is intended to 'aquire' meta-data about files
(and by far more meta-data than what ls shows).

You'd feed it filenames with xargs as well:

   xargs -0 stat --format="%i %h %n" > /tmp/f2

To output inode, number of links, and name (granted, you can get this
from ls as well, but look through the stat man page, there is a lot
more stuff you can pull out).

The one tool that does not (yet) seem to consume "null terminated
lines" is diff, and you can work around that by converting to newlines
at the time you do the diff:

  diff -u <(tr '\0' '\n' < /tmp/f1) <(tr '\0' '\n' < /tmp/f2)

And, note, all of these "convert nulls to newlines at time of need" can
be scripted, so you could have a "file-list-diff" script that contains:

  #!/bin/bash
  diff -u <(tr '\0' '\n' < "$1") <(tr '\0' '\n' < "$2")

And then you can diff two of your audit files by:

  file-list-diff /tmp/f1 /tmp/f2

And not have to remember the tr invocation and process substitution
syntax to do the same at every call.