Re: program to remove duplicates

On 9/22/2024 12:29 AM, Paul wrote:

On Sat, 9/21/2024 10:36 PM, fir wrote:

Lawrence D'Oliveiro wrote:

On Sun, 22 Sep 2024 00:18:09 +0200, fir wrote:
>

... you just need to read all files in
folder and compare it byte by byte to other files in folder of the same
size

>
For N files, that requires N × (N - 1) ÷ 2 byte-by-byte comparisons.
That’s an O(N²) algorithm.
>
There is a faster way.
>

not quite as most files have different sizes so most binary comparsions
are discarded becouse size of files differ (and those sizes i read linearly when bulding lidt of filenames)
>
what i posted seem to work ok, it odesnt work fast but hard to say if it can be optimised or it takes as long as it should..hard to say

 The normal way to do this, is do a hash check on the
files and compare the hash. You can use MD5SUM, SHA1SUM, SHA256SUM,
as a means to compare two files. If you want to be picky about
it, stick with SHA256SUM.

[...]
That's fine.
file_0.bin
file_1.png
file_2.jpg
Say they all were identical wrt their actual bytes. The hash for them would all be the same. As long as they did not hash the file name in there for some reason... ;^)