On 21.09.24 20:53, fir wrote:
>
>
> i think if to write a simple comandline program
> that remove duplicates in a given folder
[...]
I have had the same problem. My solution was to use extended file
attributes and some file checksum, eg sha512sum, also, I wrote this in
PERL (see code below). Using the file attributes, I can re-run the
program after a while without having to re-calculate the checksums.
So, this solution only works for filesystems that have extended file
attributes, but you could also use some simple database (sqlite3?) to
map checksums to pathnames.
What I did was to walk through the directory tree and check if the file
being considered already has a checksum in an extended attribute. If
not. I'll calculate the checksum and store that in the extended
attribute. Also, I store the pathname in a hash (remember, this is
PERL), key is the checksum.
If there is a collision (checksum already in the hash), I remove the new
file (and link the new filename to the old file). One could be paranoid
and do a byte-by-byte file comparison then.
If I needed to do this in a C program, I'd probably use a GList to store
the hash, but otherwise the code logic would be the same.
HTH,
Josef
#! /usr/bin/perl
use warnings;
use strict;
use File::ExtAttr ':all'; # In case of problems, maybe insert "use
Scalar:Utils;" in /usr/lib/x86_64-linux-gnu/perl5/5.22/File/ExtAttr.pm
use Digest::SHA;
use File::Find;
use Getopt::Std;
# OPTIONS:
# s: force symlink
# n: don't do the actula removing/linking
# v: be more verbose
# h: print short help
my %opt = (
s => undef,
n => undef,
v => undef,
h => undef,
);
getopts('hnsv', \%opt);
if ($opt{h}) {
print STDERR "usage: lndup [-snvh] [dirname..]\n";
print STDERR "\t-s: use symlink rather than hard link\n";
print STDERR "\t-n: don't remove/link, just show what would be done\n";
print STDERR "\t-v: be more verbose (show pathname and SHA512 sum\n";
print STDERR "\t-h: show this text\n";
exit(0);
}
my %file;
if (@ARGV == 0) {
find({ wanted => \&lndup, no_chdir => 1 }, '.');
} else {
find({ wanted => \&lndup, no_chdir => 1 }, @ARGV);
}
# NAME: lndup
# PURPOSE: To handle a single file
# ARGUMENTS: None, pathname is taken from $File::Find::name
# RETURNS: Nothing
# NOTE: The SHA512 sum of a file is calculated.
# IF a file with the same sum was already found earlier, AND
# iF both files are NOT the same (hard link) AND
# iF both files reside on the same disk
# THEN the second occurrence is removed and
# replaced by a link to the first occurrence
sub lndup {
my $pathname = $File::Find::name;
return if ! -f $pathname;
if (-s $pathname) {
my $sha512sum = getfattr($pathname, 'SHA512');
if (!defined $sha512sum) {
my $ctx = Digest::SHA->new(512);
$ctx->addfile($pathname);
$sha512sum = $ctx->hexdigest;
print STDERR "$pathname $sha512sum\n" if $opt{v};
setfattr($pathname, "SHA512", $sha512sum);
} elsif ($opt{v}) {
print STDERR "Using sha512sum from attributes\n";
}
if (exists $file{$sha512sum}) {
if (!same_file($pathname, $file{$sha512sum})) {
my $links1 = (stat($pathname))[3];
my $links2 = (stat($file{$sha512sum}))[3];
# If one of them is a symbolic link, make sure it's
$pathname
if (is_symlink($file{$sha512sum})) {
print STDERR "Swapping $pathname and
$file{$sha512sum}\n" if $opt{v};
swap($file{$sha512sum}, $pathname);
}
# If $pathname has more links than $file{$sha512sum},
# exchange the two names.
# This ensures that $file{$sha512sum} has the most links.
elsif ($links1 > $links2) {
print STDERR "Swapping $pathname and
$file{$sha512sum}\n" if $opt{v};
swap($file{$sha512sum}, $pathname);
}
print "rm \"$pathname\"; ln \"$file{$sha512sum}\"
\"$pathname\"\n";
if (! $opt{n}) {
my $same_disk = same_disk($pathname,
$file{$sha512sum});
if (unlink($pathname)) {
if (! $same_disk || $opt{s}) {
symlink($file{$sha512sum}, $pathname) ||
print STDERR "Failed to symlink($file{$sha512sum}, $pathname): $!\n";
} else {
link($file{$sha512sum}, $pathname) || print
STDERR "Failed to link($file{$sha512sum}, $pathname): $!\n";
}
} else {
print STDERR "Failed to unlink $pathname: $!\n";
}
}
# print "Removing $pathname\n";
# unlink $pathname or warn "$0: Cannot remove $_: $!\n";
}
} else {
$file{$sha512sum} = $pathname;
}
}
}
# NAME: same_disk
# PURPOSE: To check if two files are on the same disk
# ARGUMENTS: pn1, pn2: pathnames of files
# RETURNS: true if files are on the same disk, else false
# NOTE: The check is made by comparing the device numbers of the
# filesystems of the two files.
sub same_disk {
my ($pn1, $pn2) = @_;
my @s1 = stat($pn1);
my @s2 = stat($pn2);
return $s1[0] == $s2[0];
}
# NAME: same_file
# PURPOSE: To check if two files are the same
# ARGUMENTS: pn1, pn2: pathnames of files
# RETURNS: true if files are the same, else false
# NOTE: files are the same if device number AND inode number
# are identical
sub same_file {
my ($pn1, $pn2) = @_;
my @s1 = stat($pn1);
my @s2 = stat($pn2);
return ($s1[0] == $s2[0]) && ($s1[1] == $s2[1]);
}
sub is_symlink {
my ($path) = @_;
return -l $path;
}
sub swap {
my $tmp;
$tmp = $_[0];
$_[0] = $_[1];
$_[1] = $tmp;
}
|
|