How do you find duplicates from Windows SMB shares using Linux

[DustinB3403]

@DustinB3403 said in How do you find duplicates from Windows SMB shares using Linux:

@IRJ Yeah the output part is really simple, fdupes seems really simple too.

fdupes -rmsHA --sameline /target > output.log is running.

I just wasn't sure if there was any better options out there.

I just realized that the --sameline option can be replaced with -1 as in number one. The manual isn't clear about that and reading the option itself is difficult to delineate the difference.

[IRJ]

@DustinB3403 said in How do you find duplicates from Windows SMB shares using Linux:

@IRJ Yeah the output part is really simple, fdupes seems really simple too.

fdupes -rmsHA --sameline /target > output.log is running.

I just wasn't sure if there was any better options out there.

you also may want to grep for certain data if the entire output is too noisy

[DustinB3403]

@IRJ said in How do you find duplicates from Windows SMB shares using Linux:

@DustinB3403 said in How do you find duplicates from Windows SMB shares using Linux:

@IRJ Yeah the output part is really simple, fdupes seems really simple too.

fdupes -rmsHA --sameline /target > output.log is running.

I just wasn't sure if there was any better options out there.

you also may want to grep for certain data if the entire output is too noisy

Normally I would filter down, but since I'm just trying to get a grasp on the amount of potential duplication that there is, filtering at this point would only skew that number.

[pattonb]

@DustinB3403 some folks claim jdupes is faster, I have used both, and did not much of a difference.
Both work well.

[pattonb]

@pattonb to get an idea of how many dupes use the following

fdupes -r -m /directory(share to scan)

[Dashrender]

I wonder if this would run faster directly on the server in powershell instead? I'm assuming with doing this over SMB you have to download all files, run the hash - if ran locally, you get to skip the download time, I assume.

[IRJ]

@Dashrender said in How do you find duplicates from Windows SMB shares using Linux:

I wonder if this would run faster directly on the server in powershell instead? I'm assuming with doing this over SMB you have to download all files, run the hash - if ran locally, you get to skip the download time, I assume.

I gathered that the SMB shares are hosted on Linux, but I could be wrong.

If they are hosted on Windows like you are assuming, then I would agree that PowerShell would probably be most performant for this.

[Dashrender]

@IRJ said in How do you find duplicates from Windows SMB shares using Linux:

@Dashrender said in How do you find duplicates from Windows SMB shares using Linux:

I wonder if this would run faster directly on the server in powershell instead? I'm assuming with doing this over SMB you have to download all files, run the hash - if ran locally, you get to skip the download time, I assume.

I gathered that the SMB shares are hosted on Linux, but I could be wrong.

If they are hosted on Windows like you are assuming, then I would agree that PowerShell would probably be most performant for this.

The title says - Windows SMB Shares.

My guess is that Dustin is a lone wolf running a 'nix OS as his machine - and the rest of the company is using Windows. Nothing wrong with that, just my guess.

[JaredBusch]

@Dashrender said in How do you find duplicates from Windows SMB shares using Linux:

@IRJ said in How do you find duplicates from Windows SMB shares using Linux:

@Dashrender said in How do you find duplicates from Windows SMB shares using Linux:

I wonder if this would run faster directly on the server in powershell instead? I'm assuming with doing this over SMB you have to download all files, run the hash - if ran locally, you get to skip the download time, I assume.

I gathered that the SMB shares are hosted on Linux, but I could be wrong.

If they are hosted on Windows like you are assuming, then I would agree that PowerShell would probably be most performant for this.

The title says - Windows SMB Shares.

My guess is that Dustin is a lone wolf running a 'nix OS as his machine - and the rest of the company is using Windows. Nothing wrong with that, just my guess.

His company is significantly Mac.

[Dashrender]

@JaredBusch said in How do you find duplicates from Windows SMB shares using Linux:

@Dashrender said in How do you find duplicates from Windows SMB shares using Linux:

@IRJ said in How do you find duplicates from Windows SMB shares using Linux:

@Dashrender said in How do you find duplicates from Windows SMB shares using Linux:

I wonder if this would run faster directly on the server in powershell instead? I'm assuming with doing this over SMB you have to download all files, run the hash - if ran locally, you get to skip the download time, I assume.

I gathered that the SMB shares are hosted on Linux, but I could be wrong.

If they are hosted on Windows like you are assuming, then I would agree that PowerShell would probably be most performant for this.

The title says - Windows SMB Shares.

My guess is that Dustin is a lone wolf running a 'nix OS as his machine - and the rest of the company is using Windows. Nothing wrong with that, just my guess.

His company is significantly Mac.

aww, that's right - he has been asking a lot of MAC questions lately.

[DustinB3403]

@Dashrender said in How do you find duplicates from Windows SMB shares using Linux:

@JaredBusch said in How do you find duplicates from Windows SMB shares using Linux:

@Dashrender said in How do you find duplicates from Windows SMB shares using Linux:

@IRJ said in How do you find duplicates from Windows SMB shares using Linux:

@Dashrender said in How do you find duplicates from Windows SMB shares using Linux:

I wonder if this would run faster directly on the server in powershell instead? I'm assuming with doing this over SMB you have to download all files, run the hash - if ran locally, you get to skip the download time, I assume.

I gathered that the SMB shares are hosted on Linux, but I could be wrong.

If they are hosted on Windows like you are assuming, then I would agree that PowerShell would probably be most performant for this.

The title says - Windows SMB Shares.

My guess is that Dustin is a lone wolf running a 'nix OS as his machine - and the rest of the company is using Windows. Nothing wrong with that, just my guess.

His company is significantly Mac.

aww, that's right - he has been asking a lot of MAC questions lately.

Unix questions to be more precise, but yeah we are a heavy Mac shop.