File Management removing unprintable characters

psophos

@dafyre You might want to throw in an if statement there to check that the names are different before you rename.
I'm assuming the ren will rename the file even if the names are the same, but maybe it won't. Some very light testing suggests it may (no errors thrown).

1337

NTFS is utf-16. All characters are allowed in filenames on NTFS, except reserved characters like
< (less than)
> (greater than)
: (colon)
" (double quote)
/ (forward slash)
\ (backslash)
| (vertical bar or pipe)
? (question mark)
* (asterisk)
and chr(0) to chr(31).

If the backup system can't handle all allowed characters in a filename, then that is the problem that needs to be fixed.

There is no such a thing as unprintable characters. Just need the right font that has that character defined.

This is screenshot from Windows showing valid file and folder names:

JaredBusch

@Pete-S said in File Management removing unprintable characters:

Unrelated to the OP....
Why do those kanji look familiar? Like seriously...

Obsolesce

@DustinB3403 said in File Management removing unprintable characters:

So long story short I have users who use unprintable characters in file and folder paths, such as  or the little floating dot.

Can anyone think of some quick way to replace all of these in every folder and sub folder and file with a normal hyphen?

There's some built-in cmdlets to do this pretty easily.

Building on @scottalanmiller's regex, I added the exclusion of punctuation characters, because in my testing, it was replacing the "dot" before the file extension. I did not go looking for a way to just exclude dots. Someone else can do that.

This line will get each item in a directory and subdirectories -Recurse, and replace any non-"your-language"alphabet character, ignoring regular aphabet/number/punctuation characters.

Here's how I'd go about it:


# Remove the -WhatIf when you are ready to make the changes.
(Get-ChildItem -Path "C:\test" -Recurse | Rename-Item -NewName {$_.Name -replace '[^\p{L}\p{Nd}\p{P}]','-'} -WhatIf)

Using the -WhatIf switch will allow the code to be ran while telling you exactly what will change, without actually doing it. Remove the -WhatIf when you are ready to make the changes.

DustinB3403

@Obsolesce That is absolutely the perfect answer.

DustinB3403

@Pete-S said in File Management removing unprintable characters:

If the backup system can't handle all allowed characters in a filename, then that is the problem that needs to be fixed.

While I would generally agree, I have no way to control the file system in my backup location, which complains about the files and just skips them.

So while I agree, use a better back tool that isn't the answer I'm looking for.

@Obsolesce this is great, thank you!

1337

@DustinB3403

A few things to think about:

Expect some support calls when you rename customer files and they can't open them using "recent files".
Also expect problems when you rename a file that someone has open. Normally you can't => script fail.
Also expect problem if someone makes a new file with the original name. Now there will be two files that will have the same name after the renaming process => script fail.
Also hope that there aren't any application files that will not pass the regex => application fail.

Is the backup something homemade or something very old perhaps?

I suggest spending some time finding out exactly what characters are supported and making sure the regex is exactly that and not anymore restrictive than needed.
Then make sure the renaming script can handle errors mentioned above.
And I suggest some log file or email sent with files that can't be renamed so you know.

DustinB3403

@Pete-S said in File Management removing unprintable characters:

Expect some support calls when you rename customer files and they can't open them using "recent files".

Not my concern in the least

@Pete-S said in File Management removing unprintable characters:

Also expect problems when you rename a file that someone has open. Normally you can't => script fail.

Only targeting files that are marked for "archive" - so not an issue

@Pete-S said in File Management removing unprintable characters:

Also expect problem if someone makes a new file with the original name. Now there will be two files that will have the same name after the renaming process => script fail.

We already punch people in the back of the head for this

@Pete-S said in File Management removing unprintable characters:

Also hope that there aren't any application files that will not pass the regex => application fail.

Nope

@Pete-S said in File Management removing unprintable characters:

Is the backup something homemade or something very old perhaps?

Using B2 CLI to move stuff, so not old at all.

Dashrender

This really sounds like an HR problem that you're kinda solving with tech. If it's not against company policy to use those characters in file names/paths - then what you wanting to do is likely the wrong approach... and instead management should be approving you to find a new backup solution that works with those filenames.

DustinB3403

@Dashrender said in File Management removing unprintable characters:

This really sounds like an HR problem that you're kinda solving with tech. If it's not against company policy to use those characters in file names/paths - then what you wanting to do is likely the wrong approach... and instead management should be approving you to find a new backup solution that works with those filenames.

There is no policy on this, not would there be one.

This is an insane way to think about that.

DustinB3403

Okay so with a bit more offline assistance from @Obsolesce

This is what works

(Get-ChildItem -Path "some\path" -Recurse | Rename-Item -NewName {$_.Name -replace '•','-'} -verbose)

Obviously it's only hitting on that bullet point, and replacing it with a hyphen, but that's better than the operation failing over and over and not ever knowing it.

Obsolesce

@DustinB3403 said in File Management removing unprintable characters:

Okay so with a bit more offline assistance from @Obsolesce

This is what works

(Get-ChildItem -Path "some\path" -Recurse | Rename-Item -NewName {$_.Name -replace '•','-'} -verbose)

Obviously it's only hitting on that bullet point, and replacing it with a hyphen, but that's better than the operation failing over and over and not ever knowing it.

If you want to know if there's an error, you can build that in like this:


(Get-ChildItem -Path "C:\test" -Recurse | Rename-Item -NewName {$_.Name -replace '[^\p{L}\p{Nd}\p{P}]','-'} -WhatIf -ErrorAction SilentlyContinue -ErrorVariable daError)
if ($daError) {
    Write-Output "ERROR - There was an error. Pay attention : [$daError]"
}

Dashrender

@DustinB3403 said in File Management removing unprintable characters:

@Dashrender said in File Management removing unprintable characters:

This really sounds like an HR problem that you're kinda solving with tech. If it's not against company policy to use those characters in file names/paths - then what you wanting to do is likely the wrong approach... and instead management should be approving you to find a new backup solution that works with those filenames.

There is no policy on this, not would there be one.

This is an insane way to think about that.

See, you wanting to change the way users work to suit IT is generally something @scottalanmiller would say is likely wrong for the business.

1337

@DustinB3403 said in File Management removing unprintable characters:

@Pete-S said in File Management removing unprintable characters:
Is the backup something homemade or something very old perhaps?

Using B2 CLI to move stuff, so not old at all.

You have to realize that there are multi-national companies everywhere. So when you have problems with filenames, it's extremely unlikely that something modern will not support whatever it is you are doing - as long as the filename is valid on the OS you are using. And Backblaze of course supports all characters.

https://www.backblaze.com/b2/docs/string_encoding.html

Maybe there is a bug somewhere or maybe you are using it the wrong way. But this is the actual problem, not that users use whatever filename they want.

scottalanmiller

@DustinB3403 said in File Management removing unprintable characters:

@Dashrender said in File Management removing unprintable characters:

This really sounds like an HR problem that you're kinda solving with tech. If it's not against company policy to use those characters in file names/paths - then what you wanting to do is likely the wrong approach... and instead management should be approving you to find a new backup solution that works with those filenames.

There is no policy on this, not would there be one.

This is an insane way to think about that.

Well, let's back up. Why are people using those characters? Chances are they are real characters. They are getting in there somehow, and not likely from someone trying to be weird, but likely from an alternative keyboard or something.

DustinB3403

@Pete-S Using a bullet point in a damn folder or file name is not at all normal!

FFS.

If you think a file or folder named

some crap

Is used globally, you're just wrong.

DustinB3403

@scottalanmiller said in File Management removing unprintable characters:

@DustinB3403 said in File Management removing unprintable characters:

@Dashrender said in File Management removing unprintable characters:

This really sounds like an HR problem that you're kinda solving with tech. If it's not against company policy to use those characters in file names/paths - then what you wanting to do is likely the wrong approach... and instead management should be approving you to find a new backup solution that works with those filenames.

There is no policy on this, not would there be one.

This is an insane way to think about that.

Well, let's back up. Why are people using those characters? Chances are they are real characters. They are getting in there somehow, and not likely from someone trying to be weird, but likely from an alternative keyboard or something.

The reasoning as far as I've seen is it puts the content at the top of the list as the main culprit excuse. One which I'd rather not try to "fix" since it's well beyond my ability to hit these fools hard enough.

scottalanmiller

But a bullet point is printable and I bet some languages use it somewhere. But maybe it came from some odd cut and paste procedure. But something is causing this to happen, I assume that this isn't all one person?

scottalanmiller

@DustinB3403 said in File Management removing unprintable characters:

@scottalanmiller said in File Management removing unprintable characters:

@DustinB3403 said in File Management removing unprintable characters:

@Dashrender said in File Management removing unprintable characters:

This really sounds like an HR problem that you're kinda solving with tech. If it's not against company policy to use those characters in file names/paths - then what you wanting to do is likely the wrong approach... and instead management should be approving you to find a new backup solution that works with those filenames.

There is no policy on this, not would there be one.

This is an insane way to think about that.

Well, let's back up. Why are people using those characters? Chances are they are real characters. They are getting in there somehow, and not likely from someone trying to be weird, but likely from an alternative keyboard or something.

The reasoning as far as I've seen is it puts the content at the top of the list as the main culprit excuse. One which I'd rather not try to "fix" since it's well beyond my ability to hit these fools hard enough.

Well that definitely isn't a good reason if they are trying to use special characters as indexing data.

DustinB3403

@scottalanmiller said in File Management removing unprintable characters:

I assume that this isn't all one person?

Our staff are pulled onto so many projects that I wouldn't even have an idea of who or when something like a copy and paste issue occurred.

Is it possible, absolutely.

Is it also happening far too often, absolutely and thus I'll fix it when the files are closed via powershell.

@Obsolesce thanks for assisting with this.

Solved File Management removing unprintable characters