Exchange 2010 Hard Drive Space Issues

[scottalanmiller]

@wirestyle22 said in Exchange 2010 Hard Drive Space Issues:

@scottalanmiller said in Exchange 2010 Hard Drive Space Issues:

@wirestyle22 said in Exchange 2010 Hard Drive Space Issues:

@JaredBusch said in Exchange 2010 Hard Drive Space Issues:

@wirestyle22 said in Exchange 2010 Hard Drive Space Issues:

My thought is I can dismount my datastore, change the name of the log folder from Mailbox - Domain to Mailbox - Domain.old and then create a new folder named Mailbox - Domain. If the logs populate correctly I should be able to delete the old logs and do a fast full backup with the logs truncating properly.

Note: I just did this and it seems to be generating logs properly.

This was horribly risky, but is something I have done before too. I would certainly have tried other means first, but you seem to have some really odd constraints.

I'm talking to Microsoft right now and they basically just told me I have two options in this situation, after reading my logs. 1. Turn on circular logging which will increase my CPU utilization significantly (this server sits at 70%+ utilization consistently) which would most likely seize it or do what I did. I considered each option and spoke to my boss. We are running out of space rapidly and he said at this point we unfortunately are going to have to take the risk. This realistically should have been fixed a year ago, but from what I understand they were running around putting out fires.

The logs are going to the same box? Why not send logs to a central service?

The only exchange server I've personally managed was a standalone box in which the logs were stored locally, but this is an inherited server (all of them are). I question a lot of their logic honestly

You can always start shipping logs "now". Just because it wasn't done in the past doesn't mean you can't do it in the future.

[scottalanmiller]

@Dashrender said in Exchange 2010 Hard Drive Space Issues:

@scottalanmiller said in Exchange 2010 Hard Drive Space Issues:

@wirestyle22 said in Exchange 2010 Hard Drive Space Issues:

@JaredBusch said in Exchange 2010 Hard Drive Space Issues:

@wirestyle22 said in Exchange 2010 Hard Drive Space Issues:

My thought is I can dismount my datastore, change the name of the log folder from Mailbox - Domain to Mailbox - Domain.old and then create a new folder named Mailbox - Domain. If the logs populate correctly I should be able to delete the old logs and do a fast full backup with the logs truncating properly.

Note: I just did this and it seems to be generating logs properly.

This was horribly risky, but is something I have done before too. I would certainly have tried other means first, but you seem to have some really odd constraints.

I'm talking to Microsoft right now and they basically just told me I have two options in this situation, after reading my logs. 1. Turn on circular logging which will increase my CPU utilization significantly (this server sits at 70%+ utilization consistently) which would most likely seize it or do what I did. I considered each option and spoke to my boss. We are running out of space rapidly and he said at this point we unfortunately are going to have to take the risk. This realistically should have been fixed a year ago, but from what I understand they were running around putting out fires.

The logs are going to the same box? Why not send logs to a central service?

Does exchange logging work that way?

https://www.splunk.com/en_us/solutions/solution-areas/it-operations-management/microsoft-infrastructure-monitoring/splunk-app-for-microsoft-exchange.html

[Dashrender]

@scottalanmiller said in Exchange 2010 Hard Drive Space Issues:

@wirestyle22 said in Exchange 2010 Hard Drive Space Issues:

@scottalanmiller said in Exchange 2010 Hard Drive Space Issues:

@wirestyle22 said in Exchange 2010 Hard Drive Space Issues:

@JaredBusch said in Exchange 2010 Hard Drive Space Issues:

@wirestyle22 said in Exchange 2010 Hard Drive Space Issues:

My thought is I can dismount my datastore, change the name of the log folder from Mailbox - Domain to Mailbox - Domain.old and then create a new folder named Mailbox - Domain. If the logs populate correctly I should be able to delete the old logs and do a fast full backup with the logs truncating properly.

Note: I just did this and it seems to be generating logs properly.

This was horribly risky, but is something I have done before too. I would certainly have tried other means first, but you seem to have some really odd constraints.

I'm talking to Microsoft right now and they basically just told me I have two options in this situation, after reading my logs. 1. Turn on circular logging which will increase my CPU utilization significantly (this server sits at 70%+ utilization consistently) which would most likely seize it or do what I did. I considered each option and spoke to my boss. We are running out of space rapidly and he said at this point we unfortunately are going to have to take the risk. This realistically should have been fixed a year ago, but from what I understand they were running around putting out fires.

The logs are going to the same box? Why not send logs to a central service?

The only exchange server I've personally managed was a standalone box in which the logs were stored locally, but this is an inherited server (all of them are). I question a lot of their logic honestly

You can always start shipping logs "now". Just because it wasn't done in the past doesn't mean you can't do it in the future.

How does Log cleanup work in this situation with backups?
Heck, how do you replay those logs in a restore/repair situation?

[wirestyle22]

@Dashrender said in Exchange 2010 Hard Drive Space Issues:

@scottalanmiller said in Exchange 2010 Hard Drive Space Issues:

@wirestyle22 said in Exchange 2010 Hard Drive Space Issues:

@scottalanmiller said in Exchange 2010 Hard Drive Space Issues:

@wirestyle22 said in Exchange 2010 Hard Drive Space Issues:

@JaredBusch said in Exchange 2010 Hard Drive Space Issues:

@wirestyle22 said in Exchange 2010 Hard Drive Space Issues:

My thought is I can dismount my datastore, change the name of the log folder from Mailbox - Domain to Mailbox - Domain.old and then create a new folder named Mailbox - Domain. If the logs populate correctly I should be able to delete the old logs and do a fast full backup with the logs truncating properly.

Note: I just did this and it seems to be generating logs properly.

This was horribly risky, but is something I have done before too. I would certainly have tried other means first, but you seem to have some really odd constraints.

I'm talking to Microsoft right now and they basically just told me I have two options in this situation, after reading my logs. 1. Turn on circular logging which will increase my CPU utilization significantly (this server sits at 70%+ utilization consistently) which would most likely seize it or do what I did. I considered each option and spoke to my boss. We are running out of space rapidly and he said at this point we unfortunately are going to have to take the risk. This realistically should have been fixed a year ago, but from what I understand they were running around putting out fires.

The logs are going to the same box? Why not send logs to a central service?

The only exchange server I've personally managed was a standalone box in which the logs were stored locally, but this is an inherited server (all of them are). I question a lot of their logic honestly

You can always start shipping logs "now". Just because it wasn't done in the past doesn't mean you can't do it in the future.

How does Log cleanup work in this situation with backups?

I'd imagine you just need to run a custom backup of the log locations on that specific server instead of exchange

[Dashrender]

@scottalanmiller said in Exchange 2010 Hard Drive Space Issues:

@Dashrender said in Exchange 2010 Hard Drive Space Issues:

@scottalanmiller said in Exchange 2010 Hard Drive Space Issues:

@wirestyle22 said in Exchange 2010 Hard Drive Space Issues:

@JaredBusch said in Exchange 2010 Hard Drive Space Issues:

@wirestyle22 said in Exchange 2010 Hard Drive Space Issues:

My thought is I can dismount my datastore, change the name of the log folder from Mailbox - Domain to Mailbox - Domain.old and then create a new folder named Mailbox - Domain. If the logs populate correctly I should be able to delete the old logs and do a fast full backup with the logs truncating properly.

Note: I just did this and it seems to be generating logs properly.

This was horribly risky, but is something I have done before too. I would certainly have tried other means first, but you seem to have some really odd constraints.

I'm talking to Microsoft right now and they basically just told me I have two options in this situation, after reading my logs. 1. Turn on circular logging which will increase my CPU utilization significantly (this server sits at 70%+ utilization consistently) which would most likely seize it or do what I did. I considered each option and spoke to my boss. We are running out of space rapidly and he said at this point we unfortunately are going to have to take the risk. This realistically should have been fixed a year ago, but from what I understand they were running around putting out fires.

The logs are going to the same box? Why not send logs to a central service?

Does exchange logging work that way?

https://www.splunk.com/en_us/solutions/solution-areas/it-operations-management/microsoft-infrastructure-monitoring/splunk-app-for-microsoft-exchange.html

OK makes sense for tracking, etc... but what about recovery?

[scottalanmiller]

@Dashrender said in Exchange 2010 Hard Drive Space Issues:

@scottalanmiller said in Exchange 2010 Hard Drive Space Issues:

@Dashrender said in Exchange 2010 Hard Drive Space Issues:

@scottalanmiller said in Exchange 2010 Hard Drive Space Issues:

@wirestyle22 said in Exchange 2010 Hard Drive Space Issues:

@JaredBusch said in Exchange 2010 Hard Drive Space Issues:

@wirestyle22 said in Exchange 2010 Hard Drive Space Issues:

My thought is I can dismount my datastore, change the name of the log folder from Mailbox - Domain to Mailbox - Domain.old and then create a new folder named Mailbox - Domain. If the logs populate correctly I should be able to delete the old logs and do a fast full backup with the logs truncating properly.

Note: I just did this and it seems to be generating logs properly.

This was horribly risky, but is something I have done before too. I would certainly have tried other means first, but you seem to have some really odd constraints.

I'm talking to Microsoft right now and they basically just told me I have two options in this situation, after reading my logs. 1. Turn on circular logging which will increase my CPU utilization significantly (this server sits at 70%+ utilization consistently) which would most likely seize it or do what I did. I considered each option and spoke to my boss. We are running out of space rapidly and he said at this point we unfortunately are going to have to take the risk. This realistically should have been fixed a year ago, but from what I understand they were running around putting out fires.

The logs are going to the same box? Why not send logs to a central service?

Does exchange logging work that way?

https://www.splunk.com/en_us/solutions/solution-areas/it-operations-management/microsoft-infrastructure-monitoring/splunk-app-for-microsoft-exchange.html

OK makes sense for tracking, etc... but what about recovery?

Recovery of the logs, or recovery of email?

[JaredBusch]

@scottalanmiller said in Exchange 2010 Hard Drive Space Issues:

The logs are going to the same box? Why not send logs to a central service?

WTF? This is not how Exchange is designed. These are DB rollback logs. Not usage logs. They do not get shipped out for monitoring.

[Dashrender]

@scottalanmiller said in Exchange 2010 Hard Drive Space Issues:

@Dashrender said in Exchange 2010 Hard Drive Space Issues:

@scottalanmiller said in Exchange 2010 Hard Drive Space Issues:

@Dashrender said in Exchange 2010 Hard Drive Space Issues:

@scottalanmiller said in Exchange 2010 Hard Drive Space Issues:

@wirestyle22 said in Exchange 2010 Hard Drive Space Issues:

@JaredBusch said in Exchange 2010 Hard Drive Space Issues:

@wirestyle22 said in Exchange 2010 Hard Drive Space Issues:

My thought is I can dismount my datastore, change the name of the log folder from Mailbox - Domain to Mailbox - Domain.old and then create a new folder named Mailbox - Domain. If the logs populate correctly I should be able to delete the old logs and do a fast full backup with the logs truncating properly.

Note: I just did this and it seems to be generating logs properly.

This was horribly risky, but is something I have done before too. I would certainly have tried other means first, but you seem to have some really odd constraints.

I'm talking to Microsoft right now and they basically just told me I have two options in this situation, after reading my logs. 1. Turn on circular logging which will increase my CPU utilization significantly (this server sits at 70%+ utilization consistently) which would most likely seize it or do what I did. I considered each option and spoke to my boss. We are running out of space rapidly and he said at this point we unfortunately are going to have to take the risk. This realistically should have been fixed a year ago, but from what I understand they were running around putting out fires.

The logs are going to the same box? Why not send logs to a central service?

Does exchange logging work that way?

https://www.splunk.com/en_us/solutions/solution-areas/it-operations-management/microsoft-infrastructure-monitoring/splunk-app-for-microsoft-exchange.html

OK makes sense for tracking, etc... but what about recovery?

Recovery of the logs, or recovery of email?

recovery of email in case of a failure - it's my understanding that you restore you IS, then replay the logs to get the newer data back.

[scottalanmiller]

@JaredBusch said in Exchange 2010 Hard Drive Space Issues:

@scottalanmiller said in Exchange 2010 Hard Drive Space Issues:

The logs are going to the same box? Why not send logs to a central service?

WTF? This is not how Exchange is designed. These are DB rollback logs. Not usage logs. They do not get shipped out for monitoring.

Oh, sorry, didn't realize what logs we were discussing.

[scottalanmiller]

@Dashrender said in Exchange 2010 Hard Drive Space Issues:

@scottalanmiller said in Exchange 2010 Hard Drive Space Issues:

@Dashrender said in Exchange 2010 Hard Drive Space Issues:

@scottalanmiller said in Exchange 2010 Hard Drive Space Issues:

@Dashrender said in Exchange 2010 Hard Drive Space Issues:

@scottalanmiller said in Exchange 2010 Hard Drive Space Issues:

@wirestyle22 said in Exchange 2010 Hard Drive Space Issues:

@JaredBusch said in Exchange 2010 Hard Drive Space Issues:

@wirestyle22 said in Exchange 2010 Hard Drive Space Issues:

My thought is I can dismount my datastore, change the name of the log folder from Mailbox - Domain to Mailbox - Domain.old and then create a new folder named Mailbox - Domain. If the logs populate correctly I should be able to delete the old logs and do a fast full backup with the logs truncating properly.

Note: I just did this and it seems to be generating logs properly.

This was horribly risky, but is something I have done before too. I would certainly have tried other means first, but you seem to have some really odd constraints.

I'm talking to Microsoft right now and they basically just told me I have two options in this situation, after reading my logs. 1. Turn on circular logging which will increase my CPU utilization significantly (this server sits at 70%+ utilization consistently) which would most likely seize it or do what I did. I considered each option and spoke to my boss. We are running out of space rapidly and he said at this point we unfortunately are going to have to take the risk. This realistically should have been fixed a year ago, but from what I understand they were running around putting out fires.

The logs are going to the same box? Why not send logs to a central service?

Does exchange logging work that way?

https://www.splunk.com/en_us/solutions/solution-areas/it-operations-management/microsoft-infrastructure-monitoring/splunk-app-for-microsoft-exchange.html

OK makes sense for tracking, etc... but what about recovery?

Recovery of the logs, or recovery of email?

recovery of email in case of a failure - it's my understanding that you restore you IS, then replay the logs to get the newer data back.

Right, now I see, these are the DB logs, not the application logs.

[wirestyle22]

@JaredBusch Can you explain some of the risks associated with what I did just for my own knowledge? I did verify clean shutdown in powershell.

[JaredBusch]

@wirestyle22 you could have potentially been unable to mount the database again.

[momurda]

@JaredBusch said in Exchange 2010 Hard Drive Space Issues:

@wirestyle22 said in Exchange 2010 Hard Drive Space Issues:

My thought is I can dismount my datastore, change the name of the log folder from Mailbox - Domain to Mailbox - Domain.old and then create a new folder named Mailbox - Domain. If the logs populate correctly I should be able to delete the old logs and do a fast full backup with the logs truncating properly.

Note: I just did this and it seems to be generating logs properly.

This was horribly risky, but is something I have done before too. I would certainly have tried other means first, but you seem to have some really odd constraints.

I am confused here, how did this solve the problem? Youre just cutting out all those old log files and starting over? Since when can you do that in Exchange? Last time i did that(years ago) very bad things happened, like total loss of mail db. If these log files arent actually needed for Exchange to serve mail wtf are they for?

[JaredBusch]

@momurda said in Exchange 2010 Hard Drive Space Issues:

@JaredBusch said in Exchange 2010 Hard Drive Space Issues:

@wirestyle22 said in Exchange 2010 Hard Drive Space Issues:

My thought is I can dismount my datastore, change the name of the log folder from Mailbox - Domain to Mailbox - Domain.old and then create a new folder named Mailbox - Domain. If the logs populate correctly I should be able to delete the old logs and do a fast full backup with the logs truncating properly.

Note: I just did this and it seems to be generating logs properly.

This was horribly risky, but is something I have done before too. I would certainly have tried other means first, but you seem to have some really odd constraints.

I am confused here, how did this solve the problem? Youre just cutting out all those old log files and starting over? Since when can you do that in Exchange? Last time i did that(years ago) very bad things happened, like total loss of mail db. If these log files arent actually needed for Exchange to serve mail wtf are they for?

You have always been able to do that in exchange. These log files are not needed once the data in the log is wrote to the Exchange DB.

Exchange writes first to the log file and then to the DB.

In theory, the files could even be deleted with everything running because they are no longer needed once the DB is updated.

The system is designed to handle it all behind the scenes though. and not with manual interaction.

Their problem was never using the correct backup method that would cause the system to auto remove the files.

[wirestyle22]

@JaredBusch said in Exchange 2010 Hard Drive Space Issues:

In theory, the files could even be deleted with everything running because they are no longer needed once the DB is updated.

Everything up to the most recent log file as that would be in use

[jt1001001]

Now that you got it at least cleaned up, have you run a new backup yet?

[wirestyle22]

@jt1001001 said in Exchange 2010 Hard Drive Space Issues:

Now that you got it at least cleaned up, have you run a new backup yet?

I'm running it

[JaredBusch]

@wirestyle22 said in Exchange 2010 Hard Drive Space Issues:

@JaredBusch said in Exchange 2010 Hard Drive Space Issues:

In theory, the files could even be deleted with everything running because they are no longer needed once the DB is updated.

Everything up to the most recent log file as that would be in use

Correct.

[wirestyle22]

Thanks for all of your help guys. I really appreciate it

[Dashrender]

well, since you have the IS back online and the users aren't screaming.. sounds like you found a solution and things are looking good.

I'd still run a chkdsk on the system when you can take it all offline (I think that is what was wrong with your previous backups). Don't want to leave a possible funky disk out there if you can help it.

Let us know when the backups are running again, and trunking logs.