@Dashrender said in Migrate and/or replace old cert server?:

@Shuey said in Migrate and/or replace old cert server?:

The ONLY "virtualization" infrastructure that was in place when I got here was a Hyper-V console (on the same server that I referenced in my original post in this thread; the server that also has SharePoint! This server used to also be a print server and a file server on top of everything else I've already mentioned).

So the Hyper-V console was there, but no VMs?

Nope, they had three guest VMs running on it (one was a print server, one was their accounting app server, and the third was their TV media server).

@scottalanmiller said in Migrate and/or replace old cert server?:

@Shuey said in Migrate and/or replace old cert server?:

@Dashrender said in Migrate and/or replace old cert server?:

@scottalanmiller said in Migrate and/or replace old cert server?:

@Dashrender said in Migrate and/or replace old cert server?:

@scottalanmiller said in Migrate and/or replace old cert server?:

@Shuey said in Migrate and/or replace old cert server?:

First let me say that I know nothing about certificate services, IIS or SQL (all three of which are currently configured and running on this server).

Why are those together? That's not generally a best practice. I realize that Windows licensing causes some decisions that would otherwise be poor, but this seems an odd combination.

I'm betting it's mainly because the company didn't want to buy 2-3 physical servers. If they would have gone virtualized back then, they might be on different OSEs.

Right.... so assuming one bad decision leading to another.

I know you've been using virtualization since the day VMWare rolled out their first internal only beta (yes I'm kidding), but I don't feel that the SMB really started using virtualization until 2010 or later. It's likely whoever setup this server was unfamiliar with virtualization and they were working with what they knew.

I guess you could say that the bad decision was that the business had a one man/very small IT internal staff. If they had a good MSP or consulting business partner, they might have have gone another route.

The ONLY "virtualization" infrastructure that was in place when I got here was a Hyper-V console (on the same server that I referenced in my original post in this thread; the server that also has SharePoint! This server used to also be a print server and a file server on top of everything else I've already mentioned).

I deployed the VMware infrastructure about a year or so after I started working here.

Assuming that the servers were commodity and post 2005, that means that someone was slacking. Why was Hyper-V console installed but nothing else? That's weird. Did you ever figure out why?

It wasn't "Hyper-V and nothing else". It was a "DC, SharePoint, File Server, Cert Server, AND a Hyper-V host"!

@Dashrender said in Migrate and/or replace old cert server?:

@scottalanmiller said in Migrate and/or replace old cert server?:

@Dashrender said in Migrate and/or replace old cert server?:

@scottalanmiller said in Migrate and/or replace old cert server?:

@Shuey said in Migrate and/or replace old cert server?:

First let me say that I know nothing about certificate services, IIS or SQL (all three of which are currently configured and running on this server).

Why are those together? That's not generally a best practice. I realize that Windows licensing causes some decisions that would otherwise be poor, but this seems an odd combination.

I'm betting it's mainly because the company didn't want to buy 2-3 physical servers. If they would have gone virtualized back then, they might be on different OSEs.

Right.... so assuming one bad decision leading to another.

I know you've been using virtualization since the day VMWare rolled out their first internal only beta (yes I'm kidding), but I don't feel that the SMB really started using virtualization until 2010 or later. It's likely whoever setup this server was unfamiliar with virtualization and they were working with what they knew.

I guess you could say that the bad decision was that the business had a one man/very small IT internal staff. If they had a good MSP or consulting business partner, they might have have gone another route.

The ONLY "virtualization" infrastructure that was in place when I got here was a Hyper-V console (on the same server that I referenced in my original post in this thread; the server that also has SharePoint! This server used to also be a print server and a file server on top of everything else I've already mentioned).

I deployed the VMware infrastructure about a year or so after I started working here.

@Dashrender said in Migrate and/or replace old cert server?:

Now you need to see what certs you're using for SharePoint. If you're using a public cert, then it sounds like you're right.

what did you replace your Wireless RADIUS setup with?

We use local logins on all the of the equipment that used to authenticate via radius (switches mostly), and as far as wireless goes, we don't allow any workstations to connect to our domain via wireless; they are only allowed to connect to a public SSID/subnet.

@Dashrender said in Migrate and/or replace old cert server?:

@Mike-Davis said in Migrate and/or replace old cert server?:

@scottalanmiller said in Migrate and/or replace old cert server?:

@Shuey said in Migrate and/or replace old cert server?:

Is it common for every business/company that has a domain network to have a cert server for issuing/updating all of the AD account certificates?

Maybe I've lost my mind but... what is an "AD Account Certificate"?

You can integrate AD with certificate services so that the workstations use the certs for communication. I've never seen it done.

The only time I have used certificate services is to generate certificates for securing communication between Wireless APs and company owned devices.

While I haven't seen it, I've read about it in NPS (Network Policy Server setups). The machine comes on the network, checks in with the NPS, and the NPS determines what VLAN it should be on, etc, etc.

@wrx7m said in Migrate and/or replace old cert server?:

I am using AD cert services for RADIUS authentication of wireless client devices and users.

This makes more sense now! They USED to do radius authentication, as well as wireless authentication via the cert server. Since we no longer use either, it sounds like I might be safe to completely skip this project all together, and move on to the SharePoint project. What do you guys think?

@scottalanmiller said in Migrate and/or replace old cert server?:

@Shuey said in Migrate and/or replace old cert server?:

@scottalanmiller said in Migrate and/or replace old cert server?:

@Shuey and then there is the other issue... why there a SAN?

For our PACS vendor and their equipment.

That alone wouldn't qualify as a reason.

It doesn't appear that the cert services role on this server is communicating at all with our PACS servers (which we have no access rights to - our vendor only has access).

@scottalanmiller said in Migrate and/or replace old cert server?:

@Shuey and then there is the other issue... why there a SAN?

For our PACS vendor and their equipment.

@scottalanmiller said in Migrate and/or replace old cert server?:

@Shuey said in Migrate and/or replace old cert server?:

From what I understand (which is not much, lol), this server is what every workstation and user account on the domain gets its certificate from.

Which certificates would those be?

When I look at the Certification Authority console on the server, and I look at "issued certificates", I see line items like this:
"Request ID", "Requester Name", "Certificate Template", "Certificate Effective Date", "Certificate Expiration Date", etc, and I see a bunch of workstations listed.

@scottalanmiller said in Migrate and/or replace old cert server?:

@Shuey said in Migrate and/or replace old cert server?:

First let me say that I know nothing about certificate services, IIS or SQL (all three of which are currently configured and running on this server).

Why are those together? That's not generally a best practice. I realize that Windows licensing causes some decisions that would otherwise be poor, but this seems an odd combination.

Your guess is as good as mine, lol. I know it's not a good business practice, but "bad business practices" at my company are kinda like cereal and milk; they have always gone together for as long as I've known. Here's a great example reference: We have two main datacenters, which my boss refers to as "the cold room" (LOL). One of the datacenters is shared with a janitor's closet, and there's no lock on the door! Yep, literally hundreds of thousands of dollars worth of equipment that anyone in the entire building could access without restriction (one of the big dollar items in this "cold room" is an EMC SAN!!). Despite the fact that I've told my boss and upper management that this is crazy, they have done nothing to change it. Another example: The datacenter at one of our other sites has a crazy ghetto "cooling system" (if that's what you wanna call it). Prior to getting an air conditioner installed in this server room, the way they used to cool it was to open the server room door and put several floor fans in their blowing the hot air out (and that's STILL what they do when the air conditioner dies!) - and this "cold room" also has an EMC SAN!!! O_o

Thanks Mike! You raise a good question: "Why do I need a cert server?" or "What is this server role/feature currently facilitating?".

From what I understand (which is not much, lol), this server is what every workstation and user account on the domain gets its certificate from. But I'm not sure why exactly (other than the assumption of a layer of security when accounts communicate with the server(s)).

Is it common for every business/company that has a domain network to have a cert server for issuing/updating all of the AD account certificates?

I've been administering a network that I took over about 4 years ago, and I'm at another hurdle that I need to figure out. We have a single Windows 2008 R2 server that currently acts as a DC, our cert server, and our SharePoint server (which has outside access configured so our staff can access it from home to check schedules for instance).

First let me say that I know nothing about certificate services, IIS or SQL (all three of which are currently configured and running on this server).

1. What I'd like to do is setup a new cert server (one that can either "take over" by simply becoming the "new cert server", or by migrating the info from the old server to the new / and I currently have no idea which of those scenarios is possible or best practice).

2. I'd also like to demote this server so it's no longer a DC (but from what I've read online so far, if a DC is also a cert server, demoting it can cause issues if the cert services aren't migrated or removed first).

3. Finally, I'd like to eventually build a new replacement SharePoint server, BUT, in the meantime, I was considering doing a P2V of what's left of this server (essentially just an old SharePoint server) that we can host in our current VMware infrastructure, and then I can re-purpose the hardware that it used to be hosted on. Replacing SharePoint will be another project all on its own, especially since I know nothing about SharePoint installation, configuration, etc.

I'm really excited to take on these projects! I just need to overcome some of my initial fears of what seem like scary projects since I'm so green in these areas. My initial research has been overwhelming, so I'm trying to take my time and tread carefully.

Any advice or direction is greatly appreciated!

Thanks again to everyone who replied and gave feedback on this. It's great to know that there's a solid community of knowledgeable people who are willing to share their expertise - I really appreciate it!!

@scottalanmiller said in Invalid Drive Movement from HP SmartArray P411 RAID Controller with StorageWorks MSA60:

Next step... get local drives and decom that MSA60. It just sent a shot across your bow and has exposed how dangerous and precarious it is. Don't fail to heed its warning.

Absolutely Scott! I'm gonna be talking more with my boss about this as soon as possible!

You guys are not going to believe this...

First I attempted a fresh cold boot of the existing MSA, waited a couple minutes, then powered up the ESXi host, but the issue remained. I then shutdown the host and MSA, moved the drives into our spare MSA, powered it up, waited a couple minutes, then powered up the ESXi host; the issue still remained.

At that point, I figured I was pretty much screwed, and there was nothing during the initialization of the RAID controller where I had an option to re-enable a failed logical drive. So I booted into the RAID config, verified again that there were no logical drives present, and I created a new logical drive (RAID 1+0 with two spare drives; same as we did about 2 years ago when we first setup this host and storage).

Then I let the server boot back into vSphere and I accessed it via vCenter. The first thing I did was removed the host from inventory, then re-added it (I was hoping to clear all the inaccessible guest VMs this way, but it didn't clear them from the inventory). Once the host was back in my inventory, I removed each of the guest VMs one at a time. Once the inventory was cleared, I verified that no datastore existed and that the disks were basically ready and waiting as "data disks". So I went ahead and created a new datastore (again, same as we did a couple years ago, using VMFS). I was eventually prompted to specify a mount option and I had the option of "keep the existing signature". At this point, I figured it'd be worth a shot to keep the signature - if things didn't work out, I could always blow it away and re-create the datastore again. After I finished the process of building the datastore with the keep signature option, I tried navigating to the datastore to see if anything was in it - it appeared empty. Just out of curiosity, I SSH'd to the host and checked from there, and to my surprise, I could see all my old data and all my old guest VMs! I went back into vCenter and re-scanned storage and refreshed the console, and all of our old guest VMs were there! I re-registered each VM and was able to recover everything! All of our guest VMs are back up and successfully communicating on the network.

I think most people in the IT community would agree that the chances of having something like this happen are extremely low to impossible.

As far as I'm concerned, this was a miracle of God...

@scottalanmiller said in "Invalid Drive Movement" (HP Smart Array P411):

@Shuey said in "Invalid Drive Movement" (HP Smart Array P411):

We don't have a support contract on this server or the attached MSA, and they're likely way out of warranty (ProLiant DL360 G8 and a StorageWorks MSA60), so I'm not sure how much we'd have to spend in order to get HP to "help" us :-S...

A bit. Why is there an MSA out of contract? The only benefit to an MSA is the support contract. Not that that makes it worth it, but proprietary storage requires a warranty contract to be viable. The rule is that any storage of that nature needs to be decommissioned the day before the support contract runs out because there isn't necessary any path to recovery in the event of an "incident" without one. It's not a standard server that you can just fix yourself with third party parts. Sometimes you can, but as it is a closed, proprietary system, you are generally totally dependent on your support contract from the vendor to keep it working.

There is a good chance that this is a "replace the MSA and restore from backup" situation in that case.

Unfortunately, my company's philosophy on "investing in IT infrastructure" goes like this: "We'll spend hundreds to thousands of dollars every time our PACS vendor tells us they need it. Then, when they say that they need to upgrade their equipment, we'll re-purpose their old stuff for the rest of our production environment (because we don't understand the importance of spending money on the rest of our infrastructure, and we don't trust the knowledgeable people we hired in our IT department)"

@scottalanmiller said in "Invalid Drive Movement" (HP Smart Array P411):

@Shuey said in "Invalid Drive Movement" (HP Smart Array P411):

@scottalanmiller said in "Invalid Drive Movement" (HP Smart Array P411):

@Shuey said in "Invalid Drive Movement" (HP Smart Array P411):

I actually rebooted this server multiple times about a month ago when I installed updates on it. The reboots went fine. We also completely powered that server down at around the same time because I added more RAM to it. Again, after powering everything back on, the server and raid array information was all intact.

Does your normal reboot schedule of your server include a reboot of the MSA? Could it be that they were powered back on in the incorrect order? MSAs are notoriously flaky, likely that is where the issue is.

I'd call HPE support. The MSA is a flaky unit but HPE support is quite good.

We unfortunately don't have a "normal reboot schedule" of ANY for our servers :-/...

I should not have said schedule. I should have said your "Normal reboot process." Regardless of the regularity of the reboots, is the process a standard one?

I'm not sure we have a "standard"... we only reboot this particular ESXi host when absolutely necessary, and this weekend is possibly the first time we've rebooted the MSA in a year or more :-S...

@scottalanmiller said in "Invalid Drive Movement" (HP Smart Array P411):

@Shuey said in "Invalid Drive Movement" (HP Smart Array P411):

I actually rebooted this server multiple times about a month ago when I installed updates on it. The reboots went fine. We also completely powered that server down at around the same time because I added more RAM to it. Again, after powering everything back on, the server and raid array information was all intact.

Does your normal reboot schedule of your server include a reboot of the MSA? Could it be that they were powered back on in the incorrect order? MSAs are notoriously flaky, likely that is where the issue is.

I'd call HPE support. The MSA is a flaky unit but HPE support is quite good.

We unfortunately don't have a "normal reboot schedule" for ANY of our servers :-/...

I'm not even sure what the correct order is :-S... I would assume that the MSA would get powered on first, then the ESXi host. If this is correct, we have already tried doing that since we first discovered this issue today, and the issue remains :(.

We don't have a support contract on this server or the attached MSA, and they're likely way out of warranty (ProLiant DL360 G8 and a StorageWorks MSA60), so I'm not sure how much we'd have to spend in order to get HP to "help" us :-S...

@travisdh1 said in "Invalid Drive Movement" (HP Smart Array P411):

Any number of things. Do you schedule reboots on all your equipment? If not you really should for just this reason. The one server we have, XS decided the array wasn't ready in time and didn't mount the main storage volume on boot. Always nice to know these things ahead of time, right?

I actually rebooted this server multiple times about a month ago when I installed updates on it. The reboots went fine. We also completely powered that server down at around the same time because I added more RAM to it. Again, after powering everything back on, the server and raid array information was all intact.

Due to hurricane Matthew, our company shutdown all servers for two days.  One of the servers was an ESXi host with an attached HP StorageWorks MSA60.

When we logged into the vSphere client, we noticed that none of our guest VMs are available (they're all listed as "inaccessible").  And when I look at the hardware status in vSphere, the array controller and all attached drives appear as "Normal", but the drives all show up as "unconfigured disk".

We rebooted the server and tried going into the RAID config utility to see what things look like from there, but we received the following message:

"An invalid drive movement was reported during POST. Modifications to the array configuration following an invalid drive movement will result in loss of old configuration information and contents of the original logical drives".

Needless to say, we're very confused by this because nothing was "moved"; nothing changed.  We simply powered up the MSA and the server, and have been having this issue ever since.

I have two main questions/concerns:

1. Since we did nothing more than power the devices off and back on, what could've caused this to happen?  I of course have the option to rebuild the array and start over, but I'm leery about the possibility of this happening again (especially since I have no idea what caused it).

2. Is there a snowball's chance in hell that I can recover our array and guest VMs, instead of having to rebuild everything and restore our VM backups?

So my question, I think, is how do we:

• Catalogue the knowledge needed for IT? I know of this done, literally, nowhere.
• Create a path or curriculum for learning these things?
• Create a means to present comprehensive basic, foundational IT knowledge?

Very valid points and questions, but in today's "IT age", even if we could catalog and/or collect all of this info into one place, it could still take years to review and learn from it, especially for "beginners". IT has become so vast and diverse that it's impossible for a true "renaissance man" to exist; nobody can "know it all".

It's unfortunate that someone with SAM's experience level and understanding could ever be seen as "stupid" for not knowing one little thing in a sea of information, but in general, that's one of the coolest things about IT: because it's so vast, "masters" always have an opportunity to learn things from beginners.