httpd dead but pid file exists

stacksofplates

@scottalanmiller said in httpd dead but pid file exists:

Do this...
systemctl restart httpd ; tail -f /var/log/messages
And collect about a screen's worth. Let's see if anything is reported immediately following the start up.

It's RHEL 5

scottalanmiller

@stacksofplates said in httpd dead but pid file exists:

It's RHEL 5

Ouch

wirestyle22

host (root) php -v
PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/lib/php/modules/oci8.so' - libclntsh.so.11.1: cannot open shared object file: No such file or directory in Unknown on line 0
PHP 5.1.6 (cli) (built: May 22 2018 10:25:39)
Copyright (c) 1997-2006 The PHP Group
Zend Engine v2.1.0, Copyright (c) 1998-2006 Zend Technologies
host (root)[ /home/root ] ldd /usr/lib/php/modules/oci8.so
        linux-gate.so.1 =>  (0x0087d000)
        libclntsh.so.11.1 => not found
        libc.so.6 => /lib/libc.so.6 (0x00960000)
        /lib/ld-linux.so.2 (0x00567000)

libclntsh.so.11.1 => not found

wat

stacksofplates

@wirestyle22 said in httpd dead but pid file exists:

host (root) php -v
PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/lib/php/modules/oci8.so' - libclntsh.so.11.1: cannot open shared object file: No such file or directory in Unknown on line 0
PHP 5.1.6 (cli) (built: May 22 2018 10:25:39)
Copyright (c) 1997-2006 The PHP Group
Zend Engine v2.1.0, Copyright (c) 1998-2006 Zend Technologies
host (root)[ /home/root ] ldd /usr/lib/php/modules/oci8.so
        linux-gate.so.1 =>  (0x0087d000)
        libclntsh.so.11.1 => not found
        libc.so.6 => /lib/libc.so.6 (0x00960000)
        /lib/ld-linux.so.2 (0x00567000)

libclntsh.so.11.1 => not found wat

that was in your error log before:

@wirestyle22 said in httpd dead but pid file exists:

I am seeing a new entry in the logs:

PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/lib/php/modules/oci8.so' - libclntsh.so.11.1: cannot open shared object file: No such file or directory in Unknown on line 0

Did you kill the pid and remove the pid file before restarting it?

wirestyle22

@stacksofplates said in httpd dead but pid file exists:

Did you kill the pid and remove the pid file before restarting it?

Yes I did. It just re-creates the file and gives the same error.

scottalanmiller

Is oracle-lib-compat installed?

wirestyle22

@scottalanmiller said in httpd dead but pid file exists:

oracle-lib-compat

No it's not

scottalanmiller

@wirestyle22 said in httpd dead but pid file exists:

@scottalanmiller said in httpd dead but pid file exists:

oracle-lib-compat

No it's not

That's what provides that file.

scottalanmiller

Try removing the app that Apache runs and see if Apache will start when clean before looking further.

scottalanmiller

Also, consider just moving to a maintained server. RHEL 6 is one thing today, a stretch, but one thing. RHEL 5 constitutes and abandoned server of no value. Why bother with trying to fix a workload deemed "not important enough to run?"

wirestyle22

@scottalanmiller said in httpd dead but pid file exists:

Also, consider just moving to a maintained server. RHEL 6 is one thing today, a stretch, but one thing. RHEL 5 constitutes and abandoned server of no value. Why bother with trying to fix a workload deemed "not important enough to run?"

We've been told we are phasing it out but if history is any indicator that could mean years. They like to tell us what we want to hear and then not doing anything about it.

DustinB3403

No one has asked yet (and I'm assuming this) but why haven't you restored to a known working condition?

scottalanmiller

@wirestyle22 said in httpd dead but pid file exists:

@scottalanmiller said in httpd dead but pid file exists:

Also, consider just moving to a maintained server. RHEL 6 is one thing today, a stretch, but one thing. RHEL 5 constitutes and abandoned server of no value. Why bother with trying to fix a workload deemed "not important enough to run?"

We've been told we are phasing it out but if history is any indicator that could mean years. They like to tell us what we want to hear and then not doing anything about it.

Well then, just tell them it phased itself out. Problem solved since they were planning to do this anyway. If they push back, tell them that they need to move to something supportable if phasing out wasn't the actual plan.

You need to tell them what you want to hear.

wirestyle22

@DustinB3403 said in httpd dead but pid file exists:

No one has asked yet (and I'm assuming this) but why haven't you restored to a known working condition?

I need to understand why this happened in order to prevent it from happening in the future. Nothing has turned up any answers.

DustinB3403

@wirestyle22 said in httpd dead but pid file exists:

@DustinB3403 said in httpd dead but pid file exists:

No one has asked yet (and I'm assuming this) but why haven't you restored to a known working condition?

I need to understand why this happened in order to prevent it from happening in the future. Nothing has turned up any answers.

Is this something that you can restore to separate hardware/hypervisor just so the system is operable while you work on this down system?

It seems like there is a need to have this system function, but not with any real priority. So the simple approach might be to just stand up a working version and compare them side by side.

IRJ

@wirestyle22 said in httpd dead but pid file exists:

@DustinB3403 said in httpd dead but pid file exists:

No one has asked yet (and I'm assuming this) but why haven't you restored to a known working condition?

I need to understand why this happened in order to prevent it from happening in the future. Nothing has turned up any answers.

It's always nice to know root cause, but sometimes it is very difficult to replicate an issue like this.

scottalanmiller

@IRJ said in httpd dead but pid file exists:

@wirestyle22 said in httpd dead but pid file exists:

@DustinB3403 said in httpd dead but pid file exists:

No one has asked yet (and I'm assuming this) but why haven't you restored to a known working condition?

I need to understand why this happened in order to prevent it from happening in the future. Nothing has turned up any answers.

It's always nice to know root cause, but sometimes it is very difficult to replicate an issue like this.

And not worth your time. On Wall St. we had things like this happen from time to time and the question was always... what's the value of getting the system back online vs. the cost of downtime and resources to try to track down something that might never be an issue again?

Almost never did root cause analysis make the cut when actual business value was applied. It sounds great, it sounds like something IT should be doing. But when we put on our real IT hats and do our proper cost analysis, it's rarely a good idea. And to do it on a RHEL 5 system slated for decommission? Essentially zero chance that any investigation could be justified.

wirestyle22

is there a way I can ignore /usr/lib/php/modules/oci8.so just to verify apache will start if this file is not in play? I actually think a file level restore of this file will fix it but I want to test that before I offer it as a solution

scottalanmiller

@wirestyle22 said in httpd dead but pid file exists:

is there a way I can ignore /usr/lib/php/modules/oci8.so just to verify apache will start if this file is not in play? I actually think a file level restore of this file will fix it but I want to test that before I offer it as a solution

You'd "ignore" it by removing any PHP app that tries to use it.

DustinB3403

@wirestyle22 said in httpd dead but pid file exists:

is there a way I can ignore /usr/lib/php/modules/oci8.so just to verify apache will start if this file is not in play? I actually think a file level restore of this file will fix it but I want to test that before I offer it as a solution

Are you unable to take a backup of this system?

Seems like an odd thing to try and do when solutions exist for this very case of "just in case"