Need to track what PHP script is generating a file on nix
-
@guyinpv said in Need to track what PHP script is generating a file on nix:
I've got a LiquidWeb VPS with WHM/cPanel and a couple sites on there.
Some part of the Wordpress site hosted on there is creating these random files in the /var/tmp folder to the point where it will fill up over time and run out of space. All the files are about 5MB in size, and they are all named the same, with "php" followed by 6 random characters with no file extension.
Opening a file shows only binary gobbledigoook with no real clues what it's for.The file is created as the user/group of the user of the cPanel account. And as far as I've seen, it creates about 4 or 5 of these files a day at various times.
No errors appear in normal PHP error logs to match, so it does not appear to coincide with any kind of error dump. The files are also not generated through normal means by the PHP or Apache or MySQL processes.
My best guess is that they are created by a poorly written plugin. It's creating the files and not cleaning them up.
I have attempted to match up the time stamps of the files creation date with the raw access logs of the server. I went line by line opening every URL endpoint from the access log and then check if a file was created but this did not work, I couldn't get one to generate by going down the access log.
So that leaves my final method. I need some way to backtrack. To detect when such a file is written and somehow record or log what process or code is writing it.
I need some kind of linux tool to monitor the file operations of the WP site and log whenever a file gets created to the /tmp folder and by what script.
Not sure if this is even possible.If not, or it would be way too consuming of resources to monitor all file operations, I'm open to other ideas. But right now, some stupid code somewhere is writing endless 5MB files to /tmp and filling it up over and over.
Since this is a production site, I really can't experiment by messing with WP like turning off plugins and changing theme and stuff like that.
Any ideas?
Can you start by monitoring the / or /tmp folder location with Zabbix or any other monitoring tool?
-
@dbeato I should be able to install and use any tool I want, it's a VPS.
But monitoring the folder, I don't think will work. At best it would only be able to tell me that the PHP process wrote a file, but not which script did it. I would need some kind of application monitor that monitors all the PHP scripts as well as monitor when they write files to that tmp folder.
-
You want something like DTRACE, that's going to be tough.
-
@guyinpv maybe fswatch could help.
i think most distros have it in the repository.
http://emcrisostomo.github.io/fswatch/ -
I thought maybe just a simply stack trace log that could be "turned on" in Apache and/or PHP for temporary time like a few days, then turn it back off.
Logging all PHP functions for multiple days would likely produce a mountain of data, so I'd have to figure out how to save that and search it.
Wouldn't the Zend engine or some other PHP diagnostic monitoring tool be able to do this? I think it's something that can be done using Apache/PHP tools rather than underlying OS tools, I don't know.
-
You can add this manually into the code yourself: http://php.net/manual/en/function.debug-print-backtrace.php
-
This isn't a very typical task, it's a bit of a weird thing to want. You can't get this normally with any language without building it into the application itself or tracking system calls.
-
I do understand the initial intention to get rid of those files by completely eliminating their source. Try this https://wordpress.org/plugins/string-locator/ to search for "/var/tmp/" hardcoded anywhere throughout the Wordpress installation. If not successful at the moment, you might simply automate their deletion with some cron job until you get to know what produces them.
-
@darek-hamann said in Need to track what PHP script is generating a file on nix:
I do understand the initial intention to get rid of those files by completely eliminating their source. Try this https://wordpress.org/plugins/string-locator/ to search for "/var/tmp/" hardcoded anywhere throughout the Wordpress installation. If not successful at the moment, you might simply automate their deletion with some cron job until you get to know what produces them.
This was the first thing I tried. I searched through the entire themes folder and plugins folder for any reference to 'tmp' and other variations. But no luck.
The problem with WP is that everything is cobbled together from variables and system calls and WP functions. So most likely there is no place where the folder path is selected explicitly where I can search for it in this way. -
@mlnews said in Need to track what PHP script is generating a file on nix:
You can add this manually into the code yourself: http://php.net/manual/en/function.debug-print-backtrace.php
Nice, but where would I call it? So many different plugins and such, would be very hard to implement. But it only gives a backtrace, that doesn't exactly tell me when a function is writing to the file system.
-
@guyinpv said in Need to track what PHP script is generating a file on nix:
@mlnews said in Need to track what PHP script is generating a file on nix:
You can add this manually into the code yourself: http://php.net/manual/en/function.debug-print-backtrace.php
Nice, but where would I call it? So many different plugins and such, would be very hard to implement. But it only gives a backtrace, that doesn't exactly tell me when a function is writing to the file system.
That's the hard part, it has to be everywhere. This isn't a trivial thing to add to software.
-
Depending on the VPS's PHP implementation the child processes may or may not contain some useful stuff in the command line, such as which script is being executed.
Would it be helpful for you to get the PID's of any processes which open any file in a target dir, then log the full command line of that PID to a file?
If so, you can run the code below.
You should run this momentarily, exit with "CTRL-C" and check the log output. Loads of stuff writes to '/tmp/' and this will log all of it, so you might very likely fill the disk if you run out for a coffee and leave it running.
Ideally you should have a second SSH session to the VPS so you can kill it if necessary, and use 'tail -f /tmp/test/log/lsof.log' to monitor it's output in realtime.
watch -n 10 'for pid in $(lsof +D /tmp/ 2>/dev/null| awk '''/[0-9]/{print $2}'''); do if [ -n "$pid" ]; then ps f -p $pid >> /tmp/test/log/lsof.log 2>/dev/null; else sleep 0;fi;done'
The VPS probably doesn't have 'watch' installed, which runs the command every -n seconds. The rest of the commands used here should be on more or less any linux server, so you can use a while loop instead if necessary:
while true;do for pid in $(lsof +D /tmp/ 2>/dev/null| awk '/[0-9]/{print $2}'); do if [ -n "$pid" ]; then ps f -p $pid >> /tmp/path/to/log/lsof.log 2>/dev/null; else sleep 10;fi;done; done
Replace '/tmp/path/to/log/lsof.log' with whatever you want the logfile to be. '/tmp/' is the target dir to watch.
Example output is:
PID TTY STAT TIME COMMAND
25394 pts/46 R+ 0:00 dd if=/dev/urandom of=/tmp/test/test.php bs=512 count=100000
