No longer able to switch users!
Hi guys,
I'm posting here with the hope I can get some help on an issue that is happening in (possible) all my Linodes (CentOS7).
One of the procedures/policies in the company is to have the system update according to the latest packages release by CentOS.
After the last update made (clamav, java and kernel where updated) we noticed that, after rebooting the Linodes, we where no longer able to sudo into a different user, getting the following message:
[jays@jays-tst ~]$ sudo su - jayb
[sudo] password for jays:
Last login: Mon Jul 30 12:02:27 EDT 2018 on pts/0
su: cannot open session: Cannot make/remove an entry for the specified session
I can run sudo but I can't sudo to any user.
By checking the journalctl:
[jays@jays-tst ~]$ sudo journalctl -xe
Jul 31 06:35:26 jays-tst sudo[7620]: jays : TTY=pts/0 ; PWD=/home/jays ; USER=root ; COMMAND=/bin/su - jayb
Jul 31 06:35:26 jays-tst su[7624]: (to jayb) jays on pts/0
Jul 31 06:35:26 jays-tst su[7624]: pamttyaudit(su-l:session): error reading current audit status: Protocol not supported
Jul 31 06:35:26 jays-tst su[7624]: pam_unix(su-l:session): session opened for user jayb by jays(uid=0)
Jul 31 06:35:30 jays-tst sudo[7700]: jays : TTY=pts/0 ; PWD=/home/jays ; USER=root ; COMMAND=/bin/journalctl -xe
We also notice that some services started to fail on startup:
[jays@jays-tst ~]$ systemctl --failed
UNIT LOAD ACTIVE SUB DESCRIPTION
? auditd.service loaded failed failed Security Auditing Service
? NetworkManager-wait-online.service loaded failed failed Network Manager Wait Online
? rc-local.service loaded failed failed /etc/rc.d/rc.local Compatibility
Some of our findings:
[jays@jays-tst ~]$ sudo systemctl status auditd.service
? auditd.service - Security Auditing Service
Loaded: loaded (/usr/lib/systemd/system/auditd.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Mon 2018-07-30 11:42:23 EDT; 17min ago
Docs: man:auditd(8)
https://github.com/linux-audit/audit-documentation
Process: 729 ExecStart=/sbin/auditd (code=exited, status=1/FAILURE)Jul 30 11:42:23 jays-tst systemd[1]: Starting Security Auditing Service…
Jul 30 11:42:23 jays-tst auditd[778]: Error - audit support not in kernel
Jul 30 11:42:23 jays-tst auditd[778]: Cannot open netlink audit socket
Jul 30 11:42:23 jays-tst auditd[778]: The audit daemon is exiting.
Jul 30 11:42:23 jays-tst systemd[1]: auditd.service: control process exited, code=exited status=1
Jul 30 11:42:23 jays-tst systemd[1]: Failed to start Security Auditing Service.
Jul 30 11:42:23 jays-tst systemd[1]: Unit auditd.service entered failed state.
Jul 30 11:42:23 jays-tst systemd[1]: auditd.service failed.[jays@jays-tst ~]$ sudo less /var/log/messages
Jul 30 11:19:22 jays-tst auditd[720]: Error - audit support not in kernel
Jul 30 11:19:22 jays-tst auditd[720]: Cannot open netlink audit socket
Jul 30 11:19:22 jays-tst auditd[720]: The audit daemon is exiting.
Jul 30 11:19:22 jays-tst auditd: Cannot daemonize (Success)
Jul 30 11:19:22 jays-tst auditd: The audit daemon is exiting.
Jul 30 11:19:22 jays-tst systemd: auditd.service: control process exited, code=exited status=1
Jul 30 11:19:22 jays-tst systemd: Failed to start Security Auditing Service.
Jul 30 11:19:22 jays-tst systemd: Unit auditd.service entered failed state.
Jul 30 11:19:22 jays-tst systemd: auditd.service failed[jays@jays-tst ~]$ sudo su - jayb
Last login: Mon Jul 30 12:04:43 EDT 2018 on pts/0
su: cannot open session: Cannot make/remove an entry for the specified session
[jays@jays-tst ~]$ sudo less /var/log/secure
Jul 30 12:02:27 jays-tst sudo: jays : TTY=pts/0 ; PWD=/home/jays ; USER=root ; COMMAND=/bin/su - jayb
Jul 30 12:02:27 jays-tst su: pamttyaudit(su-l:session): error reading current audit status: Protocol not supported
Jul 30 12:02:27 jays-tst su: pam_unix(su-l:session): session opened for user jayb by jays(uid=0)
We tried to rollback the updated packages, but, still, the issue persists.
Affected packages:
[jays@jays-tst ~]$ sudo yum history info 88
Loaded plugins: fastestmirror
Transaction ID : 88
Begin time : Thu Jul 26 04:32:19 2018
Begin rpmdb : 596:d58235e358eedc190ea427e2b5a91ee27b52b023
End time : 04:35:34 2018 (195 seconds)
End rpmdb : 596:999ca89b0584dc699e31a6a2339b3fb930dc6de7
User : <jays>
Return-Code : Success
Command Line : update
Transaction performed with:
Installed rpm-4.11.3-32.el7.x8664 @base Installed yum-3.4.3-158.el7.centos.noarch @base Installed yum-metadata-parser-1.1.4-10.el7.x8664 @anaconda
Installed yum-plugin-fastestmirror-1.1.31-45.el7.noarch @base
Packages Altered:
Updated clamav-0.100.0-2.el7.x8664 > @epel Update 0.100.1-1.el7.x8664 > @epel
Updated clamav-data-0.100.0-2.el7.noarch > @epel
Update 0.100.1-1.el7.noarch > @epel
Updated clamav-filesystem-0.100.0-2.el7.noarch > @epel
Update 0.100.1-1.el7.noarch > @epel
Updated clamav-lib-0.100.0-2.el7.x8664 > @epel Update 0.100.1-1.el7.x8664 > @epel
Updated clamav-update-0.100.0-2.el7.x8664 > @epel Update 0.100.1-1.el7.x8664 > @epel
Updated clamd-0.100.0-2.el7.x8664 > @epel Update 0.100.1-1.el7.x8664 > @epel
Updated java-1.8.0-openjdk-1:1.8.0.171-8.b10.el75.x8664 > @updates
Update 1:1.8.0.181-3.b13.el75.x8664 > @updates
Updated java-1.8.0-openjdk-accessibility-1:1.8.0.171-8.b10.el75.x8664 @updates
Update 1:1.8.0.181-3.b13.el75.x8664 @updates
Updated java-1.8.0-openjdk-accessibility-debug-1:1.8.0.171-8.b10.el75.x8664 @updates
Update 1:1.8.0.181-3.b13.el75.x8664 @updates
Updated java-1.8.0-openjdk-debug-1:1.8.0.171-8.b10.el75.x8664 > @updates
Update 1:1.8.0.181-3.b13.el75.x8664 > @updates
Updated java-1.8.0-openjdk-demo-1:1.8.0.171-8.b10.el75.x8664 > @updates
Update 1:1.8.0.181-3.b13.el75.x8664 > @updates
Updated java-1.8.0-openjdk-demo-debug-1:1.8.0.171-8.b10.el75.x8664 @updates
Update 1:1.8.0.181-3.b13.el75.x8664 @updates
Updated java-1.8.0-openjdk-devel-1:1.8.0.171-8.b10.el75.x8664 > @updates
Update 1:1.8.0.181-3.b13.el75.x8664 > @updates
Updated java-1.8.0-openjdk-devel-debug-1:1.8.0.171-8.b10.el75.x8664 @updates
Update 1:1.8.0.181-3.b13.el75.x8664 @updates
Updated java-1.8.0-openjdk-headless-1:1.8.0.171-8.b10.el75.x8664 @updates
Update 1:1.8.0.181-3.b13.el75.x8664 @updates
Updated java-1.8.0-openjdk-headless-debug-1:1.8.0.171-8.b10.el75.x8664 @updates
Update 1:1.8.0.181-3.b13.el75.x8664 @updates
Updated java-1.8.0-openjdk-javadoc-1:1.8.0.171-8.b10.el75.noarch > @updates Update 1:1.8.0.181-3.b13.el75.noarch > @updates
Updated java-1.8.0-openjdk-javadoc-debug-1:1.8.0.171-8.b10.el75.noarch @updates Update 1:1.8.0.181-3.b13.el75.noarch @updates
Updated java-1.8.0-openjdk-javadoc-zip-1:1.8.0.171-8.b10.el75.noarch @updates Update 1:1.8.0.181-3.b13.el75.noarch @updates
Updated java-1.8.0-openjdk-javadoc-zip-debug-1:1.8.0.171-8.b10.el75.noarch @updates Update 1:1.8.0.181-3.b13.el75.noarch @updates
Updated java-1.8.0-openjdk-src-1:1.8.0.171-8.b10.el75.x8664 > @updates
Update 1:1.8.0.181-3.b13.el75.x8664 > @updates
Updated java-1.8.0-openjdk-src-debug-1:1.8.0.171-8.b10.el75.x8664 @updates
Update 1:1.8.0.181-3.b13.el75.x8664 @updates
Erase kernel-3.10.0-693.21.1.el7.x8664 > @updates Install kernel-3.10.0-862.9.1.el7.x8664 > @updates
Updated kernel-abi-whitelists-3.10.0-862.6.3.el7.noarch > @updates
Update 3.10.0-862.9.1.el7.noarch > @updates
Erase kernel-debug-3.10.0-693.21.1.el7.x8664 > @updates Install kernel-debug-3.10.0-862.9.1.el7.x8664 > @updates
Updated kernel-debug-devel-3.10.0-862.6.3.el7.x8664 > @updates Update 3.10.0-862.9.1.el7.x8664 > @updates
Erase kernel-devel-3.10.0-693.21.1.el7.x8664 > @updates Install kernel-devel-3.10.0-862.9.1.el7.x8664 > @updates
Updated kernel-headers-3.10.0-862.6.3.el7.x8664 > @updates Update 3.10.0-862.9.1.el7.x8664 > @updates
Updated kernel-tools-3.10.0-862.6.3.el7.x8664 > @updates Update 3.10.0-862.9.1.el7.x8664 > @updates
Updated kernel-tools-libs-3.10.0-862.6.3.el7.x8664 > @updates Update 3.10.0-862.9.1.el7.x8664 > @updates
Updated kernel-tools-libs-devel-3.10.0-862.6.3.el7.x8664 > @updates Update 3.10.0-862.9.1.el7.x8664 > @updates
Updated python-perf-3.10.0-862.6.3.el7.x8664 > @updates Update 3.10.0-862.9.1.el7.x8664 > @updates
history info</jays>
We already try to run a filesystem check into Rescue Mode, but, as per my understanding, all seems ok:
root@ttyS0:~# e2fsck /dev/sdb
e2fsck 1.42.13 (17-May-2015)
/dev/sdb: clean, 278372/3162112 files, 6167471/12632064 blocks
root@ttyS0:~#
root@ttyS0:~# e2fsck -f /dev/sdb
e2fsck 1.42.13 (17-May-2015)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/sdb: 278372/3162112 files (0.2% non-contiguous), 6167471/12632064 blocks
root@ttyS0:~#
After rebooting, we still have the very same issue.
Our last try was run the Linode using GRUB2, but it seems we run into a error at the reboot time. we only get the following message: Minimal BASH like line editing is supported. For the first word, TAB lists possible command completions. anywhere else TAB lists possible device or file completions.
Any help would be deeply appreciated since we are experience this in all our Linodes.
Regards,
Jay
2 Replies
As an update, in the mean while, I was able to switch users with the following work around:
on /etc/pam.d/su, we had the following two lines :
session include system-auth
session optional pam_xauth.so
I commented out the first one, restarted the session and it worked.
However the issue remains, we can't still use the auditing service.
The "Error - audit support not in kernel" sounds very odd, since we always had the auditing working.
Thanks in advance,
Jay
I have some finding on this topic and i want to update this topic in case someone runs into this very same issue.
One of my test was to use a older version of the Linode's Kernel. I tested with the versions 4.17.2-x8664-linode109 and 4.16.11-x8664-linode108, which have the very same issue.
By changing to 4.15.13-x86_64-linode106 version, the problem goes away!
In order to reproduce the issue, I created a new Linode, with a fresh CentOS7 image. The auditd package come installed by default. By running "auditctl -l" in the new fresh started Linode, I run into the very same problem!
Test scenario:
- Create a fresh Linode
- Choose CentOS7 image
- Start Linode
li1040-151 login: root
Password:
Login incorrect
li1040-151 login: root
Password:
Last failed login: Tue Jul 31 14:56:41 UTC 2018 on ttyS0
There was 1 failed login attempt since the last successful login.
Last login: Tue Jul 31 14:21:41 on ttyS0
[root@li1040-151 ~]# auditctl -l
Error - audit support not in kernel
Cannot open netlink audit socket
[root@li1040-151 ~]#
My conclusion is perhaps something was broke in between these kernel versions.