Helix P4 / Perforce not working after Maintenance update
Since my Linode Maintenance update:
"
Maintenance is required for one or more of your Linodes' physical hosts. This maintenance is necessary to patch the MDS (ZombieLoad) CPU vulnerability[1] and will incur downtime for your Linode(s).
"
I can no longer connect to P4, and i can't seem to restart the related helix services
I'm out of my depth here with Linode/Linux, so any help would be much appreciated
When attempting to connect I get the following error:
Connect to server failed; check $P4PORT.
TCP connect to *:1666 failed.
connect: *:1666: Connection refused
service --status-all; shows as:
[ + ] apache-htcacheclean
[ + ] apache2
[ + ] apparmor
[ + ] apport
[ + ] atd
[ - ] bootmisc.sh
[ + ] cgmanager
[ - ] cgproxy
[ - ] checkfs.sh
[ - ] checkroot-bootclean.sh
[ - ] checkroot.sh
[ ? ] confluence
[ + ] console-setup
[ + ] cron
[ + ] dbus
[ - ] dns-clean
[ + ] exim4
[ + ] fail2ban
[ + ] grub-common
[ - ] helix-p4dctl
[ - ] hostname.sh
[ - ] hwclock.sh
[ + ] irqbalance
[ + ] keyboard-setup
[ - ] killprocs
[ - ] kmod
[ - ] mountall-bootclean.sh
[ - ] mountall.sh
[ - ] mountdevsubfs.sh
[ - ] mountkernfs.sh
[ - ] mountnfs-bootclean.sh
[ - ] mountnfs.sh
[ + ] mysql
[ + ] networking
[ + ] ntp
[ + ] ondemand
[ + ] php7.0-fpm
[ + ] php7.2-fpm
[ - ] plymouth
[ - ] plymouth-log
[ - ] pppd-dns
[ + ] procps
[ + ] rc.local
[ + ] resolvconf
[ - ] rsync
[ + ] rsyslog
[ - ] screen-cleanup
[ - ] screen-cleanup.dpkg-new
[ - ] sendsigs
[ + ] ssh
[ + ] sysstat
[ + ] udev
[ + ] ufw
[ - ] umountfs
[ - ] umountnfs.sh
[ - ] umountroot
[ + ] unattended-upgrades
[ + ] urandom
[ - ] uuidd
service helix-p4d start; returns:
Failed to start helix-p4d.service: Unit helix-p4d.service not found.
*:~# service helix-p4dctl start
Job for helix-p4dctl.service failed because the control process exited with error code. See "systemctl status helix-p4dctl.service" and "journalctl -xe" for details.
systemctl status helix-p4dctl.service; returns:
● helix-p4dctl.service - LSB: Starts all Perforce services
Loaded: loaded (/etc/init.d/helix-p4dctl; bad; vendor preset: enabled)
Active: failed (Result: exit-code) since Tue 2019-07-02 21:41:29 BST; 7min ago
Docs: man:systemd-sysv-generator(8)
Process: 2081 ExecStart=/etc/init.d/helix-p4dctl start (code=exited, status=1/FAILURE)Jul 02 21:36:29 carbon systemd[1]: Starting LSB: Starts all Perforce services…
Jul 02 21:41:29 carbon p4dctl[2086]: error: 'master' p4d: '/opt/perforce/sbin/p4d' exited with status 2
Jul 02 21:41:29 carbon p4dctl[2086]: Started 0 services.
Jul 02 21:41:29 carbon p4dctl[2086]: error: Not all services started successfully.
Jul 02 21:41:29 carbon systemd[1]: helix-p4dctl.service: Control process exited, code=exited status=1
Jul 02 21:41:29 carbon systemd[1]: Failed to start LSB: Starts all Perforce services.
Jul 02 21:41:29 carbon systemd[1]: helix-p4dctl.service: Unit entered failed state.
Jul 02 21:41:29 carbon systemd[1]: helix-p4dctl.service: Failed with result 'exit-code'.
lines 1-14/14 (END)
● helix-p4dctl.service - LSB: Starts all Perforce services
Loaded: loaded (/etc/init.d/helix-p4dctl; bad; vendor preset: enabled)
Active: failed (Result: exit-code) since Tue 2019-07-02 21:41:29 BST; 7min ago
Docs: man:systemd-sysv-generator(8)
Process: 2081 ExecStart=/etc/init.d/helix-p4dctl start (code=exited, status=1/FAILURE)Jul 02 21:36:29 carbon systemd[1]: Starting LSB: Starts all Perforce services…
Jul 02 21:41:29 carbon p4dctl[2086]: error: 'master' p4d: '/opt/perforce/sbin/p4d' exited with status 255.
Jul 02 21:41:29 carbon p4dctl[2086]: Started 0 services.
Jul 02 21:41:29 carbon p4dctl[2086]: error: Not all services started successfully.
Jul 02 21:41:29 carbon systemd[1]: helix-p4dctl.service: Control process exited, code=exited status=1
Jul 02 21:41:29 carbon systemd[1]: Failed to start LSB: Starts all Perforce services.
Jul 02 21:41:29 carbon systemd[1]: helix-p4dctl.service: Unit entered failed state.
Jul 02 21:41:29 carbon systemd[1]: helix-p4dctl.service: Failed with result 'exit-code'.
journalctl -xe; returns the following, to which i have no idea what users it;s referring to (todds, git, minecraft as I've never seen these before):
Jul 02 21:52:43 carbon sshd[4183]: Failed password for invalid user git from 175.126.176.21 port 38688 ssh2
Jul 02 21:52:43 carbon sshd[4183]: Received disconnect from 175.126.176.21 port 38688:11: Bye Bye [preauth]
Jul 02 21:52:43 carbon sshd[4183]: Disconnected from 175.126.176.21 port 38688 [preauth]
Jul 02 21:52:50 carbon sshd[4195]: Invalid user minecraft from 128.199.202.206
Jul 02 21:52:50 carbon sshd[4195]: input_userauth_request: invalid user minecraft [preauth]
Jul 02 21:52:50 carbon sshd[4195]: pam_unix(sshd:auth): check pass; user unknown
Jul 02 21:52:50 carbon sshd[4195]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=128.
Jul 02 21:52:53 carbon sshd[4195]: Failed password for invalid user minecraft from 128.199.202.206 port 48014 ssh2
Jul 02 21:52:53 carbon sshd[4195]: Received disconnect from 128.199.202.206 port 48014:11: Bye Bye [preauth]
Jul 02 21:52:53 carbon sshd[4195]: Disconnected from 128.199.202.206 port 48014 [preauth]
Jul 02 21:53:04 carbon kernel: iptables denied: IN=eth0 OUT= MAC=f2:3c:91:bd:bd:da:84:78:ac:0d:8f:41:08:00 SRC=185.137.234.23 D
Jul 02 21:53:24 carbon kernel: iptables denied: IN=eth0 OUT= MAC=f2:3c:91:bd:bd:da:84:78:ac:0d:8f:41:08:00 SRC=185.176.27.170 D
Jul 02 21:53:25 carbon kernel: iptables denied: IN=eth0 OUT= MAC=f2:3c:91:bd:bd:da:84:78:ac:0d:8f:41:08:00 SRC=89.248.174.3 DST
Jul 02 21:53:27 carbon kernel: iptables denied: IN=eth0 OUT= MAC=f2:3c:91:bd:bd:da:84:78:ac:5a:1a:41:08:00 SRC=120.52.152.16 DS
Jul 02 21:53:39 carbon sshd[4257]: Invalid user todds from 106.12.132.187
Jul 02 21:53:39 carbon sshd[4257]: input_userauth_request: invalid user todds [preauth]
Jul 02 21:53:39 carbon sshd[4257]: pam_unix(sshd:auth): check pass; user unknown
Jul 02 21:53:39 carbon sshd[4257]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=106.
Jul 02 21:53:41 carbon sshd[4257]: Failed password for invalid user todds from 106.12.132.187 port 38930 ssh2
Jul 02 21:53:41 carbon sshd[4257]: Received disconnect from 106.12.132.187 port 38930:11: Bye Bye [preauth]
Jul 02 21:53:41 carbon sshd[4257]: Disconnected from 106.12.132.187 port 38930 [preauth]
Jul 02 21:53:44 carbon kernel: iptables denied: IN=eth0 OUT= MAC=f2:3c:91:bd:bd:da:84:78:ac:5a:1a:41:08:00 SRC=185.137.234.23 D
Jul 02 21:53:48 carbon kernel: iptables denied: IN=eth0 OUT= MAC=f2:3c:91:bd:bd:da:84:78:ac:5a:1a:41:08:00 SRC=83.97.20.36 DST=
lines 1939-1961/1961 (END)
Jul 02 21:52:43 carbon sshd[4183]: Failed password for invalid user git from 175.126.176.21 port 38688 ssh2
Jul 02 21:52:43 carbon sshd[4183]: Received disconnect from 175.126.176.21 port 38688:11: Bye Bye [preauth]
Jul 02 21:52:43 carbon sshd[4183]: Disconnected from 175.126.176.21 port 38688 [preauth]
Jul 02 21:52:50 carbon sshd[4195]: Invalid user minecraft from 128.199.202.206
Jul 02 21:52:50 carbon sshd[4195]: input_userauth_request: invalid user minecraft [preauth]
Jul 02 21:52:50 carbon sshd[4195]: pam_unix(sshd:auth): check pass; user unknown
Jul 02 21:52:50 carbon sshd[4195]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=128.199.202.206
Jul 02 21:52:53 carbon sshd[4195]: Failed password for invalid user minecraft from 128.199.202.206 port 48014 ssh2
Jul 02 21:52:53 carbon sshd[4195]: Received disconnect from 128.199.202.206 port 48014:11: Bye Bye [preauth]
Jul 02 21:52:53 carbon sshd[4195]: Disconnected from 128.199.202.206 port 48014 [preauth]
Jul 02 21:53:04 carbon kernel: iptables denied: IN=eth0 OUT= MAC=f2:3c:91:bd:bd:da:84:78:ac:0d:8f:41:08:00 SRC=185.137.234.23 DST=212.71.255.218 LEN=40 TOS=0x00 PREC=0x00 TTL=251 ID=56020 PROTO=TCP SPT
Jul 02 21:53:24 carbon kernel: iptables denied: IN=eth0 OUT= MAC=f2:3c:91:bd:bd:da:84:78:ac:0d:8f:41:08:00 SRC=185.176.27.170 DST=212.71.255.218 LEN=40 TOS=0x00 PREC=0x00 TTL=247 ID=37332 PROTO=TCP SPT
Jul 02 21:53:25 carbon kernel: iptables denied: IN=eth0 OUT= MAC=f2:3c:91:bd:bd:da:84:78:ac:0d:8f:41:08:00 SRC=89.248.174.3 DST=212.71.255.218 LEN=40 TOS=0x00 PREC=0x00 TTL=252 ID=54321 PROTO=TCP SPT=4
Jul 02 21:53:27 carbon kernel: iptables denied: IN=eth0 OUT= MAC=f2:3c:91:bd:bd:da:84:78:ac:5a:1a:41:08:00 SRC=120.52.152.16 DST=212.71.255.218 LEN=30 TOS=0x00 PREC=0x00 TTL=239 ID=17443 PROTO=UDP SPT=
Jul 02 21:53:39 carbon sshd[4257]: Invalid user todds from 106.12.132.187
Jul 02 21:53:39 carbon sshd[4257]: input_userauth_request: invalid user todds [preauth]
Jul 02 21:53:39 carbon sshd[4257]: pam_unix(sshd:auth): check pass; user unknown
Jul 02 21:53:39 carbon sshd[4257]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=106.12.132.187
Jul 02 21:53:41 carbon sshd[4257]: Failed password for invalid user todds from 106.12.132.187 port 38930 ssh2
Jul 02 21:53:41 carbon sshd[4257]: Received disconnect from 106.12.132.187 port 38930:11: Bye Bye [preauth]
Jul 02 21:53:41 carbon sshd[4257]: Disconnected from 106.12.132.187 port 38930 [preauth]
Jul 02 21:53:44 carbon kernel: iptables denied: IN=eth0 OUT= MAC=f2:3c:91:bd:bd:da:84:78:ac:5a:1a:41:08:00 SRC=185.137.234.23 DST=212.71.255.218 LEN=40 TOS=0x00 PREC=0x00 TTL=251 ID=12612 PROTO=TCP SPT
Jul 02 21:53:48 carbon kernel: iptables denied: IN=eth0 OUT= MAC=f2:3c:91:bd:bd:da:84:78:ac:5a:1a:41:08:00 SRC=83.97.20.36 DST=212.71.255.218 LEN=40 TOS=0x08 PREC=0x20 TTL=239 ID=51077 PROTO=TCP SPT=49
lines 1939-1961/1961 (END)
Jul 02 21:52:43 carbon sshd[4183]: Failed password for invalid user git from 175.126.176.21 port 38688 ssh2
Jul 02 21:52:43 carbon sshd[4183]: Received disconnect from 175.126.176.21 port 38688:11: Bye Bye [preauth]
Jul 02 21:52:43 carbon sshd[4183]: Disconnected from 175.126.176.21 port 38688 [preauth]
Jul 02 21:52:50 carbon sshd[4195]: Invalid user minecraft from 128.199.202.206
Jul 02 21:52:50 carbon sshd[4195]: input_userauth_request: invalid user minecraft [preauth]
Jul 02 21:52:50 carbon sshd[4195]: pam_unix(sshd:auth): check pass; user unknown
Jul 02 21:52:50 carbon sshd[4195]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=128.199.202.206
Jul 02 21:52:53 carbon sshd[4195]: Failed password for invalid user minecraft from 128.199.202.206 port 48014 ssh2
Jul 02 21:52:53 carbon sshd[4195]: Received disconnect from 128.199.202.206 port 48014:11: Bye Bye [preauth]
Jul 02 21:52:53 carbon sshd[4195]: Disconnected from 128.199.202.206 port 48014 [preauth]
Jul 02 21:53:04 carbon kernel: iptables denied: IN=eth0 OUT= MAC=f2:3c:91:bd:bd:da:84:78:ac:0d:8f:41:08:00 SRC=185.137.234.23 DST=212.71.255.218 LEN=40 TOS=0x00 PREC=0x00 TTL=251 ID=56020 PROTO=TCP SPT=49122 DPT=6490 WINDOW=1024 RES=0x00 SYN URGP=0
Jul 02 21:53:24 carbon kernel: iptables denied: IN=eth0 OUT= MAC=f2:3c:91:bd:bd:da:84:78:ac:0d:8f:41:08:00 SRC=185.176.27.170 DST=212.71.255.218 LEN=40 TOS=0x00 PREC=0x00 TTL=247 ID=37332 PROTO=TCP SPT=56766 DPT=18944 WINDOW=1024 RES=0x00 SYN URGP=0
Jul 02 21:53:25 carbon kernel: iptables denied: IN=eth0 OUT= MAC=f2:3c:91:bd:bd:da:84:78:ac:0d:8f:41:08:00 SRC=89.248.174.3 DST=212.71.255.218 LEN=40 TOS=0x00 PREC=0x00 TTL=252 ID=54321 PROTO=TCP SPT=47652 DPT=84 WINDOW=65535 RES=0x00 SYN URGP=0
Jul 02 21:53:27 carbon kernel: iptables denied: IN=eth0 OUT= MAC=f2:3c:91:bd:bd:da:84:78:ac:5a:1a:41:08:00 SRC=120.52.152.16 DST=212.71.255.218 LEN=30 TOS=0x00 PREC=0x00 TTL=239 ID=17443 PROTO=UDP SPT=50585 DPT=5351 LEN=10
Jul 02 21:53:39 carbon sshd[4257]: Invalid user todds from 106.12.132.187
Jul 02 21:53:39 carbon sshd[4257]: input_userauth_request: invalid user todds [preauth]
Jul 02 21:53:39 carbon sshd[4257]: pam_unix(sshd:auth): check pass; user unknown
Jul 02 21:53:39 carbon sshd[4257]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=106.12.132.187
Jul 02 21:53:41 carbon sshd[4257]: Failed password for invalid user todds from 106.12.132.187 port 38930 ssh2
Jul 02 21:53:41 carbon sshd[4257]: Received disconnect from 106.12.132.187 port 38930:11: Bye Bye [preauth]
Jul 02 21:53:41 carbon sshd[4257]: Disconnected from 106.12.132.187 port 38930 [preauth]
Jul 02 21:53:44 carbon kernel: iptables denied: IN=eth0 OUT= MAC=f2:3c:91:bd:bd:da:84:78:ac:5a:1a:41:08:00 SRC=185.137.234.23 DST=212.71.255.218 LEN=40 TOS=0x00 PREC=0x00 TTL=251 ID=12612 PROTO=TCP SPT=49122 DPT=9392 WINDOW=1024 RES=0x00 SYN URGP=0
Jul 02 21:53:48 carbon kernel: iptables denied: IN=eth0 OUT= MAC=f2:3c:91:bd:bd:da:84:78:ac:5a:1a:41:08:00 SRC=83.97.20.36 DST=212.71.255.218 LEN=40 TOS=0x08 PREC=0x20 TTL=239 ID=51077 PROTO=TCP SPT=49371 DPT=780 WINDOW=1024 RES=0x00 SYN URGP=0
6 Replies
Hey there,
Based on the errors provided in your community post, this appears to be related to a known issue with helix's startup script with a resolution addressed through step-by-step instructions in this guide.
If those troubleshooting steps don't improve this issue, we recommend booting your Linode into the kernel ain use prior to the reboot and retesting your system, as a drastic kernel jump or inline upgrade can have breaking impacts
Your journalctl output is showing attempts to access your system by guessing users and corresponding credentials, which is why you are seeing keywords for services you don't have running. They don't appear to have been successful from the output provided.
We hope that helps move your troubleshooting forward. If you encounter issues with these troubleshooting steps, please provide us what errors you observe and we'll do our best to help provide further insight.
Sincerely,
Tara T
Linode Support Team
Thank you for your help.
I cannot seem to get past the following step: (I am on Ubuntu, rather than Cent OS, is there different commands to this?)
- Enable
sudo systemctl enable helix-p4dctl
- Manually Start
sudo systemctl start helix-p4dctl
I've installed systemd, but i get the following error:
Failed to execute operation: No such file or directory
I have the exact same problem starting up the perforce server.
I followed the guide to create an startup script using systemd successfully but the result is the same than the original post when trying to start the service.
When running: systemctl start helix-p4dctl
I get this: Job for helix-p4dctl.service failed because the control process exited with error code.
See "systemctl status helix-p4dctl.service" and "journalctl -xe" for details.
And running both give me the same than when using the init.d script so the problem is not how to start it but something else
PS: the server was working fine before Linode maintenance processes were taken
I know this is an old thread, but if anyone else is experiencing the same issue, I managed to fix it by running p4d -xu (docs) to force database upgrades to be run when the service next started. I also cleaned the server locks directory with p4d -xU CleanServerLocks but I don't think that had any effect.
I found this by running p4d inside the root directory and finding no output, but the logs said something along the lines of:
Database is at old upgrade level 33. Use 'p4d -r /opt/perforce/servers/master/root -xu' to upgrade to level 39.
So here we are almost 2 years later with the same problem. Looks like every time I update the system packages I will run into this problem and this it what got it solved:
p4d -r /opt/perforce/servers/master -xu
