Need help, all outbound TCP packets blocked after yum update

L

LowRadio

Verified User
Joined
Apr 29, 2010
Messages
9
I recently did a routine update on a server.
1. yum update
2.then ./build update_versions
Never had major issues before but this time I noticed the mail queue was filling up.
Then I noticed wget, curl, ssh would not work, long story short.... outbound ICMP, UDP work fine except TCP, I can ping external domain names and ip's.
Before doing an update, I took a snapshot of the drives host on Google compute engine, since the update failed I restored the snapshots to a new VM (before the updates), tcp working again, did a yum update and tcp blocked again.

But I cannot use the snapeshot, for some reason httpd stopped serving the proper user domains and always serves the default apache is working page. (the snapshot is no good).

So I'm back to the updated instance with all the outbound tcp packets blocked.

CSF appears to been working fine but I stopped it, uninstalled it. flushed iptables, enable/disable firewalld, nothing works.

I've been google searching all day and I cannot find a solution, at this point I don't know what else I can do.
 
I can't give you support BUT!

Please do some normal work before asking support as also writing in your topic about > VERSIONS in use , versions in the updates, log file messages, . ( YUM update is i 'm pretty sure not only one update ;) at your box ? )
 
I would start by checking the logs and look for errors.

Did you do ./build update first, and then ./build update_versions?
 
Did you check with your host. We had a few issue were host just turn on firewalls on their side.
Also did you read through the yum log to see if anything jumps out at you? I yum updated just yesterday on cantos 7. No issues.
 
CentOS Linux release 7.6.1810 (Core) update to CentOS Linux release 7.7.1908 (Core)
I've been looking through logs for clues but I cannot find anything.
Yum log has nothing that I can see that is abnormal:
Code: 
Dec 05 19:06:21 Updated: libgcc-4.8.5-39.el7.x86_64
Dec 05 19:06:21 Updated: centos-release-7-7.1908.0.el7.centos.x86_64
Dec 05 19:06:22 Updated: 1:grub2-common-2.02-0.80.el7.centos.noarch
Dec 05 19:06:22 Updated: 32:bind-license-9.11.4-9.P2.el7.noarch
Dec 05 19:06:22 Updated: 1:grub2-pc-modules-2.02-0.80.el7.centos.noarch
Dec 05 19:06:22 Updated: firewalld-filesystem-0.6.3-2.el7_7.2.noarch
Dec 05 19:06:22 Updated: libX11-common-1.6.7-2.el7.noarch
Dec 05 19:06:23 Updated: kernel-headers-3.10.0-1062.7.1.el7.x86_64
Dec 05 19:06:23 Updated: 2:vim-filesystem-7.4.629-6.el7.x86_64
Dec 05 19:06:26 Updated: tzdata-2019c-1.el7.noarch
Dec 05 19:06:27 Updated: bash-4.2.46-33.el7.x86_64
Dec 05 19:06:32 Updated: glibc-common-2.17-292.el7.x86_64
Dec 05 19:06:32 Updated: nss-softokn-freebl-3.44.0-5.el7.x86_64
Dec 05 19:06:33 Updated: glibc-2.17-292.el7.x86_64
Dec 05 19:06:33 Updated: nspr-4.21.0-1.el7.x86_64
Dec 05 19:06:33 Updated: nss-util-3.44.0-3.el7.x86_64
Dec 05 19:06:33 Updated: libcom_err-1.42.9-16.el7.x86_64
Dec 05 19:06:33 Updated: libcap-2.22-10.el7.x86_64
Dec 05 19:06:33 Updated: audit-libs-2.8.5-4.el7.x86_64
Dec 05 19:06:33 Updated: libuuid-2.23.2-61.el7_7.1.x86_64
Dec 05 19:06:33 Updated: libdb-5.3.21-25.el7.x86_64
Dec 05 19:06:33 Updated: libstdc++-4.8.5-39.el7.x86_64
Dec 05 19:06:33 Updated: elfutils-libelf-0.176-2.el7.x86_64
Dec 05 19:06:33 Updated: readline-6.2-11.el7.x86_64
Dec 05 19:06:33 Updated: diffutils-3.3-5.el7.x86_64
Dec 05 19:06:35 Updated: ca-certificates-2019.2.32-76.el7_7.noarch
Dec 05 19:06:35 Updated: 1:openssl-libs-1.0.2k-19.el7.x86_64
Dec 05 19:06:36 Updated: coreutils-8.22-24.el7.x86_64
Dec 05 19:06:36 Updated: krb5-libs-1.15.1-37.el7_7.2.x86_64
Dec 05 19:06:36 Updated: libblkid-2.23.2-61.el7_7.1.x86_64
Dec 05 19:06:36 Updated: 2:shadow-utils-4.6-5.el7.x86_64
Dec 05 19:06:38 Updated: python-libs-2.7.5-86.el7.x86_64
Dec 05 19:06:38 Updated: python-2.7.5-86.el7.x86_64
Dec 05 19:06:38 Updated: libmount-2.23.2-61.el7_7.1.x86_64
Dec 05 19:06:39 Updated: glib2-2.56.1-5.el7.x86_64
Dec 05 19:06:40 Updated: binutils-2.27-41.base.el7_7.1.x86_64
Dec 05 19:06:40 Updated: e2fsprogs-libs-1.42.9-16.el7.x86_64
Dec 05 19:06:40 Updated: nss-softokn-3.44.0-5.el7.x86_64
Dec 05 19:06:41 Updated: nss-pem-1.0.3-7.el7.x86_64
Dec 05 19:06:41 Updated: nss-sysinit-3.44.0-4.el7.x86_64
Dec 05 19:06:41 Updated: nss-3.44.0-4.el7.x86_64
Dec 05 19:06:41 Updated: hostname-3.13-3.el7_7.1.x86_64
Dec 05 19:06:41 Updated: lz4-1.7.5-3.el7.x86_64
Dec 05 19:06:41 Updated: iptables-1.4.21-33.el7.x86_64
Dec 05 19:06:41 Updated: iproute-4.11.0-25.el7_7.2.x86_64
Dec 05 19:06:41 Updated: python-urllib3-1.10.2-7.el7.noarch
Dec 05 19:06:42 Updated: 1:python-google-compute-engine-20191120.00-g1.el7.noarch
Dec 05 19:06:42 Updated: python-chardet-2.2.1-3.el7.noarch
Dec 05 19:06:42 Updated: python-linux-procfs-0.4.11-4.el7.noarch
Dec 05 19:06:42 Updated: audit-libs-python-2.8.5-4.el7.x86_64
Dec 05 19:06:42 Updated: python-perf-3.10.0-1062.7.1.el7.x86_64
Dec 05 19:06:42 Updated: python-firewall-0.6.3-2.el7_7.2.noarch
Dec 05 19:06:43 Updated: grubby-8.28-26.el7.x86_64
Dec 05 19:06:43 Installed: 32:bind-export-libs-9.11.4-9.P2.el7.x86_64
Dec 05 19:06:43 Updated: libkadm5-1.15.1-37.el7_7.2.x86_64
Dec 05 19:06:43 Updated: libtirpc-0.2.4-0.16.el7.x86_64
Dec 05 19:06:43 Updated: libssh2-1.8.0-3.el7.x86_64
Dec 05 19:06:43 Updated: libcurl-7.29.0-54.el7_7.1.x86_64
Dec 05 19:06:43 Updated: curl-7.29.0-54.el7_7.1.x86_64
Dec 05 19:06:43 Installed: geoipupdate-2.5.0-1.el7.x86_64
Dec 05 19:06:43 Updated: GeoIP-1.5.0-14.el7.x86_64
Dec 05 19:06:44 Updated: 32:bind-libs-lite-9.11.4-9.P2.el7.x86_64
Dec 05 19:06:44 Updated: 32:bind-libs-9.11.4-9.P2.el7.x86_64
Dec 05 19:06:44 Updated: 2:nmap-ncat-6.40-19.el7.x86_64
Dec 05 19:06:44 Updated: libstdc++-devel-4.8.5-39.el7.x86_64
Dec 05 19:06:45 Updated: libdb-utils-5.3.21-25.el7.x86_64
Dec 05 19:06:45 Updated: rpm-libs-4.11.3-40.el7.x86_64
Dec 05 19:06:45 Updated: rpm-4.11.3-40.el7.x86_64
Dec 05 19:06:45 Updated: rpm-build-libs-4.11.3-40.el7.x86_64
Dec 05 19:06:45 Updated: rpm-python-4.11.3-40.el7.x86_64
Dec 05 19:06:45 Updated: yum-plugin-fastestmirror-1.1.31-52.el7.noarch
Dec 05 19:06:45 Updated: yum-3.4.3-163.el7.centos.noarch
Dec 05 19:06:45 Updated: libss-1.42.9-16.el7.x86_64
Dec 05 19:06:45 Updated: libcom_err-devel-1.42.9-16.el7.x86_64
Dec 05 19:06:45 Updated: krb5-devel-1.15.1-37.el7_7.2.x86_64
Dec 05 19:06:46 Updated: gpm-libs-1.20.7-6.el7.x86_64
Dec 05 19:06:46 Updated: libteam-1.27-9.el7.x86_64
Dec 05 19:06:46 Updated: teamd-1.27-9.el7.x86_64
Dec 05 19:06:46 Updated: 2:vim-minimal-7.4.629-6.el7.x86_64
Dec 05 19:06:46 Updated: bluez-libs-5.44-5.el7.x86_64
Dec 05 19:06:46 Updated: ipset-libs-7.1-1.el7.x86_64
Dec 05 19:06:46 Updated: ipset-7.1-1.el7.x86_64
Dec 05 19:06:46 Updated: freetype-2.8-14.el7.x86_64
Dec 05 19:06:46 Updated: kernel-tools-libs-3.10.0-1062.7.1.el7.x86_64
Dec 05 19:06:46 Updated: numactl-libs-2.0.12-3.el7_7.1.x86_64
Dec 05 19:06:46 Updated: libndp-1.2-9.el7.x86_64
Dec 05 19:06:47 Updated: glibc-headers-2.17-292.el7.x86_64
Dec 05 19:06:47 Updated: glibc-devel-2.17-292.el7.x86_64
Dec 05 19:06:47 Updated: libgomp-4.8.5-39.el7.x86_64
Dec 05 19:06:47 Updated: libsmartcols-2.23.2-61.el7_7.1.x86_64
Dec 05 19:06:49 Updated: 2:vim-common-7.4.629-6.el7.x86_64
Dec 05 19:06:49 Updated: 2:ethtool-4.8-10.el7.x86_64
Dec 05 19:06:49 Updated: kmod-libs-20-25.el7.x86_64
Dec 05 19:06:50 Updated: util-linux-2.23.2-61.el7_7.1.x86_64
Dec 05 19:06:50 Updated: procps-ng-3.3.10-26.el7_7.1.x86_64
Dec 05 19:06:50 Updated: kpartx-0.4.9-127.el7.x86_64
Dec 05 19:06:50 Updated: 7:device-mapper-1.02.158-2.el7_7.2.x86_64
Dec 05 19:06:50 Updated: dracut-033-564.el7.x86_64
Dec 05 19:06:50 Updated: kmod-20-25.el7.x86_64
Dec 05 19:06:50 Updated: 7:device-mapper-libs-1.02.158-2.el7_7.2.x86_64
Dec 05 19:06:50 Updated: cryptsetup-libs-2.0.3-5.el7.x86_64
Dec 05 19:06:50 Updated: systemd-libs-219-67.el7_7.2.x86_64
Dec 05 19:06:50 Updated: elfutils-libs-0.176-2.el7.x86_64
Dec 05 19:06:55 Updated: systemd-219-67.el7_7.2.x86_64
Dec 05 19:06:56 Updated: elfutils-default-yama-scope-0.176-2.el7.noarch
Dec 05 19:06:57 Updated: initscripts-9.49.47-1.el7.x86_64
Dec 05 19:06:57 Updated: systemd-sysv-219-67.el7_7.2.x86_64
Dec 05 19:06:57 Updated: 1:grub2-tools-minimal-2.02-0.80.el7.centos.x86_64
Dec 05 19:06:57 Updated: policycoreutils-2.5-33.el7.x86_64
Dec 05 19:06:57 Updated: selinux-policy-3.13.1-252.el7_7.6.noarch
Dec 05 19:06:58 Updated: 1:grub2-tools-2.02-0.80.el7.centos.x86_64
Dec 05 19:06:58 Updated: cronie-anacron-1.4.11-23.el7.x86_64
Dec 05 19:06:58 Updated: cronie-1.4.11-23.el7.x86_64
Dec 05 19:06:58 Updated: polkit-0.112-22.el7_7.1.x86_64
Dec 05 19:06:59 Updated: 1:NetworkManager-libnm-1.18.0-5.el7_7.1.x86_64
Dec 05 19:06:59 Updated: 1:NetworkManager-1.18.0-5.el7_7.1.x86_64
Dec 05 19:06:59 Updated: 12:dhcp-libs-4.2.5-77.el7.centos.x86_64
Dec 05 19:07:00 Updated: openssh-7.4p1-21.el7.x86_64
Dec 05 19:07:00 Updated: 12:dhcp-common-4.2.5-77.el7.centos.x86_64
Dec 05 19:07:00 Updated: 1:NetworkManager-wwan-1.18.0-5.el7_7.1.x86_64
Dec 05 19:07:00 Updated: 1:grub2-tools-extra-2.02-0.80.el7.centos.x86_64
Dec 05 19:07:00 Updated: 1:grub2-pc-2.02-0.80.el7.centos.x86_64
Dec 05 19:07:13 Updated: selinux-policy-targeted-3.13.1-252.el7_7.6.noarch
Dec 05 19:07:13 Updated: rpcbind-0.2.0-48.el7.x86_64
Dec 05 19:07:13 Updated: rsyslog-8.24.0-41.el7_7.2.x86_64
Dec 05 19:07:13 Updated: libdrm-2.4.97-2.el7.x86_64
Dec 05 19:07:13 Updated: libcgroup-0.41-21.el7.x86_64
Dec 05 19:07:14 Updated: policycoreutils-python-2.5-33.el7.x86_64
Dec 05 19:07:27 Updated: 1:google-compute-engine-oslogin-20191014.00-g1.el7.x86_64
Dec 05 19:07:27 Updated: ntpdate-4.2.6p5-29.el7.centos.x86_64
Dec 05 19:07:27 Updated: plymouth-core-libs-0.8.9-0.32.20140113.el7.centos.x86_64
Dec 05 19:07:27 Updated: plymouth-scripts-0.8.9-0.32.20140113.el7.centos.x86_64
Dec 05 19:07:27 Updated: plymouth-0.8.9-0.32.20140113.el7.centos.x86_64
Dec 05 19:07:27 Updated: 1:make-3.82-24.el7.x86_64
Dec 05 19:07:28 Updated: cpp-4.8.5-39.el7.x86_64
Dec 05 19:07:30 Updated: gcc-4.8.5-39.el7.x86_64
Dec 05 19:07:42 Updated: linux-firmware-20190429-72.gitddde598.el7.noarch
Dec 05 19:07:42 Updated: 1:quota-nls-4.01-19.el7.noarch
Dec 05 19:07:42 Updated: 1:quota-4.01-19.el7.x86_64
Dec 05 19:07:48 Installed: kernel-3.10.0-1062.7.1.el7.x86_64
Dec 05 19:07:49 Updated: gcc-c++-4.8.5-39.el7.x86_64
Dec 05 19:07:49 Updated: 1:openssl-1.0.2k-19.el7.x86_64
Dec 05 19:07:49 Updated: ntp-4.2.6p5-29.el7.centos.x86_64
Dec 05 19:07:50 Updated: 1:google-compute-engine-20190916.00-g2.el7.noarch
Dec 05 19:07:50 Updated: 32:bind-9.11.4-9.P2.el7.x86_64
Dec 05 19:07:50 Updated: 1:grub2-2.02-0.80.el7.centos.x86_64
Dec 05 19:07:50 Updated: 1:NetworkManager-bluetooth-1.18.0-5.el7_7.1.x86_64
Dec 05 19:07:50 Updated: 12:dhclient-4.2.5-77.el7.centos.x86_64
Dec 05 19:07:51 Updated: openssh-server-7.4p1-21.el7.x86_64
Dec 05 19:07:51 Updated: openssh-clients-7.4p1-21.el7.x86_64
Dec 05 19:07:51 Updated: 1:NetworkManager-tui-1.18.0-5.el7_7.1.x86_64
Dec 05 19:07:51 Updated: 1:NetworkManager-wifi-1.18.0-5.el7_7.1.x86_64
Dec 05 19:07:51 Updated: 1:NetworkManager-ppp-1.18.0-5.el7_7.1.x86_64
Dec 05 19:07:51 Updated: 1:NetworkManager-adsl-1.18.0-5.el7_7.1.x86_64
Dec 05 19:07:51 Updated: 1:NetworkManager-team-1.18.0-5.el7_7.1.x86_64
Dec 05 19:07:52 Updated: tuned-2.11.0-5.el7_7.1.noarch
Dec 05 19:07:52 Updated: yum-cron-3.4.3-163.el7.centos.noarch
Dec 05 19:07:52 Updated: dnsmasq-2.76-10.el7_7.1.x86_64
Dec 05 19:07:52 Updated: audit-2.8.5-4.el7.x86_64
Dec 05 19:07:52 Updated: systemd-devel-219-67.el7_7.2.x86_64
Dec 05 19:07:52 Updated: net-tools-2.0-0.25.20131004git.el7.x86_64
Dec 05 19:07:53 Updated: hwdata-0.252-9.3.el7.x86_64
Dec 05 19:07:53 Updated: firewalld-0.6.3-2.el7_7.2.noarch
Dec 05 19:07:53 Updated: biosdevname-0.7.3-2.el7.x86_64
Dec 05 19:07:53 Updated: libgudev1-219-67.el7_7.2.x86_64
Dec 05 19:07:54 Updated: 1:NetworkManager-glib-1.18.0-5.el7_7.1.x86_64
Dec 05 19:07:54 Updated: parted-3.1-31.el7.x86_64
Dec 05 19:07:54 Updated: dracut-config-rescue-033-564.el7.x86_64
Dec 05 19:07:54 Updated: 2:vim-enhanced-7.4.629-6.el7.x86_64
Dec 05 19:07:54 Updated: numactl-devel-2.0.12-3.el7_7.1.x86_64
Dec 05 19:07:54 Updated: kernel-tools-3.10.0-1062.7.1.el7.x86_64
Dec 05 19:07:55 Updated: sudo-1.8.23-4.el7_7.1.x86_64
Dec 05 19:07:55 Updated: 1:openssl-devel-1.0.2k-19.el7.x86_64
Dec 05 19:07:56 Updated: e2fsprogs-1.42.9-16.el7.x86_64
Dec 05 19:08:01 Updated: 2:nmap-6.40-19.el7.x86_64
Dec 05 19:08:01 Updated: 32:bind-utils-9.11.4-9.P2.el7.x86_64
Dec 05 19:08:01 Updated: python-requests-2.6.0-7.el7_7.noarch
Dec 05 19:08:01 Updated: nss-tools-3.44.0-4.el7.x86_64
Dec 05 19:08:01 Updated: passwd-0.79-5.el7.x86_64
Dec 05 19:08:12 Updated: google-cloud-sdk-272.0.0-1.noarch
Dec 05 19:08:13 Updated: systemtap-sdt-devel-4.0-10.el7_7.x86_64
Dec 05 19:08:13 Updated: xfsprogs-4.5.0-20.el7.x86_64
Dec 05 19:08:13 Updated: alsa-lib-1.1.8-1.el7.x86_64
Dec 05 19:08:13 Updated: gdisk-0.8.10-3.el7.x86_64
Dec 05 19:08:14 Updated: libicu-50.2-3.el7.x86_64
Dec 05 19:08:14 Updated: libdb-devel-5.3.21-25.el7.x86_64
Dec 05 19:08:14 Updated: libcap-devel-2.22-10.el7.x86_64
Dec 05 19:08:14 Updated: libjpeg-turbo-1.2.90-8.el7.x86_64
Dec 05 19:08:14 Updated: 1:dmidecode-3.2-3.el7.x86_64
Dec 05 19:08:14 Updated: libX11-1.6.7-2.el7.x86_64
Dec 05 19:08:14 Updated: psmisc-22.20-16.el7.x86_64
Dec 05 19:08:14 Updated: unzip-6.0-20.el7.x86_64
Dec 05 19:08:15 Updated: iprutils-2.4.17.1-2.el7.x86_64
Dec 05 19:08:15 Updated: patch-2.7.1-12.el7_7.x86_64
Dec 05 19:08:15 Updated: epel-release-7-12.noarch
Dec 05 19:08:15 Updated: iwl3160-firmware-22.0.7.0-72.el7.noarch
Dec 05 19:08:15 Updated: iwl135-firmware-18.168.6.1-72.el7.noarch
Dec 05 19:08:15 Updated: iwl105-firmware-18.168.6.1-72.el7.noarch
Dec 05 19:08:15 Updated: iwl2000-firmware-18.168.6.1-72.el7.noarch
Dec 05 19:08:16 Updated: iwl7260-firmware-22.0.7.0-72.el7.noarch
Dec 05 19:08:16 Updated: iwl2030-firmware-18.168.6.1-72.el7.noarch
Dec 05 19:08:16 Updated: iwl6000g2b-firmware-17.168.5.2-72.el7.noarch
Dec 05 19:08:16 Updated: perl-Archive-Tar-1.92-3.el7.noarch

I created another VM instance from the snapshots, this time I did this first:
Code: 
cd /usr/local/directadmin/custombuild
./build update
./build all d
./build rewrite_confs

outbound TCP traffic working.

Then I did yum update:

Working fine.

Except, I would use this instant but for some reason all of the sites on the server are serving the wrong page (always showing default apache is working page), also SSL is invalid, wrong domain name. (mayde due to IP address change?)

I checked the firewall settings for Google compute engine, all outbound traffic is allowed.
 
Do you have both firewalld and csf on? I would use one or the other?
 
I had CSF, disabled, uninstalled it.
firewalld enabled/disabled/stop
iptables -F -X
nothing made any difference.

Ok, so I moved my external static ip address from the production VM to the snapshot VM with working TCP, rebooted and now TCP blocked again.

Now I am thinking something is wrong with that ip address.
I've contacted google support, waiting for them to respond...
 
Well I got the snapshots of before the updates working, I won't be able to update the server for now but at least it gives me time to do more testing.
I think it would be easier just to re-build another server now that time is on my side.

I really appreciate you guys taking the time to help, thank you all. (btw no response from Google yet)
 
You must log in or register to reply here.
Back
Top