kswapd0 takes 99% to 100% CPU in RHEL 6.3-Collection of common programming errors

11 years ago
admin
11 minutes

I have 5 systems on which I have just installed RHEL6. System configuration is

CPU-i7

HDD-500 GB

RAM-2 GB

Swap- 4 GB

I have installed the same software’s and all the configuration as per old systems.

But my New systems running at low speed!

top command output is as below:

[root@localhost ~]# top

top - 12:27:56 up 24 min,  3 users,  load average: 1.00, 1.00, 0.82
Tasks: 214 total,   2 running, 212 sleeping,   0 stopped,   0 zombie
Cpu(s):  1.1%us, 13.0%sy,  0.0%ni, 85.9%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st

I don’t have such problem in my old systems which are having less configuration than new systems!

Update 1:

I agree with Michael and Vonbrand.. But I still have my system very very slow. Specially when I am compiling something in my QtCreator. Even my mouse movement got slow.

I checked by adding comment before swap partition in /etc/fstab and rebooting, The swap shows 0.0GB but kswapd0 still using 99% to 100%.

Why other processors are not utilizing full!

My RAM (2GB) is less

OR

Is there any service which m*Update 2:*anages CPU is not started?

Update 2: The output of dmesg is

  ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
  alloc irq_desc for 16 on node -1
  alloc kstat_irqs on node -1
alloc irq_2_iommu on node -1
ehci_hcd 0000:00:1a.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
ehci_hcd 0000:00:1a.0: setting latency timer to 64
ehci_hcd 0000:00:1a.0: EHCI Host Controller
ehci_hcd 0000:00:1a.0: new USB bus registered, assigned bus number 1
ehci_hcd 0000:00:1a.0: debug port 2
ehci_hcd 0000:00:1a.0: cache line size of 64 is not supported
ehci_hcd 0000:00:1a.0: irq 16, io mem 0xf7c38000
ehci_hcd 0000:00:1a.0: USB 2.0 started, EHCI 1.00
usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
usb usb1: Product: EHCI Host Controller
usb usb1: Manufacturer: Linux 2.6.32-71.el6.x86_64 ehci_hcd
usb usb1: SerialNumber: 0000:00:1a.0
usb usb1: configuration #1 chosen from 1 choice
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 3 ports detected
  alloc irq_desc for 23 on node -1
  alloc kstat_irqs on node -1
alloc irq_2_iommu on node -1
ehci_hcd 0000:00:1d.0: PCI INT A -> GSI 23 (level, low) -> IRQ 23
ehci_hcd 0000:00:1d.0: setting latency timer to 64
ehci_hcd 0000:00:1d.0: EHCI Host Controller
ehci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 2
ehci_hcd 0000:00:1d.0: debug port 2
ehci_hcd 0000:00:1d.0: cache line size of 64 is not supported
ehci_hcd 0000:00:1d.0: irq 23, io mem 0xf7c37000
ehci_hcd 0000:00:1d.0: USB 2.0 started, EHCI 1.00
usb usb2: New USB device found, idVendor=1d6b, idProduct=0002
usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
usb usb2: Product: EHCI Host Controller
usb usb2: Manufacturer: Linux 2.6.32-71.el6.x86_64 ehci_hcd
usb usb2: SerialNumber: 0000:00:1d.0
usb usb2: configuration #1 chosen from 1 choice
hub 2-0:1.0: USB hub found
hub 2-0:1.0: 3 ports detected
ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
uhci_hcd: USB Universal Host Controller Interface driver
PNP: PS/2 Controller [PNP0303:PS2K,PNP0f03:PS2M] at 0x60,0x64 irq 1,12
serio: i8042 KBD port at 0x60,0x64 irq 1
serio: i8042 AUX port at 0x60,0x64 irq 12
mice: PS/2 mouse device common for all mice
rtc_cmos 00:06: RTC can wake from S4
rtc_cmos 00:06: rtc core: registered rtc_cmos as rtc0
rtc0: alarms up to one month, y3k, 242 bytes nvram, hpet irqs
cpuidle: using governor ladder
cpuidle: using governor menu
usbcore: registered new interface driver hiddev
usbcore: registered new interface driver usbhid
usbhid: v2.6:USB HID core driver
TCP cubic registered
Initializing XFRM netlink socket
NET: Registered protocol family 17
registered taskstats version 1
IMA: No TPM chip found, activating TPM-bypass!
rtc_cmos 00:06: setting system clock to 2013-03-04 14:10:29 UTC (1362406229)
Initalizing network drop monitor service
Freeing unused kernel memory: 1220k freed
Write protecting the kernel read-only data: 7176k
dracut: dracut-004-32.el6
dracut: rd_NO_LUKS: removing cryptoluks activation
dracut: rd_NO_LVM: removing LVM activation
device-mapper: uevent: version 1.0.3
device-mapper: ioctl: 4.17.0-ioctl (2010-03-05) initialised: [email protected]
udev: starting version 147
dracut: Starting plymouth daemon
dracut: rd_NO_DM: removing DM RAID activation
dracut: rd_NO_MD: removing MD RAID activation
ahci 0000:00:1f.2: version 3.0
ahci 0000:00:1f.2: PCI INT B -> GSI 19 (level, low) -> IRQ 19
  alloc irq_desc for 31 on node -1
  alloc kstat_irqs on node -1
alloc irq_2_iommu on node -1
ahci 0000:00:1f.2: irq 31 for MSI/MSI-X
ahci 0000:00:1f.2: AHCI 0001.0300 32 slots 6 ports 6 Gbps 0x5 impl SATA mode
ahci 0000:00:1f.2: flags: 64bit ncq pm led clo pio slum part ems apst 
ahci 0000:00:1f.2: setting latency timer to 64
scsi0 : ahci
scsi1 : ahci
scsi2 : ahci
scsi3 : ahci
scsi4 : ahci
scsi5 : ahci
ata1: SATA max UDMA/133 abar m2048@0xf7c36000 port 0xf7c36100 irq 31
ata2: DUMMY
ata3: SATA max UDMA/133 abar m2048@0xf7c36000 port 0xf7c36200 irq 31
ata4: DUMMY
ata5: DUMMY
ata6: DUMMY
usb 1-1: new high speed USB device using ehci_hcd and address 2
usb 1-1: New USB device found, idVendor=8087, idProduct=0024
usb 1-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
usb 1-1: configuration #1 chosen from 1 choice
hub 1-1:1.0: USB hub found
hub 1-1:1.0: 6 ports detected
ata3: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
ata1: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
ata1.00: ACPI _SDD failed (AE 0x5)
ata3.00: ACPI _SDD failed (AE 0x5)
usb 2-1: new high speed USB device using ehci_hcd and address 2
usb 2-1: New USB device found, idVendor=8087, idProduct=0024
usb 2-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
usb 2-1: configuration #1 chosen from 1 choice
hub 2-1:1.0: USB hub found
hub 2-1:1.0: 8 ports detected
usb 2-1.3: new low speed USB device using ehci_hcd and address 3
usb 2-1.3: New USB device found, idVendor=04ca, idProduct=0062
usb 2-1.3: New USB device strings: Mfr=0, Product=2, SerialNumber=0
usb 2-1.3: Product: USB Optical Mouse
usb 2-1.3: configuration #1 chosen from 1 choice
input: USB Optical Mouse as /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.3/2-1.3:1.0/input/input3
generic-usb 0003:04CA:0062.0001: input,hidraw0: USB HID v1.11 Mouse [USB Optical Mouse] on usb-0000:00:1d.0-1.3/input0
usb 2-1.4: new low speed USB device using ehci_hcd and address 4
usb 2-1.4: New USB device found, idVendor=03f0, idProduct=0024
usb 2-1.4: New USB device strings: Mfr=1, Product=2, SerialNumber=0
usb 2-1.4: Product: HP Basic USB Keyboard
usb 2-1.4: Manufacturer: CHICONY
usb 2-1.4: configuration #1 chosen from 1 choice
input: CHICONY HP Basic USB Keyboard as /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.4/2-1.4:1.0/input/input4
generic-usb 0003:03F0:0024.0002: input,hidraw1: USB HID v1.11 Keyboard [CHICONY HP Basic USB Keyboard] on usb-0000:00:1d.0-1.4/input0
ata3: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
ata3.00: ACPI _SDD failed (AE 0x5)
ata3.00: ACPI: failed the second time, disabled
ata1: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
ata3.00: ATAPI: hp      DVD-RAM GH82N, R204, max UDMA/100, ATAPI AN
ata1.00: ACPI _SDD failed (AE 0x5)
ata1.00: ACPI: failed the second time, disabled
ata1.00: ATA-8: ST500DM002-1BD142, HP73, max UDMA/100
ata1.00: 976773168 sectors, multi 16: LBA48 NCQ (depth 31/32)
ata1.00: configured for UDMA/100
ata3.00: configured for UDMA/100
scsi 0:0:0:0: Direct-Access     ATA      ST500DM002-1BD14 HP73 PQ: 0 ANSI: 5
scsi 2:0:0:0: CD-ROM            hp       DVD-RAM GH82N    R204 PQ: 0 ANSI: 5
sd 0:0:0:0: [sda] 976773168 512-byte logical blocks: (500 GB/465 GiB)
sd 0:0:0:0: [sda] 4096-byte physical blocks
sd 0:0:0:0: [sda] Write Protect is off
sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00
sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
 sda: sda1 sda2 sda3 sda4 < sda5 sda6 sda7 >
sd 0:0:0:0: [sda] Attached SCSI disk
sr0: scsi3-mmc drive: 40x/40x writer dvd-ram cd/rw xa/form2 cdda tray
Uniform CD-ROM driver Revision: 3.20
sr 2:0:0:0: Attached scsi CD-ROM sr0
EXT4-fs (sda7): INFO: recovery required on readonly filesystem
EXT4-fs (sda7): write access will be enabled during recovery
EXT4-fs (sda7): orphan cleanup on readonly fs
EXT4-fs (sda7): ext4_orphan_cleanup: deleting unreferenced inode 15341115
EXT4-fs (sda7): ext4_orphan_cleanup: deleting unreferenced inode 1835089
EXT4-fs (sda7): ext4_orphan_cleanup: deleting unreferenced inode 1835088
EXT4-fs (sda7): ext4_orphan_cleanup: deleting unreferenced inode 1835087
EXT4-fs (sda7): ext4_orphan_cleanup: deleting unreferenced inode 1835085
EXT4-fs (sda7): ext4_orphan_cleanup: deleting unreferenced inode 1835081
EXT4-fs (sda7): ext4_orphan_cleanup: deleting unreferenced inode 1835080
EXT4-fs (sda7): ext4_orphan_cleanup: deleting unreferenced inode 1835079
EXT4-fs (sda7): ext4_orphan_cleanup: deleting unreferenced inode 1835076
EXT4-fs (sda7): ext4_orphan_cleanup: deleting unreferenced inode 1835075
EXT4-fs (sda7): ext4_orphan_cleanup: deleting unreferenced inode 1835067
EXT4-fs (sda7): 11 orphan inodes deleted
EXT4-fs (sda7): recovery complete
EXT4-fs (sda7): mounted filesystem with ordered data mode
dracut: Mounted root filesystem /dev/sda7
dracut: Loading SELinux policy
SELinux:  Disabled at runtime.
SELinux:  Unregistering netfilter hooks
type=1404 audit(1362406240.622:2): selinux=0 auid=4294967295 ses=4294967295
dracut: /sbin/load_policy: Can't load policy: No such file or directory
dracut: Switching root
udev: starting version 147
e1000e: Intel(R) PRO/1000 Network Driver - 1.2.7-k2
e1000e: Copyright (c) 1999 - 2009 Intel Corporation.
  alloc irq_desc for 20 on node -1
  alloc kstat_irqs on node -1
alloc irq_2_iommu on node -1
e1000e 0000:00:19.0: PCI INT A -> GSI 20 (level, low) -> IRQ 20
e1000e 0000:00:19.0: setting latency timer to 64
  alloc irq_desc for 32 on node -1
  alloc kstat_irqs on node -1
alloc irq_2_iommu on node -1
e1000e 0000:00:19.0: irq 32 for MSI/MSI-X
0000:00:19.0: eth0: (PCI Express:2.5GB/s:Width x1) 6c:3b:e5:13:26:21
0000:00:19.0: eth0: Intel(R) PRO/1000 Network Connection
0000:00:19.0: eth0: MAC: 10, PHY: 11, PBA No: 0100ff-0ff
sd 0:0:0:0: Attached scsi generic sg0 type 0
sr 2:0:0:0: Attached scsi generic sg1 type 5
ACPI: WMI: Mapper loaded
EXT4-fs (sda5): mounted filesystem with ordered data mode
microcode: CPU0 sig=0x306a9, pf=0x2, revision=0x12
platform microcode: firmware: requesting intel-ucode/06-3a-09
microcode: CPU1 sig=0x306a9, pf=0x2, revision=0x12
platform microcode: firmware: requesting intel-ucode/06-3a-09
microcode: CPU2 sig=0x306a9, pf=0x2, revision=0x12
platform microcode: firmware: requesting intel-ucode/06-3a-09
microcode: CPU3 sig=0x306a9, pf=0x2, revision=0x12
platform microcode: firmware: requesting intel-ucode/06-3a-09
microcode: CPU4 sig=0x306a9, pf=0x2, revision=0x12
platform microcode: firmware: requesting intel-ucode/06-3a-09
microcode: CPU5 sig=0x306a9, pf=0x2, revision=0x12
platform microcode: firmware: requesting intel-ucode/06-3a-09
microcode: CPU6 sig=0x306a9, pf=0x2, revision=0x12
platform microcode: firmware: requesting intel-ucode/06-3a-09
microcode: CPU7 sig=0x306a9, pf=0x2, revision=0x12
platform microcode: firmware: requesting intel-ucode/06-3a-09
Microcode Update Driver: v2.00 , Peter Oruba
Microcode Update Driver: v2.00 removed.
ip_tables: (C) 2000-2006 Netfilter Core Team
NET: Registered protocol family 10
lo: Disabled Privacy Extensions
e1000e 0000:00:19.0: irq 32 for MSI/MSI-X
e1000e 0000:00:19.0: irq 32 for MSI/MSI-X
ADDRCONF(NETDEV_UP): eth0: link is not ready
e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
eth0: no IPv6 routers present
RPC: Registered udp transport module.
RPC: Registered tcp transport module.
RPC: Registered tcp NFSv4.1 backchannel transport module.
Bridge firewalling registered
virbr0: starting userspace STP failed, starting kernel STP
nf_conntrack version 0.5.0 (16384 buckets, 65536 max)
Ebtables v2.0 registered
fuse init (API version 7.13)
ip6_tables: (C) 2000-2006 Netfilter Core Team
lo: Disabled Privacy Extensions
tun: Universal TUN/TAP device driver, 1.6
tun: (C) 1999-2004 Max Krasnyansky 
device vnet0 entered promiscuous mode
virbr0: topology change detected, propagating
virbr0: port 1(vnet0) entering forwarding state
vnet0: no IPv6 routers present
kvm: 3639: cpu0 unimplemented perfctr wrmsr: 0xc1 data 0x0
kvm: 3639: cpu0 unimplemented perfctr wrmsr: 0x186 data 0x130079
kvm: 3639: cpu0 unimplemented perfctr wrmsr: 0xc1 data 0xffcc3e1a
kvm: 3639: cpu0 unimplemented perfctr wrmsr: 0x186 data 0x530079
kvm: 3639: cpu1 unimplemented perfctr wrmsr: 0xc1 data 0x0
kvm: 3639: cpu1 unimplemented perfctr wrmsr: 0x186 data 0x130079
kvm: 3639: cpu1 unimplemented perfctr wrmsr: 0xc1 data 0xffcc3e1a
kvm: 3639: cpu1 unimplemented perfctr wrmsr: 0x186 data 0x530079
kvm: 3639: cpu2 unimplemented perfctr wrmsr: 0xc1 data 0x0
virbr0: port 1(vnet0) entering disabled state
device vnet0 left promiscuous mode
virbr0: port 1(vnet0) entering disabled state

I didn’t noticed anything related to swap here.

Update 3: See my Answer Below:

  1. Kernel Bug: I think my problem is with kernel as noticed by goldilocks. As I increased my RAM to 4GB all processes are stable, even with core 2 intel processor box with 2GB RAM is also working fine.

  2. Like @vonbrand points out, the CPU usage percentage is by default based on a single CPU. So if a process has multiple threads, on a multi-core or multi-CPU system this number can very easily exceed 100%.

    Take a look at the totals found in the Cpu(s) line near the top; it says that your system was spending 1.1% of its time in userspace, 13.0% in system code (basically the kernel), and 85.9% idle. This aligns well with a multi-core CPU where one core is operating at 100%.

    You can switch top between so-called Irix mode and Solaris mode by pressing I. With Irix mode off, the CPU usage displayed will be across all CPUs and/or cores, whereas with Irix mode on, one fully utilized core will display as 100% even on a fully-equipped multiprocessor system. Hence, with Irix mode off, on a hypothetical 10-core system with two cores fully utilized and the others fully idle, CPU usage will be shown as the expected 20% rather than 200%.

  3. The % CPU is on basis of one CPU, so it can very well be several times 100%. And load averages (the mean number of processes waiting for CPU) is low, the machine is almost 90% idle. No reason to worry.