Details
-
Type: Bug
-
Status: Closed
-
Priority: Minor
-
Resolution: Fixed
-
Fix Version/s: OpenVZ-legacy
-
Component/s: Containers::Kernel
-
Security Level: Public
-
Environment:Operating System: Other
Platform: x86_64 (AMD64)
-
External issue URL:
-
External issue ID:2255
Description
Getting a kernel panic every week:
[root@vz ~]# uname -a
Linux vz.xyz.com 2.6.32-042stab049.6 #1 SMP Mon Feb 6 19:17:43 MSK 2012 x86_64 x86_64 x86_64 GNU/Linux
[root@vz ~]# cat /etc/redhat-release
CentOS release 6.2 (Final)
This is as much as my terminal server logs captured:
th xt_helper xt_esp xt_CONNSECMARK xt_comment nfnetlink ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables arc4 ecb ppp_mppe ppp_deflate zlib_deflate ppp_async ppp_generic slhc crc_ccitt serio_raw k8temp amd64_edac_mod edac_core edac_mce_amd sg i2c_nforce2 i2c_core ext4 mbcache jbd2 raid1 sr_mod cdrom sd_mod crc_t10dif forcedeth pata_amd ata_generic pata_acpi sata_nv dm_mirror dm_region_hash dm_log dm_mod be2iscsi bnx2i cnic uio ipv6 cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio libiscsi_tcp qla4xxx iscsi_boot_sysfs libiscsi scsi_transport_iscsi [last unloaded: scsi_wait_scan]
[597276.142067] CPU 1
[597276.142067] Modules linked in: vzethdev simfs vzrst vzcpt nfs lockd fscache nfs_acl auth_rpcgss sunrpc vzdquota ip6table_mangle nf_nat_ftp vzevent vznetdev vzmon vzdev xt_sctp xt_realm xt_NOTRACK xt_dccp xt_CONNMARK xt_connmark xt_connbytes xt_CLASSIFY nfnetlink_queue nfnetlink_log xt_recent xt_DSCP nf_nat_irc xt_conntrack ipt_LOG nf_conntrack_irc nf_conntrack_ftp xt_hl xt_tcpmss xt_TCPMSS xt_dscp ipt_REDIRECT ipt_REJECT tun nf_conntrack_netbios_ns nf_conntrack_broadcast iptable_nat nf_nat iptable_mangle nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables xt_string xt_statistic xt_SECMARK xt_quota xt_policy xt_pkttype xt_physdev xt_NFQUEUE xt_multiport xt_MARK xt_mark xt_mac xt_limit xt_length xt_helper xt_esp xt_CONNSECMARK xt_comment nfnetlink ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables arc4 ecb ppp_mppe ppp_deflate zlib_deflate ppp_async ppp_generic slhc crc_ccitt serio_raw k8temp amd64_edac_mod edac_core edac_mce_amd sg i2c_nforce2 i2c_core ext4 mbcache jbd2 raid1 sr_mod cdrom sd_mod crc_t10dif forcedeth pata_amd ata_generic pata_acpi sata_nv dm_mirror dm_region_hash dm_log dm_mod be2iscsi bnx2i cnic uio ipv6 cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio libiscsi_tcp qla4xxx iscsi_boot_sysfs libiscsi scsi_transport_iscsi [last unloaded: scsi_wait_scan]
[597276.142067]
[597276.142067] Pid: 946200, comm: set_keyserver_u veid: 1035 Not tainted 2.6.32-042stab049.6 #1 042stab049_6 empty empty/S2927
[597276.142067] RIP: 0010:[<ffffffff814e191e>] [<ffffffff814e191e>] _spin_lock+0x1e/0x30
[597276.142067] RSP: 0018:ffff88000babfe98 EFLAGS: 00000293
[597276.142067] RAX: 0000000000003c92 RBX: ffff88000babfe98 RCX: 0000000000000000
[597276.142067] RDX: 0000000000003c90 RSI: ffffffff817a1438 RDI: ffffffff823cca00
[597276.142067] RBP: ffffffff8100bcce R08: 0000000000000000 R09: 000000000000000e
[597276.142067] R10: 0000000000000000 R11: 0000000000000246 R12: ffffffff81042b54
[597276.142067] R13: ffff88000babff18 R14: 0000000000000014 R15: ffff8803c5c004a8
[597276.142067] FS: 00007f54f714b6e0(0000) GS:ffff880028280000(0000) knlGS:00000000b774f6c0
[597276.142067] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[597276.142067] CR2: 00007f54f66747e0 CR3: 00000002a8a5b000 CR4: 00000000000006e0
[597276.142067] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[597276.142067] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[597276.142067] Process set_keyserver_u (pid: 946200, veid: 1035, threadinfo ffff88000babe000, task ffff8800ad7b3290)
[597276.142067] Stack:
[597276.142067] ffff88000babfec8 ffffffff811a93fb 000000001d12a781 ffff880336840980
[597276.142067] <0> 0000000000000000 0000000000000000 ffff88000babfed8 ffffffff81410fd9
[597276.142067] <0> ffff88000babff28 ffffffff8141143e ffff880000000001 ffff880000000002
[597276.142067] Call Trace:
[597276.142067] [<ffffffff811a93fb>] ? new_inode+0x3b/0xb0
[597276.142067] [<ffffffff81410fd9>] ? sock_alloc+0x19/0x60
[597276.142067] [<ffffffff8141143e>] ? __sock_create+0x9e/0x210
[597276.142067] [<ffffffff81411610>] ? sock_create+0x30/0x40
[597276.142067] [<ffffffff8141182c>] ? sys_socket+0x3c/0x80
[597276.142067] [<ffffffff8100b182>] ? system_call_fastpath+0x16/0x1b
[597276.142067] Code: 00 00 00 01 74 05 e8 12 d1 d8 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89
[597276.142067] Call Trace:
[597276.142067] [<ffffffff811a93fb>] ? new_inode+0x3b/0xb0
[597276.142067] [<ffffffff81410fd9>] ? sock_alloc+0x19/0x60
[597276.142067] [<ffffffff8141143e>] ? __sock_create+0x9e/0x210
[597276.142067] [<ffffffff81411610>] ? sock_create+0x30/0x40
[597276.142067] [<ffffffff8141182c>] ? sys_socket+0x3c/0x80
[597276.142067] [<ffffffff8100b182>] ? system_call_fastpath+0x16/0x1b
[597276.206006] BUG: soft lockup - CPU#2 stuck for 67s! [update_deploy_p:946201]
[597276.206009] Modules linked in: vzethdev simfs vzrst vzcpt nfs lockd fscache nfs_acl auth_rpcgss sunrpc vzdquota ip6table_mangle nf_nat_ftp vzevent vznetdev vzmon vzdev xt_sctp xt_realm xt_NOTRACK xt_dccp xt_CONNMARK xt_connmark xt_connbytes xt_CLASSIFY nfnetlink_queue nfnetlink_log xt_recent xt_DSCP nf_nat_irc xt_conntrack ipt_LOG nf_conntrack_irc nf_conntrack_ftp xt_hl xt_tcpmss xt_TCPMSS xt_dscp ipt_REDIRECT ipt_REJECT tun nf_conntrack_netbios_ns nf_conntrack_broadcast iptable_nat nf_nat iptable_mangle nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables xt_string xt_statistic xt_SECMARK xt_quota xt_policy xt_pkttype xt_physdev xt_NFQUEUE xt_multiport xt_MARK xt_mark xt_mac xt_limit xt_length xt_helper xt_esp xt_CONNSECMARK xt_comment nfnetlink ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables arc4 ecb ppp_mppe ppp_deflate zlib_deflate ppp_async ppp_generic slhc crc_ccitt serio_raw k8temp amd64_edac_mod edac_core edac_mce_amd sg i2c_nforce2 i2c_core ext4
mbcache jbd2 raid1 sr_mod cdrom sd_mod crc_t10dif forcedeth pata_amd ata_generic pata_acpi sata_nv dm_mirror dm_region_hash dm_log dm_mod be2iscsi bnx2i cnic uio ipv6 cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio libiscsi_tcp qla4xxx iscsi_boot_sysfs libiscsi scsi_transport_iscsi [last unloaded: scsi_wait_scan]
[597276.206085] CPU 2
[597276.206087] Modules linked in: vzethdev simfs vzrst vzcpt nfs lockd fscache nfs_acl auth_rpcgss sunrpc vzdquota ip6table_mangle nf_nat_ftp vzevent vznetdev vzmon vzdev xt_sctp xt_realm xt_NOTRACK xt_dccp xt_CONNMARK xt_connmark xt_connbytes xt_CLASSIFY nfnetlink_queue nfnetlink_log xt_recent xt_DSCP nf_nat_irc xt_conntrack ipt_LOG nf_conntrack_irc nf_conntrack_ftp xt_hl xt_tcpmss xt_TCPMSS xt_dscp ipt_REDIRECT ipt_REJECT tun nf_conntrack_netbios_ns nf_conntrack_broadcast iptable_nat nf_nat iptable_mangle nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables xt_string xt_statistic xt_SECMARK xt_quota xt_policy xt_pkttype xt_physdev xt_NFQUEUE xt_multiport xt_MARK xt_mark xt_mac xt_limit xt_length xt_helper xt_esp xt_CONNSECMARK xt_comment nfnetlink ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables arc4 ecb ppp_mppe ppp_deflate zlib_deflate ppp_async ppp_generic slhc crc_ccitt serio_raw k8temp amd64_edac_mod edac_core edac_mce_amd sg i2c_nforce2 i2c_core ext4 mbcache jbd2 raid1 sr_mod cdrom sd_mod crc_t10dif forcedeth pata_amd ata_generic pata_acpi sata_nv dm_mirror dm_region_hash dm_log dm_mod be2iscsi bnx2i cnic uio ipv6 cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio libiscsi_tcp qla4xxx iscsi_boot_sysfs libiscsi scsi_transport_iscsi [last unloaded: scsi_wait_scan]
[597276.206133]
[597276.206133] Pid: 946201, comm: update_deploy_p veid: 1035 Not tainted 2.6.32-042stab049.6 #1 042stab049_6 empty empty/S2927
[597276.206133] RIP: 0010:[<ffffffff814e191e>] [<ffffffff814e191e>] _spin_lock+0x1e/0x30
[597276.206133] RSP: 0018:ffff8800ad635e98 EFLAGS: 00000297
[597276.206133] RAX: 0000000000003c94 RBX: ffff8800ad635e98 RCX: 0000000000000000
[597276.206133] RDX: 0000000000003c90 RSI: ffffffff817a1438 RDI: ffffffff823cca00
[597276.206133] RBP: ffffffff8100bcce R08: 0000000000000000 R09: 0000000000000003
[597276.206133] R10: 0000000000000000 R11: 0000000000000246 R12: ffffffff81042b54
[597276.206133] R13: ffff8800ad635f18 R14: 0000000000000014 R15: ffff8803151d8c78
[597276.206133] FS: 00007f4f0dc176e0(0000) GS:ffff880230400000(0000) knlGS:00000000b774f6c0
[597276.206133] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[597276.206133] CR2: 00007f4f0d1407e0 CR3: 00000002310a1000 CR4: 00000000000006e0
[597276.206133] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[597276.206133] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[597276.206133] Process update_deploy_p (pid: 946201, veid: 1035, threadinfo ffff8800ad634000, task ffff8803d03a4800)
[597276.206133] Stack:
[597276.206133] ffff8800ad635ec8 ffffffff811a93fb 000000001d12a781 ffff880336840980
[597276.206133] <0> 00 ppp_mppe ppp_deflate zlib_deflate ppp_async ppp_generic slhc crc_ccitt serio_raw k8temp amd64_edac_mod edac_core edac_mce_amd sg i2c_nforce2 i2c_core ext4 mbcache jbd2 raid1 sr_mod cdrom sd_mod crc_t10dif forcedeth pata_amd ata_generic pata_acpi sata_nv dm_mirror dm_region_hash dm_log dm_mod be2iscsi bnx2i cnic uio ipv6 cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio libiscsi_tcp qla4xxx iscsi_boot_sysfs libiscsi scsi_transport_iscsi [last unloaded: scsi_wait_scan]
[597276.268102] CPU 3
[597276.268103] Modules 0000(0000) knlGS:00000000b774f6c0
[597276.268192] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[597276.268192] CR2: 00007f246f7067e0 CR3: 0000000407807000 CR4: 00000000000006e0
[597276.268192] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[597276.268192] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[597276.268192] Process update_deploy_p (pid: 946188, veid: 1063, threadinfo ffff8800c232e000, task ffff8804239049c0)
[597276.268192] Stack:
[597276.268192] ffff8800c232fec8 ffffffff811a93fb 0000000009ffd874 ffff880225129200
[597276.268192] <0> 0000000000000000 0000000000000000 ffff8800c232fed8 ffffffff81410fd9
[597276.268192] <0> ffff8800c232ff28 ffffffff8141143e ffff880000000001 ffff880000000002
[597276.268192] Call Trace:
[597276.268192] [<ffffffff811a93fb>] ? new_inode+0x3b/0xb0
[597276.268192] [<ffffffff81410fd9>] ? sock_alloc+0x19/0x60
[597276.268192] [<ffffffff8141143e>] ? __sock_create+0x9e/0x210
[597276.268192] [<ffffffff81411610>] ? sock_crestring xt_statistic xt_SECMARK xt_quota xt_policy xt_pkttype xt_physdev xt_NFQUEUE xt_multiport xt_MARK xt_mark xt_mac xt_limit xt_length xt_helper xt_esp xt_CONNSECMARK xt_comment nfnetlink ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables arc4 ecb ppp_mppe ppp_deflate zlib_deflate ppp_async ppp_generic slhc crc_ccitt serio_raw k8temp amd64_edac_mod edac_core edac_mce_amd sg i2c_nforce2 i2c_core ext4 mbcache jbd2 raid1 sr_mod cdrom sd_mod crc_t10dif forcedeth pata_amd ata_generic pata_acpi sata_nv dm_mirror dm_region_hash dm_log dm_mod be2iscsi bnx2i cnic uio ipv6 cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio libiscsi_tcp qla4xxx iscsi_boot_sysfs libiscsi scsi_transport_iscsi [last unloaded: scsi_wait_scan]
[597276.079003]
[597276.079003] Pid: 390166, comm: apache2 veid: 1025 Not tainted 2.6.32-042stab049.6 #1 042stab049_6 empty empty/S2927
[597276.079003] RIP: 0010:[<ffffffff814e191e>] [<ffffffff814e191e>] _spin_lock+0x1e/0x30
[597276.079003] RSP: 0018:ffff8803cc2b df c9 c3 0f 1f 40 00 55 48 89
[597276.079003] Call Trace:
[597276.079003] [<ffffffff81261685>] ? _atomic_dec_and_lock+0x55/0x80
[597276.079003] [<ffffffff811a77b0>] ? iput+0x30/0x70
[597276.079003] [<ffffffff811a4028>] ? dentry_iput+0x98/0x110
[597276.079003] [<ffffffff811a419c>] ? d_kill+0x3c/0x70
[597276.079003] [<ffffffff811a5933>] ? dput+0xa3/0x1d0
[597276.079003] [<ffffffff8118e9e1>] ? __fput+0x1d1/0x280
[597276.079003] [<ffffffff8118eab5>] ? fput+0x25/0x30
[597276.079003] [<fffffff
[597360.268180] RIP: 0010:[<ffffffff814e1921>] [<ffffffff814e1921>] _spin_lock+0x21/0x30
[597360.268191] RSP: 0018:ffff8800c232fe98 EFLAGS: 00000293
[597360.268192] RAX: 0000000000003c93 RBX: ffff8800c232fe98 RCX: 0000000000000000
[597360.268192] RDX: 0000000000003c90 RSI: ffffffff817a1438 RDI: ffffffff823cca00
[597360.268192] RBP: ffffffff8100bcce R08: 0000000000000000 R09: 000000000000000f
[597360.268192] R10: 0000000000000000 R11: 0000000000000246 R12: ffffffff81042b54
[597360.268192] R13: ffff8800c232ff18 R14: 0000000000000014 R15: ffff880336b5ce90
[597360.268192] FS: 00007f24701dd6e0(0000) GS:ffff880230480000(0000) knlGS:00000000b774f6c0
[597360.268192] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[597360.268192] CR2: 00007f246f7067e0 CR3: 0000000407807000 CR4: 00000000000006e0
[597360.268192] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[597360.268192] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[597360.268192] Process update_deploy_p (pid: 946188, veid: 1063, threadinfo ffff8800c232e000, task ffff8804239049c0)
[597360.268192] Stack:
[597360.268192] ffff8800c232fec8 ffffffff811a93fb 0000000009ffd874 ffff880225129200
[597360.268192] <0> 0000000000000000 0000000000000000 ffff8800c232fed8 ffffffff81410fd9
[597360.268192] <0> ffff8800c232ff28 ffffffff8141143e ffff880000000001 ffff880000000002
[597360.268192] Call Trace:
[597360.268192] [<ffffffff811a93fb>] ? new_inode+0x3b/0xb0
[597360.268192] [<ffffffff81410fd9>] ? sock_alloc+0x19/0x60
[597360.268192] [<ffffffff8141143e>] ? __sock_create+0x9e/0x210
[597360.268192] [<ffffffff81411610>] ? sock_create+0x30/0x40
[597360.268192] [<ffffffff8141182c>] ? sys_socket+0x3c/0x80
[597360.268192] [<ffffffff8100b182>] ? system_call_fastpath+0x16/0x1b
[597360.268192] Code: 01 74 05 e8 12 d1 d8 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f
[597360.268192] Call Trace:
[597360.268192] [<ffffffff811a93fb>] ? new_inode+0x3b/0xb0
[597360.268192] [<ffffffff81410fd9>] ? sock_alloc+0x19/0x60
[597360.268192] [<ffffffff8141143e>] ? __sock_create+0x9e/0x210
[597360.268192] [<ffffffff81411610>] ? sock_create+0x30/0x40
[597360.268192] [<ffffffff8141182c>] ? sys_socket+0x3c/0x80
[597360.268192] [<ffffffff8100b182>] ? system_call_fastpath+0x16/0x1b
[597360.079003] xt_multiport xt_MARK xt_mark xt_mac xt_limit xt_length xt_helper xt_esp xt_CONNSECMARK xt_comment nfnetlink ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables arc4 ecb ppp_mppe ppp_deflate zlib_deflate ppp_async ppp_generic slhc crc_ccitt serio_raw k8temp amd64_edac_mod edac_core edac_mce_amd sg i2c_nforce2 i2c_core ext4 mbcache jbd2 raid1 sr_mod cdrom sd_mod crc_t10dif forcedeth pata_amd ata_generic pata_acpi sata_nv dm_mirror dm_region_hash dm_log dm_mod be2iscsi bnx2i cnic uio ipv6 cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio libiscsi_tcp qla4xxx iscsi_boot_sysfs libiscsi scsi_transport_iscsi [last unloaded: scsi_wait_scan]
[597360.079003] CPU 0
[597360.079003] Modules linked in: vzethdev simfs vzrst vzcpt nfs lockd fscache nfs_acl auth_rpcgss sunrpc vzdquota ip6table_mangle nf_nat_ftp vzevent vznetdev vzmon vzdev xt_sctp xt_realm xt_NOTRACK xt_dccp xt_CONNMARK xt_connmark xt_connbytes xt_CLASSIFY nfnetlink_queue nfnetlink_log xt_recent xt_DSCP nf_nat_irc xt_conntrack ipt_LOG nf_conntrack_irc nf_conntrack_ftp xt_hl xt_tcpmss xt_TCPMSS xt_dscp ipt_REDIRECT ipt_REJECT tun nf_conntrack_netbios_ns nf_conntrack_broadcast iptable_nat nf_nat iptable_mangle nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables xt_string xt_statistic xt_SECMARK xt_quota xt_policy xt_pkttype xt_physdev xt_NFQUEUE xt_multiport xt_MARK xt_mark xt_mac xt_limit xt_length xt_helper xt_esp xt_CONNSECMARK xt_comment nfnetlink ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables arc4 ecb ppp_mppe ppp_deflate zlib_deflate ppp_async ppp_generic slhc crc_ccitt serio_raw k8temp amd64_edac_mod edac_core edac_mce_amd sg i2c_nforce2 i2c_core ext4 mbcache jbd2 raid1 sr_mod cdrom sd_mod crc_t10dif forcedeth pata_amd ata_generic pata_acpi sata_nv dm_mirror dm_region_hash dm_log dm_mod be2iscsi bnx2i cnic uio ipv6 cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio libiscsi_tcp qla4xxx iscsi_boot_sysfs libiscsi scsi_transport_iscsi [last unloaded: scsi_wait_scan]
[597360.079003]
[597360.079003] Pid: 390166, comm: apache2 veid: 1025 Not tainted 2.6.32-042stab049.6 #1 042stab049_6 empty empty/S2927
[597360.079003] RIP: 0010:[<ffffffff814e191e>] [<ffffffff814e191e>] _spin_lock+0x1e/0x30
[597360.079003] RSP: 0018:ffff8803cc201df8 EFLAGS: 00000297
[597360.079003] RAX: 0000000000003c91 RBX: ffff8803cc201df8 RCX: 0000000000000002
[597360.079003] RDX: 0000000000003c90 RSI: ffffffff823cca00 RDI: ffffffff823cca00
[597360.079003] RBP: ffffffff8100bcce R08: 0000000000000000 R09: 0000000000000000
[597360.079003] R10: 0000000000000000 R11: 0000000000000206 R12: ffff8802243416b0
[597360.079003] R13: ffff8802243416b0 R14: ffff8802243416b0 R15: ffff8802243416b0
[597360.079003] FS: 000000004d377960(0063) GS:ffff880028200000(0000) knlGS:00000000b774f6c0
[597360.079003] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[597360.079003] CR2: 000000004d36cf90 CR3: 0000000111323000 CR4: 00000000000006f0
[597360.079003] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[597360.079003] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[597360.079003] Process apache2 (pid: 390166, veid: 1025, threadinfo ffff8803cc200000, task ffff880329357410)
[597360.079003] Stack:
[597360.079003] ffff8803cc201e28 ffffffff81261685 ffff8803cc201e08 ffffffff823cca00
[597360.079003] <0> ffff880012918150 ffff8801862a8790 ffff8803cc201e48 ffffffff811a77b0
[597360.079003] <0> ffff8803cc201e48 ffff880012918150 ffff8803cc201e68 ffffffff811a4028
[597360.079003] Call Trace:
[597360.079003] [<ffffffff81261685>] ? _atomic_dec_and_lock+0x55/0x80
[597360.079003] [<ffffffff811a77b0>] ? iput+0x30/0x70
[597360.079003] [<ffffffff811a4028>] ? dentry_iput+0x98/0x110
[597360.079003] [<ffffffff811a419c>] ? d_kill+0x3c/0x70
[597360.079003] [<ffffffff811a5933>] ? dput+0xa3/0x1d0
[597360.079003] [<ffffffff8118e9e1>] ? __fput+0x1d1/0x280
[597360.079003] [<ffffffff8118eab5>] ? fput+0x25/0x30
[597360.079003] [<ffffffff8118a01d>] ? filp_close+0x5d/0x90
[597360.079003] [<ffffffff8118a0f5>] ? sys_close+0xa5/0x100
[597360.079003] [<ffffffff8100b182>] ? system_call_fastpath+0x16/0x1b
[597360.079003] Code: 00 00 00 01 74 05 e8 12 d1 d8 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89
[597360.079003] Call Trace:
[597360.079003] [<ffffffff81261685>] ? _atomic_dec_and_lock+0x55/0x80
[597360.079003] [<ffffffff811a77b0>] ? iput+0x30/0x70
[597360.079003] [<ffffffff811a4028>] ? dentry_iput+0x98/0x110
[597360.079003] [<ffffffff811a419c>] ? d_kill+0x3c/0x70
[597360.079003] [<ffffffff811a5933>] ? dput+0xa3/0x1d0
[597360.079003] [<ffffffff8118e9e1>] ? __fput+0x1d1/0x280
[597360.079003] [<ffffffff8118eab5>] ? fput+0x25/0x30
[597360.079003] [<ffffffff8118a01d>] ? filp_close+0x5d/0x90
[597360.079003] [<ffffffff8118a0f5>] ? sys_close+0xa5/0x100
[597360.079003] [<ffffffff8100b182>] ? system_call_fastpath+0x16/0x1b
[root@vz ~]# uname -a
Linux vz.xyz.com 2.6.32-042stab049.6 #1 SMP Mon Feb 6 19:17:43 MSK 2012 x86_64 x86_64 x86_64 GNU/Linux
[root@vz ~]# cat /etc/redhat-release
CentOS release 6.2 (Final)
This is as much as my terminal server logs captured:
th xt_helper xt_esp xt_CONNSECMARK xt_comment nfnetlink ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables arc4 ecb ppp_mppe ppp_deflate zlib_deflate ppp_async ppp_generic slhc crc_ccitt serio_raw k8temp amd64_edac_mod edac_core edac_mce_amd sg i2c_nforce2 i2c_core ext4 mbcache jbd2 raid1 sr_mod cdrom sd_mod crc_t10dif forcedeth pata_amd ata_generic pata_acpi sata_nv dm_mirror dm_region_hash dm_log dm_mod be2iscsi bnx2i cnic uio ipv6 cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio libiscsi_tcp qla4xxx iscsi_boot_sysfs libiscsi scsi_transport_iscsi [last unloaded: scsi_wait_scan]
[597276.142067] CPU 1
[597276.142067] Modules linked in: vzethdev simfs vzrst vzcpt nfs lockd fscache nfs_acl auth_rpcgss sunrpc vzdquota ip6table_mangle nf_nat_ftp vzevent vznetdev vzmon vzdev xt_sctp xt_realm xt_NOTRACK xt_dccp xt_CONNMARK xt_connmark xt_connbytes xt_CLASSIFY nfnetlink_queue nfnetlink_log xt_recent xt_DSCP nf_nat_irc xt_conntrack ipt_LOG nf_conntrack_irc nf_conntrack_ftp xt_hl xt_tcpmss xt_TCPMSS xt_dscp ipt_REDIRECT ipt_REJECT tun nf_conntrack_netbios_ns nf_conntrack_broadcast iptable_nat nf_nat iptable_mangle nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables xt_string xt_statistic xt_SECMARK xt_quota xt_policy xt_pkttype xt_physdev xt_NFQUEUE xt_multiport xt_MARK xt_mark xt_mac xt_limit xt_length xt_helper xt_esp xt_CONNSECMARK xt_comment nfnetlink ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables arc4 ecb ppp_mppe ppp_deflate zlib_deflate ppp_async ppp_generic slhc crc_ccitt serio_raw k8temp amd64_edac_mod edac_core edac_mce_amd sg i2c_nforce2 i2c_core ext4 mbcache jbd2 raid1 sr_mod cdrom sd_mod crc_t10dif forcedeth pata_amd ata_generic pata_acpi sata_nv dm_mirror dm_region_hash dm_log dm_mod be2iscsi bnx2i cnic uio ipv6 cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio libiscsi_tcp qla4xxx iscsi_boot_sysfs libiscsi scsi_transport_iscsi [last unloaded: scsi_wait_scan]
[597276.142067]
[597276.142067] Pid: 946200, comm: set_keyserver_u veid: 1035 Not tainted 2.6.32-042stab049.6 #1 042stab049_6 empty empty/S2927
[597276.142067] RIP: 0010:[<ffffffff814e191e>] [<ffffffff814e191e>] _spin_lock+0x1e/0x30
[597276.142067] RSP: 0018:ffff88000babfe98 EFLAGS: 00000293
[597276.142067] RAX: 0000000000003c92 RBX: ffff88000babfe98 RCX: 0000000000000000
[597276.142067] RDX: 0000000000003c90 RSI: ffffffff817a1438 RDI: ffffffff823cca00
[597276.142067] RBP: ffffffff8100bcce R08: 0000000000000000 R09: 000000000000000e
[597276.142067] R10: 0000000000000000 R11: 0000000000000246 R12: ffffffff81042b54
[597276.142067] R13: ffff88000babff18 R14: 0000000000000014 R15: ffff8803c5c004a8
[597276.142067] FS: 00007f54f714b6e0(0000) GS:ffff880028280000(0000) knlGS:00000000b774f6c0
[597276.142067] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[597276.142067] CR2: 00007f54f66747e0 CR3: 00000002a8a5b000 CR4: 00000000000006e0
[597276.142067] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[597276.142067] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[597276.142067] Process set_keyserver_u (pid: 946200, veid: 1035, threadinfo ffff88000babe000, task ffff8800ad7b3290)
[597276.142067] Stack:
[597276.142067] ffff88000babfec8 ffffffff811a93fb 000000001d12a781 ffff880336840980
[597276.142067] <0> 0000000000000000 0000000000000000 ffff88000babfed8 ffffffff81410fd9
[597276.142067] <0> ffff88000babff28 ffffffff8141143e ffff880000000001 ffff880000000002
[597276.142067] Call Trace:
[597276.142067] [<ffffffff811a93fb>] ? new_inode+0x3b/0xb0
[597276.142067] [<ffffffff81410fd9>] ? sock_alloc+0x19/0x60
[597276.142067] [<ffffffff8141143e>] ? __sock_create+0x9e/0x210
[597276.142067] [<ffffffff81411610>] ? sock_create+0x30/0x40
[597276.142067] [<ffffffff8141182c>] ? sys_socket+0x3c/0x80
[597276.142067] [<ffffffff8100b182>] ? system_call_fastpath+0x16/0x1b
[597276.142067] Code: 00 00 00 01 74 05 e8 12 d1 d8 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89
[597276.142067] Call Trace:
[597276.142067] [<ffffffff811a93fb>] ? new_inode+0x3b/0xb0
[597276.142067] [<ffffffff81410fd9>] ? sock_alloc+0x19/0x60
[597276.142067] [<ffffffff8141143e>] ? __sock_create+0x9e/0x210
[597276.142067] [<ffffffff81411610>] ? sock_create+0x30/0x40
[597276.142067] [<ffffffff8141182c>] ? sys_socket+0x3c/0x80
[597276.142067] [<ffffffff8100b182>] ? system_call_fastpath+0x16/0x1b
[597276.206006] BUG: soft lockup - CPU#2 stuck for 67s! [update_deploy_p:946201]
[597276.206009] Modules linked in: vzethdev simfs vzrst vzcpt nfs lockd fscache nfs_acl auth_rpcgss sunrpc vzdquota ip6table_mangle nf_nat_ftp vzevent vznetdev vzmon vzdev xt_sctp xt_realm xt_NOTRACK xt_dccp xt_CONNMARK xt_connmark xt_connbytes xt_CLASSIFY nfnetlink_queue nfnetlink_log xt_recent xt_DSCP nf_nat_irc xt_conntrack ipt_LOG nf_conntrack_irc nf_conntrack_ftp xt_hl xt_tcpmss xt_TCPMSS xt_dscp ipt_REDIRECT ipt_REJECT tun nf_conntrack_netbios_ns nf_conntrack_broadcast iptable_nat nf_nat iptable_mangle nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables xt_string xt_statistic xt_SECMARK xt_quota xt_policy xt_pkttype xt_physdev xt_NFQUEUE xt_multiport xt_MARK xt_mark xt_mac xt_limit xt_length xt_helper xt_esp xt_CONNSECMARK xt_comment nfnetlink ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables arc4 ecb ppp_mppe ppp_deflate zlib_deflate ppp_async ppp_generic slhc crc_ccitt serio_raw k8temp amd64_edac_mod edac_core edac_mce_amd sg i2c_nforce2 i2c_core ext4
mbcache jbd2 raid1 sr_mod cdrom sd_mod crc_t10dif forcedeth pata_amd ata_generic pata_acpi sata_nv dm_mirror dm_region_hash dm_log dm_mod be2iscsi bnx2i cnic uio ipv6 cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio libiscsi_tcp qla4xxx iscsi_boot_sysfs libiscsi scsi_transport_iscsi [last unloaded: scsi_wait_scan]
[597276.206085] CPU 2
[597276.206087] Modules linked in: vzethdev simfs vzrst vzcpt nfs lockd fscache nfs_acl auth_rpcgss sunrpc vzdquota ip6table_mangle nf_nat_ftp vzevent vznetdev vzmon vzdev xt_sctp xt_realm xt_NOTRACK xt_dccp xt_CONNMARK xt_connmark xt_connbytes xt_CLASSIFY nfnetlink_queue nfnetlink_log xt_recent xt_DSCP nf_nat_irc xt_conntrack ipt_LOG nf_conntrack_irc nf_conntrack_ftp xt_hl xt_tcpmss xt_TCPMSS xt_dscp ipt_REDIRECT ipt_REJECT tun nf_conntrack_netbios_ns nf_conntrack_broadcast iptable_nat nf_nat iptable_mangle nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables xt_string xt_statistic xt_SECMARK xt_quota xt_policy xt_pkttype xt_physdev xt_NFQUEUE xt_multiport xt_MARK xt_mark xt_mac xt_limit xt_length xt_helper xt_esp xt_CONNSECMARK xt_comment nfnetlink ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables arc4 ecb ppp_mppe ppp_deflate zlib_deflate ppp_async ppp_generic slhc crc_ccitt serio_raw k8temp amd64_edac_mod edac_core edac_mce_amd sg i2c_nforce2 i2c_core ext4 mbcache jbd2 raid1 sr_mod cdrom sd_mod crc_t10dif forcedeth pata_amd ata_generic pata_acpi sata_nv dm_mirror dm_region_hash dm_log dm_mod be2iscsi bnx2i cnic uio ipv6 cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio libiscsi_tcp qla4xxx iscsi_boot_sysfs libiscsi scsi_transport_iscsi [last unloaded: scsi_wait_scan]
[597276.206133]
[597276.206133] Pid: 946201, comm: update_deploy_p veid: 1035 Not tainted 2.6.32-042stab049.6 #1 042stab049_6 empty empty/S2927
[597276.206133] RIP: 0010:[<ffffffff814e191e>] [<ffffffff814e191e>] _spin_lock+0x1e/0x30
[597276.206133] RSP: 0018:ffff8800ad635e98 EFLAGS: 00000297
[597276.206133] RAX: 0000000000003c94 RBX: ffff8800ad635e98 RCX: 0000000000000000
[597276.206133] RDX: 0000000000003c90 RSI: ffffffff817a1438 RDI: ffffffff823cca00
[597276.206133] RBP: ffffffff8100bcce R08: 0000000000000000 R09: 0000000000000003
[597276.206133] R10: 0000000000000000 R11: 0000000000000246 R12: ffffffff81042b54
[597276.206133] R13: ffff8800ad635f18 R14: 0000000000000014 R15: ffff8803151d8c78
[597276.206133] FS: 00007f4f0dc176e0(0000) GS:ffff880230400000(0000) knlGS:00000000b774f6c0
[597276.206133] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[597276.206133] CR2: 00007f4f0d1407e0 CR3: 00000002310a1000 CR4: 00000000000006e0
[597276.206133] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[597276.206133] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[597276.206133] Process update_deploy_p (pid: 946201, veid: 1035, threadinfo ffff8800ad634000, task ffff8803d03a4800)
[597276.206133] Stack:
[597276.206133] ffff8800ad635ec8 ffffffff811a93fb 000000001d12a781 ffff880336840980
[597276.206133] <0> 00 ppp_mppe ppp_deflate zlib_deflate ppp_async ppp_generic slhc crc_ccitt serio_raw k8temp amd64_edac_mod edac_core edac_mce_amd sg i2c_nforce2 i2c_core ext4 mbcache jbd2 raid1 sr_mod cdrom sd_mod crc_t10dif forcedeth pata_amd ata_generic pata_acpi sata_nv dm_mirror dm_region_hash dm_log dm_mod be2iscsi bnx2i cnic uio ipv6 cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio libiscsi_tcp qla4xxx iscsi_boot_sysfs libiscsi scsi_transport_iscsi [last unloaded: scsi_wait_scan]
[597276.268102] CPU 3
[597276.268103] Modules 0000(0000) knlGS:00000000b774f6c0
[597276.268192] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[597276.268192] CR2: 00007f246f7067e0 CR3: 0000000407807000 CR4: 00000000000006e0
[597276.268192] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[597276.268192] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[597276.268192] Process update_deploy_p (pid: 946188, veid: 1063, threadinfo ffff8800c232e000, task ffff8804239049c0)
[597276.268192] Stack:
[597276.268192] ffff8800c232fec8 ffffffff811a93fb 0000000009ffd874 ffff880225129200
[597276.268192] <0> 0000000000000000 0000000000000000 ffff8800c232fed8 ffffffff81410fd9
[597276.268192] <0> ffff8800c232ff28 ffffffff8141143e ffff880000000001 ffff880000000002
[597276.268192] Call Trace:
[597276.268192] [<ffffffff811a93fb>] ? new_inode+0x3b/0xb0
[597276.268192] [<ffffffff81410fd9>] ? sock_alloc+0x19/0x60
[597276.268192] [<ffffffff8141143e>] ? __sock_create+0x9e/0x210
[597276.268192] [<ffffffff81411610>] ? sock_crestring xt_statistic xt_SECMARK xt_quota xt_policy xt_pkttype xt_physdev xt_NFQUEUE xt_multiport xt_MARK xt_mark xt_mac xt_limit xt_length xt_helper xt_esp xt_CONNSECMARK xt_comment nfnetlink ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables arc4 ecb ppp_mppe ppp_deflate zlib_deflate ppp_async ppp_generic slhc crc_ccitt serio_raw k8temp amd64_edac_mod edac_core edac_mce_amd sg i2c_nforce2 i2c_core ext4 mbcache jbd2 raid1 sr_mod cdrom sd_mod crc_t10dif forcedeth pata_amd ata_generic pata_acpi sata_nv dm_mirror dm_region_hash dm_log dm_mod be2iscsi bnx2i cnic uio ipv6 cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio libiscsi_tcp qla4xxx iscsi_boot_sysfs libiscsi scsi_transport_iscsi [last unloaded: scsi_wait_scan]
[597276.079003]
[597276.079003] Pid: 390166, comm: apache2 veid: 1025 Not tainted 2.6.32-042stab049.6 #1 042stab049_6 empty empty/S2927
[597276.079003] RIP: 0010:[<ffffffff814e191e>] [<ffffffff814e191e>] _spin_lock+0x1e/0x30
[597276.079003] RSP: 0018:ffff8803cc2b df c9 c3 0f 1f 40 00 55 48 89
[597276.079003] Call Trace:
[597276.079003] [<ffffffff81261685>] ? _atomic_dec_and_lock+0x55/0x80
[597276.079003] [<ffffffff811a77b0>] ? iput+0x30/0x70
[597276.079003] [<ffffffff811a4028>] ? dentry_iput+0x98/0x110
[597276.079003] [<ffffffff811a419c>] ? d_kill+0x3c/0x70
[597276.079003] [<ffffffff811a5933>] ? dput+0xa3/0x1d0
[597276.079003] [<ffffffff8118e9e1>] ? __fput+0x1d1/0x280
[597276.079003] [<ffffffff8118eab5>] ? fput+0x25/0x30
[597276.079003] [<fffffff
[597360.268180] RIP: 0010:[<ffffffff814e1921>] [<ffffffff814e1921>] _spin_lock+0x21/0x30
[597360.268191] RSP: 0018:ffff8800c232fe98 EFLAGS: 00000293
[597360.268192] RAX: 0000000000003c93 RBX: ffff8800c232fe98 RCX: 0000000000000000
[597360.268192] RDX: 0000000000003c90 RSI: ffffffff817a1438 RDI: ffffffff823cca00
[597360.268192] RBP: ffffffff8100bcce R08: 0000000000000000 R09: 000000000000000f
[597360.268192] R10: 0000000000000000 R11: 0000000000000246 R12: ffffffff81042b54
[597360.268192] R13: ffff8800c232ff18 R14: 0000000000000014 R15: ffff880336b5ce90
[597360.268192] FS: 00007f24701dd6e0(0000) GS:ffff880230480000(0000) knlGS:00000000b774f6c0
[597360.268192] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[597360.268192] CR2: 00007f246f7067e0 CR3: 0000000407807000 CR4: 00000000000006e0
[597360.268192] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[597360.268192] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[597360.268192] Process update_deploy_p (pid: 946188, veid: 1063, threadinfo ffff8800c232e000, task ffff8804239049c0)
[597360.268192] Stack:
[597360.268192] ffff8800c232fec8 ffffffff811a93fb 0000000009ffd874 ffff880225129200
[597360.268192] <0> 0000000000000000 0000000000000000 ffff8800c232fed8 ffffffff81410fd9
[597360.268192] <0> ffff8800c232ff28 ffffffff8141143e ffff880000000001 ffff880000000002
[597360.268192] Call Trace:
[597360.268192] [<ffffffff811a93fb>] ? new_inode+0x3b/0xb0
[597360.268192] [<ffffffff81410fd9>] ? sock_alloc+0x19/0x60
[597360.268192] [<ffffffff8141143e>] ? __sock_create+0x9e/0x210
[597360.268192] [<ffffffff81411610>] ? sock_create+0x30/0x40
[597360.268192] [<ffffffff8141182c>] ? sys_socket+0x3c/0x80
[597360.268192] [<ffffffff8100b182>] ? system_call_fastpath+0x16/0x1b
[597360.268192] Code: 01 74 05 e8 12 d1 d8 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f
[597360.268192] Call Trace:
[597360.268192] [<ffffffff811a93fb>] ? new_inode+0x3b/0xb0
[597360.268192] [<ffffffff81410fd9>] ? sock_alloc+0x19/0x60
[597360.268192] [<ffffffff8141143e>] ? __sock_create+0x9e/0x210
[597360.268192] [<ffffffff81411610>] ? sock_create+0x30/0x40
[597360.268192] [<ffffffff8141182c>] ? sys_socket+0x3c/0x80
[597360.268192] [<ffffffff8100b182>] ? system_call_fastpath+0x16/0x1b
[597360.079003] xt_multiport xt_MARK xt_mark xt_mac xt_limit xt_length xt_helper xt_esp xt_CONNSECMARK xt_comment nfnetlink ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables arc4 ecb ppp_mppe ppp_deflate zlib_deflate ppp_async ppp_generic slhc crc_ccitt serio_raw k8temp amd64_edac_mod edac_core edac_mce_amd sg i2c_nforce2 i2c_core ext4 mbcache jbd2 raid1 sr_mod cdrom sd_mod crc_t10dif forcedeth pata_amd ata_generic pata_acpi sata_nv dm_mirror dm_region_hash dm_log dm_mod be2iscsi bnx2i cnic uio ipv6 cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio libiscsi_tcp qla4xxx iscsi_boot_sysfs libiscsi scsi_transport_iscsi [last unloaded: scsi_wait_scan]
[597360.079003] CPU 0
[597360.079003] Modules linked in: vzethdev simfs vzrst vzcpt nfs lockd fscache nfs_acl auth_rpcgss sunrpc vzdquota ip6table_mangle nf_nat_ftp vzevent vznetdev vzmon vzdev xt_sctp xt_realm xt_NOTRACK xt_dccp xt_CONNMARK xt_connmark xt_connbytes xt_CLASSIFY nfnetlink_queue nfnetlink_log xt_recent xt_DSCP nf_nat_irc xt_conntrack ipt_LOG nf_conntrack_irc nf_conntrack_ftp xt_hl xt_tcpmss xt_TCPMSS xt_dscp ipt_REDIRECT ipt_REJECT tun nf_conntrack_netbios_ns nf_conntrack_broadcast iptable_nat nf_nat iptable_mangle nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables xt_string xt_statistic xt_SECMARK xt_quota xt_policy xt_pkttype xt_physdev xt_NFQUEUE xt_multiport xt_MARK xt_mark xt_mac xt_limit xt_length xt_helper xt_esp xt_CONNSECMARK xt_comment nfnetlink ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables arc4 ecb ppp_mppe ppp_deflate zlib_deflate ppp_async ppp_generic slhc crc_ccitt serio_raw k8temp amd64_edac_mod edac_core edac_mce_amd sg i2c_nforce2 i2c_core ext4 mbcache jbd2 raid1 sr_mod cdrom sd_mod crc_t10dif forcedeth pata_amd ata_generic pata_acpi sata_nv dm_mirror dm_region_hash dm_log dm_mod be2iscsi bnx2i cnic uio ipv6 cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio libiscsi_tcp qla4xxx iscsi_boot_sysfs libiscsi scsi_transport_iscsi [last unloaded: scsi_wait_scan]
[597360.079003]
[597360.079003] Pid: 390166, comm: apache2 veid: 1025 Not tainted 2.6.32-042stab049.6 #1 042stab049_6 empty empty/S2927
[597360.079003] RIP: 0010:[<ffffffff814e191e>] [<ffffffff814e191e>] _spin_lock+0x1e/0x30
[597360.079003] RSP: 0018:ffff8803cc201df8 EFLAGS: 00000297
[597360.079003] RAX: 0000000000003c91 RBX: ffff8803cc201df8 RCX: 0000000000000002
[597360.079003] RDX: 0000000000003c90 RSI: ffffffff823cca00 RDI: ffffffff823cca00
[597360.079003] RBP: ffffffff8100bcce R08: 0000000000000000 R09: 0000000000000000
[597360.079003] R10: 0000000000000000 R11: 0000000000000206 R12: ffff8802243416b0
[597360.079003] R13: ffff8802243416b0 R14: ffff8802243416b0 R15: ffff8802243416b0
[597360.079003] FS: 000000004d377960(0063) GS:ffff880028200000(0000) knlGS:00000000b774f6c0
[597360.079003] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[597360.079003] CR2: 000000004d36cf90 CR3: 0000000111323000 CR4: 00000000000006f0
[597360.079003] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[597360.079003] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[597360.079003] Process apache2 (pid: 390166, veid: 1025, threadinfo ffff8803cc200000, task ffff880329357410)
[597360.079003] Stack:
[597360.079003] ffff8803cc201e28 ffffffff81261685 ffff8803cc201e08 ffffffff823cca00
[597360.079003] <0> ffff880012918150 ffff8801862a8790 ffff8803cc201e48 ffffffff811a77b0
[597360.079003] <0> ffff8803cc201e48 ffff880012918150 ffff8803cc201e68 ffffffff811a4028
[597360.079003] Call Trace:
[597360.079003] [<ffffffff81261685>] ? _atomic_dec_and_lock+0x55/0x80
[597360.079003] [<ffffffff811a77b0>] ? iput+0x30/0x70
[597360.079003] [<ffffffff811a4028>] ? dentry_iput+0x98/0x110
[597360.079003] [<ffffffff811a419c>] ? d_kill+0x3c/0x70
[597360.079003] [<ffffffff811a5933>] ? dput+0xa3/0x1d0
[597360.079003] [<ffffffff8118e9e1>] ? __fput+0x1d1/0x280
[597360.079003] [<ffffffff8118eab5>] ? fput+0x25/0x30
[597360.079003] [<ffffffff8118a01d>] ? filp_close+0x5d/0x90
[597360.079003] [<ffffffff8118a0f5>] ? sys_close+0xa5/0x100
[597360.079003] [<ffffffff8100b182>] ? system_call_fastpath+0x16/0x1b
[597360.079003] Code: 00 00 00 01 74 05 e8 12 d1 d8 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89
[597360.079003] Call Trace:
[597360.079003] [<ffffffff81261685>] ? _atomic_dec_and_lock+0x55/0x80
[597360.079003] [<ffffffff811a77b0>] ? iput+0x30/0x70
[597360.079003] [<ffffffff811a4028>] ? dentry_iput+0x98/0x110
[597360.079003] [<ffffffff811a419c>] ? d_kill+0x3c/0x70
[597360.079003] [<ffffffff811a5933>] ? dput+0xa3/0x1d0
[597360.079003] [<ffffffff8118e9e1>] ? __fput+0x1d1/0x280
[597360.079003] [<ffffffff8118eab5>] ? fput+0x25/0x30
[597360.079003] [<ffffffff8118a01d>] ? filp_close+0x5d/0x90
[597360.079003] [<ffffffff8118a0f5>] ? sys_close+0xa5/0x100
[597360.079003] [<ffffffff8100b182>] ? system_call_fastpath+0x16/0x1b
Attachments
Issue Links
- is duplicated by
-
OVZ-5396 softlocks in 55.7
- Resolved