Details
-
Type: Bug
-
Status: Resolved
-
Priority: Major
-
Resolution: Fixed
-
Fix Version/s: OpenVZ-legacy
-
Component/s: Containers::Kernel
-
Security Level: Public
-
Environment:Centos 6, kernel 2.6.32-042stab133.1
vzctl version 4.10-4.g6f81921
ploop-1.15-1.x86_64
-
Fixed in Build:vzkernel-2.6.32-042stab133.2
Description
Hello!
We have node crash on 2.6.32-042stab133.1 with
<1>[78689.758852] BUG: unable to handle kernel NULL pointer dereference at 0000000000000248
<1>[78689.758892] IP: [<ffffffff810d1d11>] ub_sock_tcp_chargerecv+0x21/0x220
<4>[78689.758923] Kernel PGD 0
<4>[78689.758940] User PGD 0
<4>[78689.758957] Oops: 0000 [#1] SMP
<4>[78689.758981] last sysfs file: /sys/devices/system/node/node1/numastat
<4>[78689.759001] CPU 17
<4>[78689.759011] Modules linked in: sch_sfq cls_u32 sch_htb coretemp xt_limit ts_kmp xt_string ipt_addrtype drbg ansi_cprng ctr twofish_x86_64 twofish_common camellia serpent blowfish cast5 des_generic cbc crypto_null aesni_intel ablk_helper cryptd lrw gf128mul glue_helper aes_x86_64 aes_generic xcbc rmd160 sha512_generic xt_TCPMSS ipt_REJECT xt_conntrack xt_multiport vzethdev pio_kaio pio_nfs pio_direct pfmt_raw pfmt_ploop1 ploop simfs vziolimit vzdquota ip6t_REJECT ip6table_mangle vzevent 8021q garp stp llc vznetdev bonding xt_comment iptable_filter ip6table_filter ip6_tables ipip tunnel4 nf_conntrack_ftp xt_recent vzrst vzcpt vzmon vzdev nfs lockd fscache auth_rpcgss nfs_acl sunrpc ipt_MASQUERADE iptable_raw iptable_mangle xt_connlimit ipt_REDIRECT xt_owner nf_conntrack_ipv6 nf_defrag_ipv6 xt_state ipt_LOG xfrm_ipcomp xfrm4_mode_transport pppol2tp pppox xfrm6_mode_tunnel xfrm4_mode_tunnel esp6 ipv6 esp4 af_key arc4 ecb ppp_mppe ppp_deflate zlib_deflate ppp_async ppp_generic slhc crc_ccitt fuse iptable_nat nf_nat ip_tables nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 tun freq_table mperf power_meter acpi_ipmi ipmi_si ipmi_msghandler iTCO_wdt iTCO_vendor_support joydev dcdbas sb_edac edac_core lpc_ich mfd_core shpchp sg tg3 ptp pps_core ext4 jbd2 mbcache sd_mod crc_t10dif megaraid_sas wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]
<4>[78689.761497]
<4>[78689.761756] Pid: 0, comm: swapper veid: 0 Not tainted 2.6.32-042stab133.1 #1 042stab133 Dell Inc. PowerEdge R720/0C4Y3R
<4>[78689.762297] RIP: 0010:[<ffffffff810d1d11>] [<ffffffff810d1d11>] ub_sock_tcp_chargerecv+0x21/0x220
<4>[78689.762831] RSP: 0018:ffff8830f1203920 EFLAGS: 00010282
<4>[78689.763097] RAX: ffff885b1f040a80 RBX: 0000000000000000 RCX: 0000000000000000
<4>[78689.763367] RDX: 0000000000000002 RSI: ffff885b1f040a80 RDI: 0000000000000000
<4>[78689.763662] RBP: ffff8830f1203960 R08: ffffffff81ade580 R09: ffff885ffd9c809c
<4>[78689.763937] R10: 0000000000000002 R11: 0000000000000002 R12: ffff8859560e8340
<4>[78689.764208] R13: 00000000e7ae6aaa R14: ffff885b1f040a80 R15: 0000000000000e80
<4>[78689.764487] FS: 0000000000000000(0000) GS:ffff8830f1200000(0000) knlGS:0000000000000000
<4>[78689.765013] CS: 0010 DS: 0018 ES: 0018 CR0: 0000000080050033
<4>[78689.765279] CR2: 0000000000000248 CR3: 0000000001a8e000 CR4: 00000000000607e0
<4>[78689.765565] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
<4>[78689.765847] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
<4>[78689.766125] Process swapper (pid: 0, veid: 0, threadinfo ffff883017264000, task ffff88301725c180)
<4>[78689.766677] Stack:
<4>[78689.766941] ffff8830ffffffff ffff886017077340 0000000000000000 0000000000000000
<4>[78689.767225] <d> ffff8859560e8340 00000000e7ae6aaa ffff885b1f040a80 0000000000000e80
<4>[78689.767769] <d> ffff8830f1203a10 ffffffff814edc73 ffff8830f1203990 ffff8830f12039c0
<4>[78689.768643] Call Trace:
<4>[78689.768898] <IRQ>
<4>[78689.769163] [<ffffffff814edc73>] tcp_collapse+0x213/0x400
<4>[78689.769429] [<ffffffff814ee10f>] tcp_try_rmem_schedule+0x2af/0x3d0
<4>[78689.769722] [<ffffffff814ee600>] tcp_data_queue+0x3d0/0xcd0
<4>[78689.769991] [<ffffffff81568325>] ? _read_unlock_bh+0x15/0x20
<4>[78689.770260] [<ffffffff814ef0b0>] ? tcp_validate_incoming+0x1b0/0x320
<4>[78689.770538] [<ffffffff814f1d19>] tcp_rcv_established+0x3d9/0x860
<4>[78689.770860] [<ffffffffa02e6d15>] tcp_v6_do_rcv+0x3d5/0x720 [ipv6]
<4>[78689.771147] [<ffffffffa035e436>] ? ipv6_confirm+0x136/0x1e0 [nf_conntrack_ipv6]
<4>[78689.771699] [<ffffffffa02e95e6>] tcp_v6_rcv+0x626/0x9c0 [ipv6]
<4>[78689.771974] [<ffffffffa02c33a7>] ip6_input_finish+0x177/0x430 [ipv6]
<4>[78689.772248] [<ffffffffa02c36b8>] ip6_input+0x58/0x60 [ipv6]
<4>[78689.772521] [<ffffffffa02c2da5>] ip6_rcv_finish+0x45/0x50 [ipv6]
<4>[78689.772793] [<ffffffffa02c3150>] ipv6_rcv+0x3a0/0x480 [ipv6]
<4>[78689.773061] [<ffffffff814d71f7>] ? ip_rcv+0x287/0x370
<4>[78689.773330] [<ffffffff8149989a>] __netif_receive_skb+0x24a/0x600
<4>[78689.773598] [<ffffffff81499cea>] process_backlog+0x9a/0x100
<4>[78689.779774] [<ffffffff8149f3e6>] net_rx_action+0x116/0x330
<4>[78689.780046] [<ffffffff81090bea>] __do_softirq+0x12a/0x2b0
<4>[78689.780320] [<ffffffff815719cc>] call_softirq+0x1c/0x30
<4>[78689.780592] [<ffffffff8100eb85>] do_softirq+0x65/0xa0
<4>[78689.780861] [<ffffffff810907f5>] irq_exit+0xd5/0xe0
<4>[78689.781130] [<ffffffff81572825>] do_IRQ+0x75/0xf0
<4>[78689.781397] [<ffffffff81570f13>] ret_from_intr+0x0/0x11
<4>[78689.781663] <EOI>
<4>[78689.781932] [<ffffffff813192ae>] ? intel_idle+0x12e/0x250
<4>[78689.782198] [<ffffffff81319291>] ? intel_idle+0x111/0x250
<4>[78689.782563] [<ffffffff81013a59>] ? sched_clock+0x9/0x10
<4>[78689.782828] [<ffffffff810bcaed>] ? sched_clock_cpu+0xcd/0x110
<4>[78689.783096] [<ffffffff81460cde>] cpuidle_idle_call+0x8e/0xf0
<4>[78689.783361] [<ffffffff8100a179>] cpu_idle+0xb9/0x110
<4>[78689.783627] [<ffffffff8155d6b0>] start_secondary+0x314/0x36a
<4>[78689.783893] Code: 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 48 83 ec 40 48 89 5d d8 4c 89 65 e0 4c 89 6d e8 4c 89 75 f0 4c 89 7d f8 0f 1f 44 00 00 <48> 83 bf 48 02 00 00 00 49 89 fc 48 89 f3 41 89 d7 0f 84 82 00
<1>[78689.785050] RIP [<ffffffff810d1d11>] ub_sock_tcp_chargerecv+0x21/0x220
<4>[78689.785326] RSP <ffff8830f1203920>
<4>[78689.785587] CR2: 0000000000000248
We can upload core dumps if it need and etc.
We have node crash on 2.6.32-042stab133.1 with
<1>[78689.758852] BUG: unable to handle kernel NULL pointer dereference at 0000000000000248
<1>[78689.758892] IP: [<ffffffff810d1d11>] ub_sock_tcp_chargerecv+0x21/0x220
<4>[78689.758923] Kernel PGD 0
<4>[78689.758940] User PGD 0
<4>[78689.758957] Oops: 0000 [#1] SMP
<4>[78689.758981] last sysfs file: /sys/devices/system/node/node1/numastat
<4>[78689.759001] CPU 17
<4>[78689.759011] Modules linked in: sch_sfq cls_u32 sch_htb coretemp xt_limit ts_kmp xt_string ipt_addrtype drbg ansi_cprng ctr twofish_x86_64 twofish_common camellia serpent blowfish cast5 des_generic cbc crypto_null aesni_intel ablk_helper cryptd lrw gf128mul glue_helper aes_x86_64 aes_generic xcbc rmd160 sha512_generic xt_TCPMSS ipt_REJECT xt_conntrack xt_multiport vzethdev pio_kaio pio_nfs pio_direct pfmt_raw pfmt_ploop1 ploop simfs vziolimit vzdquota ip6t_REJECT ip6table_mangle vzevent 8021q garp stp llc vznetdev bonding xt_comment iptable_filter ip6table_filter ip6_tables ipip tunnel4 nf_conntrack_ftp xt_recent vzrst vzcpt vzmon vzdev nfs lockd fscache auth_rpcgss nfs_acl sunrpc ipt_MASQUERADE iptable_raw iptable_mangle xt_connlimit ipt_REDIRECT xt_owner nf_conntrack_ipv6 nf_defrag_ipv6 xt_state ipt_LOG xfrm_ipcomp xfrm4_mode_transport pppol2tp pppox xfrm6_mode_tunnel xfrm4_mode_tunnel esp6 ipv6 esp4 af_key arc4 ecb ppp_mppe ppp_deflate zlib_deflate ppp_async ppp_generic slhc crc_ccitt fuse iptable_nat nf_nat ip_tables nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 tun freq_table mperf power_meter acpi_ipmi ipmi_si ipmi_msghandler iTCO_wdt iTCO_vendor_support joydev dcdbas sb_edac edac_core lpc_ich mfd_core shpchp sg tg3 ptp pps_core ext4 jbd2 mbcache sd_mod crc_t10dif megaraid_sas wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]
<4>[78689.761497]
<4>[78689.761756] Pid: 0, comm: swapper veid: 0 Not tainted 2.6.32-042stab133.1 #1 042stab133 Dell Inc. PowerEdge R720/0C4Y3R
<4>[78689.762297] RIP: 0010:[<ffffffff810d1d11>] [<ffffffff810d1d11>] ub_sock_tcp_chargerecv+0x21/0x220
<4>[78689.762831] RSP: 0018:ffff8830f1203920 EFLAGS: 00010282
<4>[78689.763097] RAX: ffff885b1f040a80 RBX: 0000000000000000 RCX: 0000000000000000
<4>[78689.763367] RDX: 0000000000000002 RSI: ffff885b1f040a80 RDI: 0000000000000000
<4>[78689.763662] RBP: ffff8830f1203960 R08: ffffffff81ade580 R09: ffff885ffd9c809c
<4>[78689.763937] R10: 0000000000000002 R11: 0000000000000002 R12: ffff8859560e8340
<4>[78689.764208] R13: 00000000e7ae6aaa R14: ffff885b1f040a80 R15: 0000000000000e80
<4>[78689.764487] FS: 0000000000000000(0000) GS:ffff8830f1200000(0000) knlGS:0000000000000000
<4>[78689.765013] CS: 0010 DS: 0018 ES: 0018 CR0: 0000000080050033
<4>[78689.765279] CR2: 0000000000000248 CR3: 0000000001a8e000 CR4: 00000000000607e0
<4>[78689.765565] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
<4>[78689.765847] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
<4>[78689.766125] Process swapper (pid: 0, veid: 0, threadinfo ffff883017264000, task ffff88301725c180)
<4>[78689.766677] Stack:
<4>[78689.766941] ffff8830ffffffff ffff886017077340 0000000000000000 0000000000000000
<4>[78689.767225] <d> ffff8859560e8340 00000000e7ae6aaa ffff885b1f040a80 0000000000000e80
<4>[78689.767769] <d> ffff8830f1203a10 ffffffff814edc73 ffff8830f1203990 ffff8830f12039c0
<4>[78689.768643] Call Trace:
<4>[78689.768898] <IRQ>
<4>[78689.769163] [<ffffffff814edc73>] tcp_collapse+0x213/0x400
<4>[78689.769429] [<ffffffff814ee10f>] tcp_try_rmem_schedule+0x2af/0x3d0
<4>[78689.769722] [<ffffffff814ee600>] tcp_data_queue+0x3d0/0xcd0
<4>[78689.769991] [<ffffffff81568325>] ? _read_unlock_bh+0x15/0x20
<4>[78689.770260] [<ffffffff814ef0b0>] ? tcp_validate_incoming+0x1b0/0x320
<4>[78689.770538] [<ffffffff814f1d19>] tcp_rcv_established+0x3d9/0x860
<4>[78689.770860] [<ffffffffa02e6d15>] tcp_v6_do_rcv+0x3d5/0x720 [ipv6]
<4>[78689.771147] [<ffffffffa035e436>] ? ipv6_confirm+0x136/0x1e0 [nf_conntrack_ipv6]
<4>[78689.771699] [<ffffffffa02e95e6>] tcp_v6_rcv+0x626/0x9c0 [ipv6]
<4>[78689.771974] [<ffffffffa02c33a7>] ip6_input_finish+0x177/0x430 [ipv6]
<4>[78689.772248] [<ffffffffa02c36b8>] ip6_input+0x58/0x60 [ipv6]
<4>[78689.772521] [<ffffffffa02c2da5>] ip6_rcv_finish+0x45/0x50 [ipv6]
<4>[78689.772793] [<ffffffffa02c3150>] ipv6_rcv+0x3a0/0x480 [ipv6]
<4>[78689.773061] [<ffffffff814d71f7>] ? ip_rcv+0x287/0x370
<4>[78689.773330] [<ffffffff8149989a>] __netif_receive_skb+0x24a/0x600
<4>[78689.773598] [<ffffffff81499cea>] process_backlog+0x9a/0x100
<4>[78689.779774] [<ffffffff8149f3e6>] net_rx_action+0x116/0x330
<4>[78689.780046] [<ffffffff81090bea>] __do_softirq+0x12a/0x2b0
<4>[78689.780320] [<ffffffff815719cc>] call_softirq+0x1c/0x30
<4>[78689.780592] [<ffffffff8100eb85>] do_softirq+0x65/0xa0
<4>[78689.780861] [<ffffffff810907f5>] irq_exit+0xd5/0xe0
<4>[78689.781130] [<ffffffff81572825>] do_IRQ+0x75/0xf0
<4>[78689.781397] [<ffffffff81570f13>] ret_from_intr+0x0/0x11
<4>[78689.781663] <EOI>
<4>[78689.781932] [<ffffffff813192ae>] ? intel_idle+0x12e/0x250
<4>[78689.782198] [<ffffffff81319291>] ? intel_idle+0x111/0x250
<4>[78689.782563] [<ffffffff81013a59>] ? sched_clock+0x9/0x10
<4>[78689.782828] [<ffffffff810bcaed>] ? sched_clock_cpu+0xcd/0x110
<4>[78689.783096] [<ffffffff81460cde>] cpuidle_idle_call+0x8e/0xf0
<4>[78689.783361] [<ffffffff8100a179>] cpu_idle+0xb9/0x110
<4>[78689.783627] [<ffffffff8155d6b0>] start_secondary+0x314/0x36a
<4>[78689.783893] Code: 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 48 83 ec 40 48 89 5d d8 4c 89 65 e0 4c 89 6d e8 4c 89 75 f0 4c 89 7d f8 0f 1f 44 00 00 <48> 83 bf 48 02 00 00 00 49 89 fc 48 89 f3 41 89 d7 0f 84 82 00
<1>[78689.785050] RIP [<ffffffff810d1d11>] ub_sock_tcp_chargerecv+0x21/0x220
<4>[78689.785326] RSP <ffff8830f1203920>
<4>[78689.785587] CR2: 0000000000000248
We can upload core dumps if it need and etc.