Details
-
Type: Bug
-
Status: Resolved
-
Priority: Major
-
Resolution: Fixed
-
Fix Version/s: OpenVZ-legacy
-
Component/s: Containers::Kernel
-
Security Level: Public
-
Environment:Hardware: Intel R2208WTTR (Motherboard: S2600WTTR)
Network interface: Ethernet controller: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 (rev 01)
Software: Scientific Linux 6.8 x86_64
Kernel: 2.6.32-042stab117.16
Description
>Description of problem:
Random reboot on the server (it may possible to trigger with a specially crafted packet but this not yet confirmed)
>How reproducible:
Not yet confirmed to reproducible
>Steps to Reproduce:
1.
2.
3.
>Actual results:
<1>[ 7282.072723] BUG: unable to handle kernel paging request at ffff880972b46c0c
<1>[ 7282.073140] IP: [<ffffffffa01d7862>] ixgbe_xmit_frame_ring+0x2d2/0xe90 [ixgbe]
<4>[ 7282.073658] PGD 1a8e063 PUD 0
<4>[ 7282.074040] Oops: 0000 [#1] SMP
<4>[ 7282.074421] last sysfs file: /sys/devices/system/cpu/online
<4>[ 7282.074682] CPU 0
<4>[ 7282.074753] Modules linked in: vzethdev pio_kaio pio_nfs pio_direct pfmt_raw pfmt_ploop1 ploop simfs vzrst vzcpt nfs lockd fscache auth_rpcgss nfs_acl sunrpc vziolimit vzdquota xt_owner xt_mac ipt_REDIRECT xt_state xt_length xt_hl xt_tcpmss xt_TCPMSS xt_DSCP xt_dscp xt_limit vzevent xt_recent iptable_nat xt_comment xt_AUDIT ipt_REJECT ipt_addrtype xt_MARK iptable_mangle xt_helper iptable_raw xt_multiport xt_conntrack ipt_ULOG xt_NFLOG nfnetlink_log ipt_LOG nf_nat_tftp nf_nat_snmp_basic nf_conntrack_snmp nf_nat_sip nf_nat_pptp nf_nat_proto_gre nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat_amanda nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 ts_kmp nf_conntrack_amanda nf_conntrack_sane nf_conntrack_tftp nf_conntrack_sip nf_conntrack_proto_udplite nf_conntrack_proto_sctp nf_conntrack_pptp nf_conntrack_proto_gre nf_conntrack_netlink nfnetlink nf_conntrack_netbios_ns nf_conntrack_broadcast nf_conntrack_irc nf_conntrack_h323 nf_conntrack_ftp nf_conntrack iptable_filter ip_tables tun vznetdev vzmon vzdev ipv6 ipmi_devintf iTCO_wdt iTCO_vendor_support power_meter acpi_ipmi ipmi_si ipmi_msghandler joydev sb_edac edac_core raid10 lpc_ich mfd_core i2c_i801 i2c_core sg ixgbe dca ptp pps_core mdio ext4 jbd2 mbcache raid1 sd_mod crc_t10dif mpt3sas scsi_transport_sas raid_class xhci_hcd ahci wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]
<4>[ 7282.086520]
<4>[ 7282.086766] Pid: 0, comm: swapper veid: 0 Not tainted 2.6.32-042stab117.16 #1 042stab117_16 Intel Corporation S2600WTTR/S2600WTTR
<4>[ 7282.087417] RIP: 0010:[<ffffffffa01d7862>] [<ffffffffa01d7862>] ixgbe_xmit_frame_ring+0x2d2/0xe90 [ixgbe]
<4>[ 7282.087936] RSP: 0018:ffff88004a203670 EFLAGS: 00010286
<4>[ 7282.088196] RAX: 0000000000000001 RBX: ffffc9001703b2b0 RCX: ffff880872b46c60
<4>[ 7282.088461] RDX: ffff880872b46c00 RSI: ffff88086db6d000 RDI: ffff880972b46bff
<4>[ 7282.088727] RBP: ffff88004a2036f0 R08: ffff880878ca0860 R09: ffff880871c60436
<4>[ 7282.088992] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88086db6d380
<4>[ 7282.089258] R13: ffff88033fe9a1c0 R14: ffff88033fe9a1c0 R15: 0000000000000000
<4>[ 7282.089525] FS: 0000000000000000(0000) GS:ffff88004a200000(0000) knlGS:0000000000000000
<4>[ 7282.089969] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
<4>[ 7282.090229] CR2: ffff880972b46c0c CR3: 0000000001a8d000 CR4: 00000000001407f0
<4>[ 7282.090494] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
<4>[ 7282.090761] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
<4>[ 7282.091028] Process swapper (pid: 0, veid: 0, threadinfo ffffffff81a00000, task ffffffff81a95020)
<4>[ 7282.091476] Stack:
<4>[ 7282.091723] ffff880872b46c52 0000000000000046 0000000000000004 0000000000000246
<4>[ 7282.092181] <d> 000000005804c2b4 000000003afbfe0c ffff88004a2036c0 ffffffff8147ab7e
<4>[ 7282.092944] <d> ffff88033fe9a1c0 00ff88087326e6e8 ffff880876352a80 ffff88033fe9a1c0
<4>[ 7282.093945] Call Trace:
<4>[ 7282.094194] <IRQ>
<4>[ 7282.094510] [<ffffffff8147ab7e>] ? __skb_clone+0x2e/0x170
<4>[ 7282.094776] [<ffffffffa01d8463>] ixgbe_xmit_frame+0x43/0x90 [ixgbe]
<4>[ 7282.095041] [<ffffffff8148f58a>] dev_hard_start_xmit+0x11a/0x500
<4>[ 7282.095305] [<ffffffff81485164>] ? skb_tx_hash+0x14/0x20
<4>[ 7282.095565] [<ffffffff814852c8>] ? __netdev_pick_tx+0x158/0x1b0
<4>[ 7282.095844] [<ffffffff814ad0ba>] sch_direct_xmit+0x15a/0x1c0
<4>[ 7282.096106] [<ffffffff8148fbe8>] dev_queue_xmit+0x278/0x370
<4>[ 7282.096366] [<ffffffff814cc704>] ip_finish_output+0x184/0x340
<4>[ 7282.096627] [<ffffffff814cc978>] ip_output+0xb8/0xc0
<4>[ 7282.096885] [<ffffffff814cbbff>] ? __ip_local_out+0x9f/0xb0
<4>[ 7282.097144] [<ffffffff814cbc35>] ip_local_out+0x25/0x30
<4>[ 7282.097404] [<ffffffffa0465468>] reject_tg+0x3e8/0x3f0 [ipt_REJECT]
<4>[ 7282.097667] [<ffffffffa0461132>] ? addrtype_mt_v1+0xb2/0x110 [ipt_addrtype]
<4>[ 7282.097935] [<ffffffffa035534b>] ipt_do_table+0x3fb/0x6b0 [ip_tables]
<4>[ 7282.098202] [<ffffffff81068c91>] ? enqueue_boosted_entity+0x41/0xc0
<4>[ 7282.098466] [<ffffffffa0355215>] ? ipt_do_table+0x2c5/0x6b0 [ip_tables]
<4>[ 7282.098735] [<ffffffffa035b163>] ipt_local_in_hook+0x23/0x28 [iptable_filter]
<4>[ 7282.099178] [<ffffffff814baed9>] nf_iterate+0x69/0xb0
<4>[ 7282.099438] [<ffffffff814c6410>] ? ip_local_deliver_finish+0x0/0x310
<4>[ 7282.099701] [<ffffffff814bb096>] nf_hook_slow+0x76/0x120
<4>[ 7282.099961] [<ffffffff814c6410>] ? ip_local_deliver_finish+0x0/0x310
<4>[ 7282.100227] [<ffffffff814c677a>] ip_local_deliver+0x5a/0xa0
<4>[ 7282.100487] [<ffffffff814c5c2d>] ip_rcv_finish+0x12d/0x440
<4>[ 7282.100747] [<ffffffff814c61c4>] ip_rcv+0x284/0x360
<4>[ 7282.101006] [<ffffffff81489168>] __netif_receive_skb+0x248/0x5f0
<4>[ 7282.101270] [<ffffffff8148d138>] netif_receive_skb+0x58/0x60
<4>[ 7282.101531] [<ffffffff8148d240>] napi_skb_finish+0x50/0x70
<4>[ 7282.101793] [<ffffffff8148ead9>] napi_gro_receive+0x39/0x50
<4>[ 7282.102058] [<ffffffffa01d2ca7>] ixgbe_clean_rx_irq+0x507/0xc10 [ixgbe]
<4>[ 7282.102328] [<ffffffffa01d387c>] ixgbe_poll+0x39c/0x790 [ixgbe]
<4>[ 7282.102594] [<ffffffff81015189>] ? read_tsc+0x9/0x10
<4>[ 7282.102854] [<ffffffff810bd3df>] ? ktime_get+0x6f/0x100
<4>[ 7282.103114] [<ffffffff810bd3df>] ? ktime_get+0x6f/0x100
<4>[ 7282.103375] [<ffffffff8148ec00>] net_rx_action+0x110/0x310
<4>[ 7282.103636] [<ffffffff8108cb65>] __do_softirq+0x125/0x2a0
<4>[ 7282.103896] [<ffffffff8100c4cc>] call_softirq+0x1c/0x30
<4>[ 7282.104156] [<ffffffff810103c5>] do_softirq+0x65/0xa0
<4>[ 7282.104415] [<ffffffff8108c96d>] irq_exit+0xcd/0xd0
<4>[ 7282.104674] [<ffffffff8155d385>] do_IRQ+0x75/0xf0
<4>[ 7282.104931] [<ffffffff8100bb13>] ret_from_intr+0x0/0x11
<4>[ 7282.105191] <EOI>
<4>[ 7282.105504] [<ffffffff8130beae>] ? intel_idle+0xfe/0x1b0
<4>[ 7282.105779] [<ffffffff8130be91>] ? intel_idle+0xe1/0x1b0
<4>[ 7282.106038] [<ffffffff81015139>] ? sched_clock+0x9/0x10
<4>[ 7282.106297] [<ffffffff810b888d>] ? sched_clock_cpu+0xcd/0x110
<4>[ 7282.106559] [<ffffffff81450d2a>] cpuidle_idle_call+0x7a/0xe0
<4>[ 7282.106821] [<ffffffff8100a046>] cpu_idle+0xb6/0x110
<4>[ 7282.107081] [<ffffffff81539725>] rest_init+0x85/0x90
<4>[ 7282.107339] [<ffffffff81c48109>] start_kernel+0x406/0x413
<4>[ 7282.107599] [<ffffffff81c4733a>] x86_64_start_reservations+0x125/0x129
<4>[ 7282.107862] [<ffffffff81c47453>] x86_64_start_kernel+0x115/0x124
<4>[ 7282.108121] Code: dd 0f 84 e2 06 00 00 66 83 ff 08 0f 85 cc 00 00 00 80 79 09 06 0f 85 c2 00 00 00 41 8b be c4 00 00 00 48 01 d7 0f 84 b2 00 00 00 <0f> b6 57 0d f6 c2 01 0f 85 a5 00 00 00 80 e2 02 75 0b 41 3a 44
<1>[ 7282.112927] RIP [<ffffffffa01d7862>] ixgbe_xmit_frame_ring+0x2d2/0xe90 [ixgbe]
<4>[ 7282.113439] RSP <ffff88004a203670>
<4>[ 7282.113691] CR2: ffff880972b46c0c
>Expected results:
Continue to work
>Host OS:
Scientific Linux 6.8 x86_64
>Guest OS:
No guest OS running on the server
>Additional info (see https://openvz.org/Reporting_OpenVZ_problem):
Related reports:
https://www.centos.org/forums/viewtopic.php?t=58879
https://access.redhat.com/solutions/2374831
https://patchwork.ozlabs.org/patch/208450/
https://github.com/MISL-EBU-System-SW/mainline-public/commit/c6f408996c625cb950cad024f90e50519f94713c
Random reboot on the server (it may possible to trigger with a specially crafted packet but this not yet confirmed)
>How reproducible:
Not yet confirmed to reproducible
>Steps to Reproduce:
1.
2.
3.
>Actual results:
<1>[ 7282.072723] BUG: unable to handle kernel paging request at ffff880972b46c0c
<1>[ 7282.073140] IP: [<ffffffffa01d7862>] ixgbe_xmit_frame_ring+0x2d2/0xe90 [ixgbe]
<4>[ 7282.073658] PGD 1a8e063 PUD 0
<4>[ 7282.074040] Oops: 0000 [#1] SMP
<4>[ 7282.074421] last sysfs file: /sys/devices/system/cpu/online
<4>[ 7282.074682] CPU 0
<4>[ 7282.074753] Modules linked in: vzethdev pio_kaio pio_nfs pio_direct pfmt_raw pfmt_ploop1 ploop simfs vzrst vzcpt nfs lockd fscache auth_rpcgss nfs_acl sunrpc vziolimit vzdquota xt_owner xt_mac ipt_REDIRECT xt_state xt_length xt_hl xt_tcpmss xt_TCPMSS xt_DSCP xt_dscp xt_limit vzevent xt_recent iptable_nat xt_comment xt_AUDIT ipt_REJECT ipt_addrtype xt_MARK iptable_mangle xt_helper iptable_raw xt_multiport xt_conntrack ipt_ULOG xt_NFLOG nfnetlink_log ipt_LOG nf_nat_tftp nf_nat_snmp_basic nf_conntrack_snmp nf_nat_sip nf_nat_pptp nf_nat_proto_gre nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat_amanda nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 ts_kmp nf_conntrack_amanda nf_conntrack_sane nf_conntrack_tftp nf_conntrack_sip nf_conntrack_proto_udplite nf_conntrack_proto_sctp nf_conntrack_pptp nf_conntrack_proto_gre nf_conntrack_netlink nfnetlink nf_conntrack_netbios_ns nf_conntrack_broadcast nf_conntrack_irc nf_conntrack_h323 nf_conntrack_ftp nf_conntrack iptable_filter ip_tables tun vznetdev vzmon vzdev ipv6 ipmi_devintf iTCO_wdt iTCO_vendor_support power_meter acpi_ipmi ipmi_si ipmi_msghandler joydev sb_edac edac_core raid10 lpc_ich mfd_core i2c_i801 i2c_core sg ixgbe dca ptp pps_core mdio ext4 jbd2 mbcache raid1 sd_mod crc_t10dif mpt3sas scsi_transport_sas raid_class xhci_hcd ahci wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]
<4>[ 7282.086520]
<4>[ 7282.086766] Pid: 0, comm: swapper veid: 0 Not tainted 2.6.32-042stab117.16 #1 042stab117_16 Intel Corporation S2600WTTR/S2600WTTR
<4>[ 7282.087417] RIP: 0010:[<ffffffffa01d7862>] [<ffffffffa01d7862>] ixgbe_xmit_frame_ring+0x2d2/0xe90 [ixgbe]
<4>[ 7282.087936] RSP: 0018:ffff88004a203670 EFLAGS: 00010286
<4>[ 7282.088196] RAX: 0000000000000001 RBX: ffffc9001703b2b0 RCX: ffff880872b46c60
<4>[ 7282.088461] RDX: ffff880872b46c00 RSI: ffff88086db6d000 RDI: ffff880972b46bff
<4>[ 7282.088727] RBP: ffff88004a2036f0 R08: ffff880878ca0860 R09: ffff880871c60436
<4>[ 7282.088992] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88086db6d380
<4>[ 7282.089258] R13: ffff88033fe9a1c0 R14: ffff88033fe9a1c0 R15: 0000000000000000
<4>[ 7282.089525] FS: 0000000000000000(0000) GS:ffff88004a200000(0000) knlGS:0000000000000000
<4>[ 7282.089969] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
<4>[ 7282.090229] CR2: ffff880972b46c0c CR3: 0000000001a8d000 CR4: 00000000001407f0
<4>[ 7282.090494] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
<4>[ 7282.090761] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
<4>[ 7282.091028] Process swapper (pid: 0, veid: 0, threadinfo ffffffff81a00000, task ffffffff81a95020)
<4>[ 7282.091476] Stack:
<4>[ 7282.091723] ffff880872b46c52 0000000000000046 0000000000000004 0000000000000246
<4>[ 7282.092181] <d> 000000005804c2b4 000000003afbfe0c ffff88004a2036c0 ffffffff8147ab7e
<4>[ 7282.092944] <d> ffff88033fe9a1c0 00ff88087326e6e8 ffff880876352a80 ffff88033fe9a1c0
<4>[ 7282.093945] Call Trace:
<4>[ 7282.094194] <IRQ>
<4>[ 7282.094510] [<ffffffff8147ab7e>] ? __skb_clone+0x2e/0x170
<4>[ 7282.094776] [<ffffffffa01d8463>] ixgbe_xmit_frame+0x43/0x90 [ixgbe]
<4>[ 7282.095041] [<ffffffff8148f58a>] dev_hard_start_xmit+0x11a/0x500
<4>[ 7282.095305] [<ffffffff81485164>] ? skb_tx_hash+0x14/0x20
<4>[ 7282.095565] [<ffffffff814852c8>] ? __netdev_pick_tx+0x158/0x1b0
<4>[ 7282.095844] [<ffffffff814ad0ba>] sch_direct_xmit+0x15a/0x1c0
<4>[ 7282.096106] [<ffffffff8148fbe8>] dev_queue_xmit+0x278/0x370
<4>[ 7282.096366] [<ffffffff814cc704>] ip_finish_output+0x184/0x340
<4>[ 7282.096627] [<ffffffff814cc978>] ip_output+0xb8/0xc0
<4>[ 7282.096885] [<ffffffff814cbbff>] ? __ip_local_out+0x9f/0xb0
<4>[ 7282.097144] [<ffffffff814cbc35>] ip_local_out+0x25/0x30
<4>[ 7282.097404] [<ffffffffa0465468>] reject_tg+0x3e8/0x3f0 [ipt_REJECT]
<4>[ 7282.097667] [<ffffffffa0461132>] ? addrtype_mt_v1+0xb2/0x110 [ipt_addrtype]
<4>[ 7282.097935] [<ffffffffa035534b>] ipt_do_table+0x3fb/0x6b0 [ip_tables]
<4>[ 7282.098202] [<ffffffff81068c91>] ? enqueue_boosted_entity+0x41/0xc0
<4>[ 7282.098466] [<ffffffffa0355215>] ? ipt_do_table+0x2c5/0x6b0 [ip_tables]
<4>[ 7282.098735] [<ffffffffa035b163>] ipt_local_in_hook+0x23/0x28 [iptable_filter]
<4>[ 7282.099178] [<ffffffff814baed9>] nf_iterate+0x69/0xb0
<4>[ 7282.099438] [<ffffffff814c6410>] ? ip_local_deliver_finish+0x0/0x310
<4>[ 7282.099701] [<ffffffff814bb096>] nf_hook_slow+0x76/0x120
<4>[ 7282.099961] [<ffffffff814c6410>] ? ip_local_deliver_finish+0x0/0x310
<4>[ 7282.100227] [<ffffffff814c677a>] ip_local_deliver+0x5a/0xa0
<4>[ 7282.100487] [<ffffffff814c5c2d>] ip_rcv_finish+0x12d/0x440
<4>[ 7282.100747] [<ffffffff814c61c4>] ip_rcv+0x284/0x360
<4>[ 7282.101006] [<ffffffff81489168>] __netif_receive_skb+0x248/0x5f0
<4>[ 7282.101270] [<ffffffff8148d138>] netif_receive_skb+0x58/0x60
<4>[ 7282.101531] [<ffffffff8148d240>] napi_skb_finish+0x50/0x70
<4>[ 7282.101793] [<ffffffff8148ead9>] napi_gro_receive+0x39/0x50
<4>[ 7282.102058] [<ffffffffa01d2ca7>] ixgbe_clean_rx_irq+0x507/0xc10 [ixgbe]
<4>[ 7282.102328] [<ffffffffa01d387c>] ixgbe_poll+0x39c/0x790 [ixgbe]
<4>[ 7282.102594] [<ffffffff81015189>] ? read_tsc+0x9/0x10
<4>[ 7282.102854] [<ffffffff810bd3df>] ? ktime_get+0x6f/0x100
<4>[ 7282.103114] [<ffffffff810bd3df>] ? ktime_get+0x6f/0x100
<4>[ 7282.103375] [<ffffffff8148ec00>] net_rx_action+0x110/0x310
<4>[ 7282.103636] [<ffffffff8108cb65>] __do_softirq+0x125/0x2a0
<4>[ 7282.103896] [<ffffffff8100c4cc>] call_softirq+0x1c/0x30
<4>[ 7282.104156] [<ffffffff810103c5>] do_softirq+0x65/0xa0
<4>[ 7282.104415] [<ffffffff8108c96d>] irq_exit+0xcd/0xd0
<4>[ 7282.104674] [<ffffffff8155d385>] do_IRQ+0x75/0xf0
<4>[ 7282.104931] [<ffffffff8100bb13>] ret_from_intr+0x0/0x11
<4>[ 7282.105191] <EOI>
<4>[ 7282.105504] [<ffffffff8130beae>] ? intel_idle+0xfe/0x1b0
<4>[ 7282.105779] [<ffffffff8130be91>] ? intel_idle+0xe1/0x1b0
<4>[ 7282.106038] [<ffffffff81015139>] ? sched_clock+0x9/0x10
<4>[ 7282.106297] [<ffffffff810b888d>] ? sched_clock_cpu+0xcd/0x110
<4>[ 7282.106559] [<ffffffff81450d2a>] cpuidle_idle_call+0x7a/0xe0
<4>[ 7282.106821] [<ffffffff8100a046>] cpu_idle+0xb6/0x110
<4>[ 7282.107081] [<ffffffff81539725>] rest_init+0x85/0x90
<4>[ 7282.107339] [<ffffffff81c48109>] start_kernel+0x406/0x413
<4>[ 7282.107599] [<ffffffff81c4733a>] x86_64_start_reservations+0x125/0x129
<4>[ 7282.107862] [<ffffffff81c47453>] x86_64_start_kernel+0x115/0x124
<4>[ 7282.108121] Code: dd 0f 84 e2 06 00 00 66 83 ff 08 0f 85 cc 00 00 00 80 79 09 06 0f 85 c2 00 00 00 41 8b be c4 00 00 00 48 01 d7 0f 84 b2 00 00 00 <0f> b6 57 0d f6 c2 01 0f 85 a5 00 00 00 80 e2 02 75 0b 41 3a 44
<1>[ 7282.112927] RIP [<ffffffffa01d7862>] ixgbe_xmit_frame_ring+0x2d2/0xe90 [ixgbe]
<4>[ 7282.113439] RSP <ffff88004a203670>
<4>[ 7282.113691] CR2: ffff880972b46c0c
>Expected results:
Continue to work
>Host OS:
Scientific Linux 6.8 x86_64
>Guest OS:
No guest OS running on the server
>Additional info (see https://openvz.org/Reporting_OpenVZ_problem):
Related reports:
https://www.centos.org/forums/viewtopic.php?t=58879
https://access.redhat.com/solutions/2374831
https://patchwork.ozlabs.org/patch/208450/
https://github.com/MISL-EBU-System-SW/mainline-public/commit/c6f408996c625cb950cad024f90e50519f94713c