Details
-
Type:
Bug
-
Status: Resolved
-
Priority:
Critical
-
Resolution: Fixed
-
Fix Version/s: OpenVZ-legacy
-
Component/s: Containers::Kernel
-
Security Level: Public
-
Environment:Centos 6, kernel 2.6.32-042stab120.3
vzctl version 4.9.4
ploop-1.15-1.x86_64
Description
Hello!
After upgrade to 2.6.32-042stab120.3 from 2.6.32-042stab117.16 I saw strange activity via disk and freezy vzctl start processes.
On 2.6.32-042stab117.16 this node correctly start with all containers about 10 min.
But on stab120.3 this node cannot boot correct from 30+ min.
It have freezed processes like -
uptime
15:27:39 up 28 min, 1 user, load average: 8.20, 8.56, 11.57
ps aux | grep vzctl
root 2879 0.0 0.0 30980 2036 ? S 14:59 0:00 /usr/sbin/vzctl start 19458
root 2907 0.0 0.0 30980 2052 ? S 14:59 0:00 /usr/sbin/vzctl start 82932
root 2923 0.0 0.0 30980 2052 ? S 14:59 0:00 /usr/sbin/vzctl start 97569
root 3367 0.0 0.0 30980 824 ? D 14:59 0:00 /usr/sbin/vzctl start 97569
root 3799 0.0 0.0 30980 760 ? D 14:59 0:00 /usr/sbin/vzctl start 19458
root 4756 0.0 0.0 30980 772 ? D 14:59 0:00 /usr/sbin/vzctl start 82932
And many warnings in dmesg like
[ 241.466339] INFO: task vzctl:3367 blocked for more than 120 seconds.
[ 241.466512] Not tainted 2.6.32-042stab120.3 #1
[ 241.466680] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 241.466963] vzctl D ffff881065ae4240 0 3367 1 0 0x00000080
[ 241.467345] ffff88106545fbc8 0000000000000086 0000000000000000 0000000000800000
[ 241.467814] ffff881064f12980 ffff88106545fbe0 00000005d429e0a3 ffff881000000000
[ 241.470922] ffff881000000000 0000000000000000 00000000fffbcb85 ffffffff810b5071
[ 241.471391] Call Trace:
[ 241.471561] [<ffffffff810b5071>] ? lock_hrtimer_base+0x31/0x60
[ 241.471736] [<ffffffff81555595>] schedule_timeout+0x215/0x2e0
[ 241.471910] [<ffffffff8107b284>] ? enqueue_task_fair+0xa4/0x1f0
[ 241.472081] [<ffffffff81067f06>] ? enqueue_task+0x66/0x80
[ 241.472250] [<ffffffff815552f4>] wait_for_completion+0xe4/0x120
[ 241.472425] [<ffffffff8106d420>] ? default_wake_function+0x0/0x20
[ 241.472606] [<ffffffffa0502379>] vps_rst_restore_tree+0x259/0x2c0 [vzrst]
[ 241.472780] [<ffffffffa050461c>] vps_rst_undump+0x18c/0x210 [vzrst]
[ 241.472953] [<ffffffffa05014c1>] rst_ioctl+0x681/0x790 [vzrst]
[ 241.473125] [<ffffffffa0500e40>] ? rst_ioctl+0x0/0x790 [vzrst]
[ 241.473298] [<ffffffff8123276e>] proc_reg_unlocked_ioctl+0xde/0x100
[ 241.473476] [<ffffffff811d6882>] vfs_ioctl+0x22/0xa0
[ 241.473648] [<ffffffff810c90bb>] ? ub_memory_uncharge+0x5b/0x90
[ 241.473821] [<ffffffff811d6d66>] do_vfs_ioctl+0x3c6/0x5b0
[ 241.473991] [<ffffffff811d6f9f>] sys_ioctl+0x4f/0x80
[ 241.474163] [<ffffffff8110657e>] ? __audit_syscall_exit+0x25e/0x290
[ 241.474338] [<ffffffff8100b1a2>] system_call_fastpath+0x16/0x1b
[ 241.474517] INFO: task vzctl:3799 blocked for more than 120 seconds.
[ 241.474687] Not tainted 2.6.32-042stab120.3 #1
[ 241.474854] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 241.475142] vzctl D ffff881063506f40 0 3799 1 0 0x00000080
[ 241.475521] ffff88106342bbc8 0000000000000086 0000000000000000 0000000000800000
[ 241.475986] ffff881060825180 ffff88106342bbe0 000000061c0789d7 ffff881000000000
[ 241.476453] ffff881000000000 0000000000000000 00000000fffbd03b ffff880069e5f428
[ 241.476918] Call Trace:
[ 241.477083] [<ffffffff81078168>] ? update_cfs_shares+0x108/0x160
[ 241.477255] [<ffffffff81555595>] schedule_timeout+0x215/0x2e0
[ 241.477431] [<ffffffff8107b2dd>] ? enqueue_task_fair+0xfd/0x1f0
[ 241.477604] [<ffffffff81067f06>] ? enqueue_task+0x66/0x80
[ 241.477777] [<ffffffff815552f4>] wait_for_completion+0xe4/0x120
[ 241.477951] [<ffffffff8106d420>] ? default_wake_function+0x0/0x20
[ 241.478127] [<ffffffffa0502379>] vps_rst_restore_tree+0x259/0x2c0 [vzrst]
[ 241.478298] [<ffffffffa050461c>] vps_rst_undump+0x18c/0x210 [vzrst]
[ 241.478469] [<ffffffffa05014c1>] rst_ioctl+0x681/0x790 [vzrst]
[ 241.478636] [<ffffffffa0500e40>] ? rst_ioctl+0x0/0x790 [vzrst]
[ 241.478800] [<ffffffff8123276e>] proc_reg_unlocked_ioctl+0xde/0x100
[ 241.478966] [<ffffffff811d6882>] vfs_ioctl+0x22/0xa0
[ 241.479130] [<ffffffff810c90bb>] ? ub_memory_uncharge+0x5b/0x90
[ 241.479295] [<ffffffff811d6d66>] do_vfs_ioctl+0x3c6/0x5b0
[ 241.479463] [<ffffffff811d6f9f>] sys_ioctl+0x4f/0x80
[ 241.479627] [<ffffffff8110657e>] ? __audit_syscall_exit+0x25e/0x290
[ 241.479793] [<ffffffff8100b1a2>] system_call_fastpath+0x16/0x1b
[ 241.479964] INFO: task systemd:4451 blocked for more than 120 seconds.
[ 241.480133] Not tainted 2.6.32-042stab120.3 #1
[ 241.480300] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 241.480590] systemd D ffff881060825180 0 4451 3799 19458 0x00000084
[ 241.480965] ffff88106045bc70 0000000000000086 0000000000000000 0000000000800000
[ 241.481432] ffff881003b64d80 ffff88106045bc88 0000000e06f0a65a ffffffff0000008e
[ 241.481894] ffff88106045bc20 0000000000000000 00000000fffc549b ffffffff00000000
[ 241.482359] Call Trace:
[ 241.482525] [<ffffffff81555595>] schedule_timeout+0x215/0x2e0
[ 241.482695] [<ffffffff8107b284>] ? enqueue_task_fair+0xa4/0x1f0
[ 241.482867] [<ffffffff81067f06>] ? enqueue_task+0x66/0x80
[ 241.483038] [<ffffffff815552f4>] wait_for_completion+0xe4/0x120
[ 241.483206] [<ffffffff8106d420>] ? default_wake_function+0x0/0x20
[ 241.483380] [<ffffffffa0502680>] make_baby+0x2a0/0x420 [vzrst]
[ 241.483554] [<ffffffffa0503326>] hook+0xb26/0x1af0 [vzrst]
[ 241.483725] [<ffffffff8108010e>] ? __mmdrop+0x4e/0x80
[ 241.483894] [<ffffffff81076eb8>] ? finish_task_switch+0xc8/0x120
[ 241.484067] [<ffffffffa0502800>] ? hook+0x0/0x1af0 [vzrst]
[ 241.484240] [<ffffffffa05007ff>] child_rip+0xa/0x1b [vzrst]
[ 241.484414] [<ffffffffa0502800>] ? hook+0x0/0x1af0 [vzrst]
[ 241.484585] [<ffffffffa0502800>] ? hook+0x0/0x1af0 [vzrst]
[ 241.484756] [<ffffffffa05007f5>] ? child_rip+0x0/0x1b [vzrst]
[ 241.484931] INFO: task vzctl:4756 blocked for more than 120 seconds.
[ 241.485102] Not tainted 2.6.32-042stab120.3 #1
[ 241.485271] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 241.485560] vzctl D ffff88105fac4c80 0 4756 1 0 0x00000080
[ 241.485935] ffff88105e43fbc8 0000000000000082 0000000000000000 0000000000800000
[ 241.486404] ffff881064fcb200 ffff88105e43fbe0 0000000674b03da3 ffff881000000000
[ 241.486867] ffff881000000000 0000000000000000 00000000fffbd60b ffffffff810b5071
[ 241.487336] Call Trace:
[ 241.487509] [<ffffffff810b5071>] ? lock_hrtimer_base+0x31/0x60
[ 241.487684] [<ffffffff81555595>] schedule_timeout+0x215/0x2e0
[ 241.487857] [<ffffffff8107b284>] ? enqueue_task_fair+0xa4/0x1f0
[ 241.488031] [<ffffffff81067f06>] ? enqueue_task+0x66/0x80
[ 241.488203] [<ffffffff815552f4>] wait_for_completion+0xe4/0x120
[ 241.488379] [<ffffffff8106d420>] ? default_wake_function+0x0/0x20
[ 241.488558] [<ffffffffa0502379>] vps_rst_restore_tree+0x259/0x2c0 [vzrst]
[ 241.488734] [<ffffffffa050461c>] vps_rst_undump+0x18c/0x210 [vzrst]
[ 241.488909] [<ffffffffa05014c1>] rst_ioctl+0x681/0x790 [vzrst]
[ 241.489084] [<ffffffffa0500e40>] ? rst_ioctl+0x0/0x790 [vzrst]
[ 241.489265] [<ffffffff8123276e>] proc_reg_unlocked_ioctl+0xde/0x100
[ 241.489441] [<ffffffff811d6882>] vfs_ioctl+0x22/0xa0
[ 241.489612] [<ffffffff810c90bb>] ? ub_memory_uncharge+0x5b/0x90
[ 241.489785] [<ffffffff811d6d66>] do_vfs_ioctl+0x3c6/0x5b0
[ 241.489954] [<ffffffff811d6f9f>] sys_ioctl+0x4f/0x80
[ 241.490124] [<ffffffff8110657e>] ? __audit_syscall_exit+0x25e/0x290
[ 241.490298] [<ffffffff8100b1a2>] system_call_fastpath+0x16/0x1b
[ 241.490495] INFO: task upstart-udev-br:7363 blocked for more than 120 seconds.
[ 241.490780] Not tainted 2.6.32-042stab120.3 #1
[ 241.490948] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 241.491233] upstart-udev- D ffff8810390ca6c0 0 7363 4019 97569 0x00000084
[ 241.491614] ffff88102fe13e70 0000000000000086 0000000000000000 ffff8810637ebdf8
[ 241.492080] 0000000000000001 ffff8810637ebe10 0000000b26ec1297 0000000000000000
[ 241.492547] ffff88102fe13df0 0000000000000000 00000000fffc2664 ffffffff81061909
[ 241.492999] Call Trace:
[ 241.493161] [<ffffffff81061909>] ? __wake_up_common+0x59/0x90
[ 241.493328] [<ffffffffa050291b>] hook+0x11b/0x1af0 [vzrst]
[ 241.493497] [<ffffffff8115a26e>] ? free_pages+0x3e/0x40
[ 241.493662] [<ffffffff8108010e>] ? __mmdrop+0x4e/0x80
[ 241.493825] [<ffffffff81076eb8>] ? finish_task_switch+0xc8/0x120
[ 241.493993] [<ffffffffa0502800>] ? hook+0x0/0x1af0 [vzrst]
[ 241.494160] [<ffffffffa05007ff>] child_rip+0xa/0x1b [vzrst]
[ 241.494330] [<ffffffffa0502800>] ? hook+0x0/0x1af0 [vzrst]
[ 241.494503] [<ffffffffa0502800>] ? hook+0x0/0x1af0 [vzrst]
[ 241.494675] [<ffffffffa05007f5>] ? child_rip+0x0/0x1b [vzrst]
[ 241.494846] INFO: task udevd:7368 blocked for more than 120 seconds.
[ 241.495016] Not tainted 2.6.32-042stab120.3 #1
[ 241.495184] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 241.495473] udevd D ffff88102fe32480 0 7368 4019 97569 0x00000080
[ 241.495850] ffff88102fe3fe70 0000000000000082 00000000000001c8 ffff8810637ebdf8
[ 241.496313] 0000000000000001 ffff8810637ebe10 0000000000000000 0000000000000000
[ 241.496778] ffff88102fe3fdf0 ffffffff8106d432 ffff88102fe3fe40 ffffffff81061909
[ 241.497241] Call Trace:
[ 241.497407] [<ffffffff8106d432>] ? default_wake_function+0x12/0x20
[ 241.497584] [<ffffffff81061909>] ? __wake_up_common+0x59/0x90
[ 241.497758] [<ffffffffa050291b>] hook+0x11b/0x1af0 [vzrst]
[ 241.497929] [<ffffffff8108913a>] ? do_exit+0x64a/0x9e0
[ 241.498099] [<ffffffff81076eb8>] ? finish_task_switch+0xc8/0x120
[ 241.498274] [<ffffffffa0502800>] ? hook+0x0/0x1af0 [vzrst]
[ 241.498449] [<ffffffffa05007ff>] child_rip+0xa/0x1b [vzrst]
[ 241.498621] [<ffffffffa0502800>] ? hook+0x0/0x1af0 [vzrst]
[ 241.498790] [<ffffffffa0502800>] ? hook+0x0/0x1af0 [vzrst]
[ 241.498961] [<ffffffffa05007f5>] ? child_rip+0x0/0x1b [vzrst]
[ 241.499131] INFO: task udevd:7376 blocked for more than 120 seconds.
[ 241.499302] Not tainted 2.6.32-042stab120.3 #1
[ 241.499471] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 241.499761] udevd D ffff88102fea6540 0 7376 7368 97569 0x00000084
[ 241.500137] ffff88102feb7e70 0000000000000086 0000000000000000 ffff88102fe3fdf8
[ 241.500603] 0000000000000001 ffff88102fe3fe10 0000000b3b621526 0000000000000000
[ 241.501067] ffff88102feb7df0 0000000000000000 00000000fffc2664 ffffffff81061909
[ 241.501534] Call Trace:
[ 241.501698] [<ffffffff81061909>] ? __wake_up_common+0x59/0x90
[ 241.501869] [<ffffffffa050291b>] hook+0x11b/0x1af0 [vzrst]
[ 241.502037] [<ffffffff81076eb8>] ? finish_task_switch+0xc8/0x120
[ 241.502209] [<ffffffffa0502800>] ? hook+0x0/0x1af0 [vzrst]
[ 241.502385] [<ffffffffa05007ff>] child_rip+0xa/0x1b [vzrst]
[ 241.502558] [<ffffffffa0502800>] ? hook+0x0/0x1af0 [vzrst]
[ 241.502730] [<ffffffffa0502800>] ? hook+0x0/0x1af0 [vzrst]
[ 241.502901] [<ffffffffa05007f5>] ? child_rip+0x0/0x1b [vzrst]
[ 241.503072] INFO: task udevd:7377 blocked for more than 120 seconds.
[ 241.503243] Not tainted 2.6.32-042stab120.3 #1
[ 241.503414] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 241.506335] udevd D ffff88102fe32f80 0 7377 7368 97569 0x00000080
[ 241.506714] ffff88102fccfe70 0000000000000086 00000000000001c8 ffff88102fe3fdf8
[ 241.507166] 0000000000000001 ffff88102fe3fe10 0000000000000000 0000000000000000
[ 241.507623] ffff88102fccfdf0 ffffffff8106d432 ffff88102fccfe40 ffffffff81061909
[ 241.508076] Call Trace:
[ 241.508235] [<ffffffff8106d432>] ? default_wake_function+0x12/0x20
[ 241.508403] [<ffffffff81061909>] ? __wake_up_common+0x59/0x90
[ 241.508573] [<ffffffffa050291b>] hook+0x11b/0x1af0 [vzrst]
[ 241.508743] [<ffffffff810b65e4>] ? switch_task_namespaces+0x24/0x60
[ 241.508915] [<ffffffff8108913a>] ? do_exit+0x64a/0x9e0
[ 241.509084] [<ffffffff81076eb8>] ? finish_task_switch+0xc8/0x120
[ 241.509256] [<ffffffffa0502800>] ? hook+0x0/0x1af0 [vzrst]
[ 241.509429] [<ffffffffa05007ff>] child_rip+0xa/0x1b [vzrst]
[ 241.509602] [<ffffffffa0502800>] ? hook+0x0/0x1af0 [vzrst]
[ 241.509774] [<ffffffffa0502800>] ? hook+0x0/0x1af0 [vzrst]
[ 241.509945] [<ffffffffa05007f5>] ? child_rip+0x0/0x1b [vzrst]
[ 241.510114] INFO: task upstart-socket-:7378 blocked for more than 120 seconds.
[ 241.510401] Not tainted 2.6.32-042stab120.3 #1
[ 241.510567] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 241.510852] upstart-socke D ffff88102fcd1080 0 7378 4019 97569 0x00000084
[ 241.511228] ffff88102fcd7e70 0000000000000082 0000000000000000 ffff8810637ebdf8
[ 241.511696] 0000000000000001 ffff8810637ebe10 0000000b5ff04850 0000000000000000
[ 241.512159] ffff88102fcd7df0 0000000000000000 00000000fffc2a73 ffffffff81061909
[ 241.512624] Call Trace:
[ 241.512789] [<ffffffff81061909>] ? __wake_up_common+0x59/0x90
[ 241.512961] [<ffffffffa050291b>] hook+0x11b/0x1af0 [vzrst]
[ 241.513129] [<ffffffff8108913a>] ? do_exit+0x64a/0x9e0
[ 241.513301] [<ffffffff81076eb8>] ? finish_task_switch+0xc8/0x120
[ 241.513477] [<ffffffffa0502800>] ? hook+0x0/0x1af0 [vzrst]
[ 241.513648] [<ffffffffa05007ff>] child_rip+0xa/0x1b [vzrst]
[ 241.513818] [<ffffffffa0502800>] ? hook+0x0/0x1af0 [vzrst]
[ 241.513989] [<ffffffffa0502800>] ? hook+0x0/0x1af0 [vzrst]
[ 241.514161] [<ffffffffa05007f5>] ? child_rip+0x0/0x1b [vzrst]
[ 241.514331] INFO: task rsyslogd:7389 blocked for more than 120 seconds.
[ 241.514504] Not tainted 2.6.32-042stab120.3 #1
[ 241.514674] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 241.514962] rsyslogd D ffff881058d70440 0 7389 4019 97569 0x00000080
[ 241.515336] ffff88102b08fe70 0000000000000086 00000000000001c8 ffff8810637ebdf8
[ 241.515804] 0000000000000001 ffff8810637ebe10 0000000000000000 0000000000000000
[ 241.516268] ffff88102b08fdf0 ffffffff8106d432 ffff88102b08fe40 ffffffff81061909
[ 241.516736] Call Trace:
[ 241.516901] [<ffffffff8106d432>] ? default_wake_function+0x12/0x20
[ 241.517073] [<ffffffff81061909>] ? __wake_up_common+0x59/0x90
[ 241.517244] [<ffffffffa050291b>] hook+0x11b/0x1af0 [vzrst]
[ 241.517417] [<ffffffff81076eb8>] ? finish_task_switch+0xc8/0x120
[ 241.517590] [<ffffffffa0502800>] ? hook+0x0/0x1af0 [vzrst]
[ 241.517762] [<ffffffffa05007ff>] child_rip+0xa/0x1b [vzrst]
[ 241.517934] [<ffffffffa0502800>] ? hook+0x0/0x1af0 [vzrst]
[ 241.518106] [<ffffffffa0502800>] ? hook+0x0/0x1af0 [vzrst]
[ 241.518276] [<ffffffffa05007f5>] ? child_rip+0x0/0x1b [vzrst]
After upgrade to 2.6.32-042stab120.3 from 2.6.32-042stab117.16 I saw strange activity via disk and freezy vzctl start processes.
On 2.6.32-042stab117.16 this node correctly start with all containers about 10 min.
But on stab120.3 this node cannot boot correct from 30+ min.
It have freezed processes like -
uptime
15:27:39 up 28 min, 1 user, load average: 8.20, 8.56, 11.57
ps aux | grep vzctl
root 2879 0.0 0.0 30980 2036 ? S 14:59 0:00 /usr/sbin/vzctl start 19458
root 2907 0.0 0.0 30980 2052 ? S 14:59 0:00 /usr/sbin/vzctl start 82932
root 2923 0.0 0.0 30980 2052 ? S 14:59 0:00 /usr/sbin/vzctl start 97569
root 3367 0.0 0.0 30980 824 ? D 14:59 0:00 /usr/sbin/vzctl start 97569
root 3799 0.0 0.0 30980 760 ? D 14:59 0:00 /usr/sbin/vzctl start 19458
root 4756 0.0 0.0 30980 772 ? D 14:59 0:00 /usr/sbin/vzctl start 82932
And many warnings in dmesg like
[ 241.466339] INFO: task vzctl:3367 blocked for more than 120 seconds.
[ 241.466512] Not tainted 2.6.32-042stab120.3 #1
[ 241.466680] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 241.466963] vzctl D ffff881065ae4240 0 3367 1 0 0x00000080
[ 241.467345] ffff88106545fbc8 0000000000000086 0000000000000000 0000000000800000
[ 241.467814] ffff881064f12980 ffff88106545fbe0 00000005d429e0a3 ffff881000000000
[ 241.470922] ffff881000000000 0000000000000000 00000000fffbcb85 ffffffff810b5071
[ 241.471391] Call Trace:
[ 241.471561] [<ffffffff810b5071>] ? lock_hrtimer_base+0x31/0x60
[ 241.471736] [<ffffffff81555595>] schedule_timeout+0x215/0x2e0
[ 241.471910] [<ffffffff8107b284>] ? enqueue_task_fair+0xa4/0x1f0
[ 241.472081] [<ffffffff81067f06>] ? enqueue_task+0x66/0x80
[ 241.472250] [<ffffffff815552f4>] wait_for_completion+0xe4/0x120
[ 241.472425] [<ffffffff8106d420>] ? default_wake_function+0x0/0x20
[ 241.472606] [<ffffffffa0502379>] vps_rst_restore_tree+0x259/0x2c0 [vzrst]
[ 241.472780] [<ffffffffa050461c>] vps_rst_undump+0x18c/0x210 [vzrst]
[ 241.472953] [<ffffffffa05014c1>] rst_ioctl+0x681/0x790 [vzrst]
[ 241.473125] [<ffffffffa0500e40>] ? rst_ioctl+0x0/0x790 [vzrst]
[ 241.473298] [<ffffffff8123276e>] proc_reg_unlocked_ioctl+0xde/0x100
[ 241.473476] [<ffffffff811d6882>] vfs_ioctl+0x22/0xa0
[ 241.473648] [<ffffffff810c90bb>] ? ub_memory_uncharge+0x5b/0x90
[ 241.473821] [<ffffffff811d6d66>] do_vfs_ioctl+0x3c6/0x5b0
[ 241.473991] [<ffffffff811d6f9f>] sys_ioctl+0x4f/0x80
[ 241.474163] [<ffffffff8110657e>] ? __audit_syscall_exit+0x25e/0x290
[ 241.474338] [<ffffffff8100b1a2>] system_call_fastpath+0x16/0x1b
[ 241.474517] INFO: task vzctl:3799 blocked for more than 120 seconds.
[ 241.474687] Not tainted 2.6.32-042stab120.3 #1
[ 241.474854] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 241.475142] vzctl D ffff881063506f40 0 3799 1 0 0x00000080
[ 241.475521] ffff88106342bbc8 0000000000000086 0000000000000000 0000000000800000
[ 241.475986] ffff881060825180 ffff88106342bbe0 000000061c0789d7 ffff881000000000
[ 241.476453] ffff881000000000 0000000000000000 00000000fffbd03b ffff880069e5f428
[ 241.476918] Call Trace:
[ 241.477083] [<ffffffff81078168>] ? update_cfs_shares+0x108/0x160
[ 241.477255] [<ffffffff81555595>] schedule_timeout+0x215/0x2e0
[ 241.477431] [<ffffffff8107b2dd>] ? enqueue_task_fair+0xfd/0x1f0
[ 241.477604] [<ffffffff81067f06>] ? enqueue_task+0x66/0x80
[ 241.477777] [<ffffffff815552f4>] wait_for_completion+0xe4/0x120
[ 241.477951] [<ffffffff8106d420>] ? default_wake_function+0x0/0x20
[ 241.478127] [<ffffffffa0502379>] vps_rst_restore_tree+0x259/0x2c0 [vzrst]
[ 241.478298] [<ffffffffa050461c>] vps_rst_undump+0x18c/0x210 [vzrst]
[ 241.478469] [<ffffffffa05014c1>] rst_ioctl+0x681/0x790 [vzrst]
[ 241.478636] [<ffffffffa0500e40>] ? rst_ioctl+0x0/0x790 [vzrst]
[ 241.478800] [<ffffffff8123276e>] proc_reg_unlocked_ioctl+0xde/0x100
[ 241.478966] [<ffffffff811d6882>] vfs_ioctl+0x22/0xa0
[ 241.479130] [<ffffffff810c90bb>] ? ub_memory_uncharge+0x5b/0x90
[ 241.479295] [<ffffffff811d6d66>] do_vfs_ioctl+0x3c6/0x5b0
[ 241.479463] [<ffffffff811d6f9f>] sys_ioctl+0x4f/0x80
[ 241.479627] [<ffffffff8110657e>] ? __audit_syscall_exit+0x25e/0x290
[ 241.479793] [<ffffffff8100b1a2>] system_call_fastpath+0x16/0x1b
[ 241.479964] INFO: task systemd:4451 blocked for more than 120 seconds.
[ 241.480133] Not tainted 2.6.32-042stab120.3 #1
[ 241.480300] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 241.480590] systemd D ffff881060825180 0 4451 3799 19458 0x00000084
[ 241.480965] ffff88106045bc70 0000000000000086 0000000000000000 0000000000800000
[ 241.481432] ffff881003b64d80 ffff88106045bc88 0000000e06f0a65a ffffffff0000008e
[ 241.481894] ffff88106045bc20 0000000000000000 00000000fffc549b ffffffff00000000
[ 241.482359] Call Trace:
[ 241.482525] [<ffffffff81555595>] schedule_timeout+0x215/0x2e0
[ 241.482695] [<ffffffff8107b284>] ? enqueue_task_fair+0xa4/0x1f0
[ 241.482867] [<ffffffff81067f06>] ? enqueue_task+0x66/0x80
[ 241.483038] [<ffffffff815552f4>] wait_for_completion+0xe4/0x120
[ 241.483206] [<ffffffff8106d420>] ? default_wake_function+0x0/0x20
[ 241.483380] [<ffffffffa0502680>] make_baby+0x2a0/0x420 [vzrst]
[ 241.483554] [<ffffffffa0503326>] hook+0xb26/0x1af0 [vzrst]
[ 241.483725] [<ffffffff8108010e>] ? __mmdrop+0x4e/0x80
[ 241.483894] [<ffffffff81076eb8>] ? finish_task_switch+0xc8/0x120
[ 241.484067] [<ffffffffa0502800>] ? hook+0x0/0x1af0 [vzrst]
[ 241.484240] [<ffffffffa05007ff>] child_rip+0xa/0x1b [vzrst]
[ 241.484414] [<ffffffffa0502800>] ? hook+0x0/0x1af0 [vzrst]
[ 241.484585] [<ffffffffa0502800>] ? hook+0x0/0x1af0 [vzrst]
[ 241.484756] [<ffffffffa05007f5>] ? child_rip+0x0/0x1b [vzrst]
[ 241.484931] INFO: task vzctl:4756 blocked for more than 120 seconds.
[ 241.485102] Not tainted 2.6.32-042stab120.3 #1
[ 241.485271] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 241.485560] vzctl D ffff88105fac4c80 0 4756 1 0 0x00000080
[ 241.485935] ffff88105e43fbc8 0000000000000082 0000000000000000 0000000000800000
[ 241.486404] ffff881064fcb200 ffff88105e43fbe0 0000000674b03da3 ffff881000000000
[ 241.486867] ffff881000000000 0000000000000000 00000000fffbd60b ffffffff810b5071
[ 241.487336] Call Trace:
[ 241.487509] [<ffffffff810b5071>] ? lock_hrtimer_base+0x31/0x60
[ 241.487684] [<ffffffff81555595>] schedule_timeout+0x215/0x2e0
[ 241.487857] [<ffffffff8107b284>] ? enqueue_task_fair+0xa4/0x1f0
[ 241.488031] [<ffffffff81067f06>] ? enqueue_task+0x66/0x80
[ 241.488203] [<ffffffff815552f4>] wait_for_completion+0xe4/0x120
[ 241.488379] [<ffffffff8106d420>] ? default_wake_function+0x0/0x20
[ 241.488558] [<ffffffffa0502379>] vps_rst_restore_tree+0x259/0x2c0 [vzrst]
[ 241.488734] [<ffffffffa050461c>] vps_rst_undump+0x18c/0x210 [vzrst]
[ 241.488909] [<ffffffffa05014c1>] rst_ioctl+0x681/0x790 [vzrst]
[ 241.489084] [<ffffffffa0500e40>] ? rst_ioctl+0x0/0x790 [vzrst]
[ 241.489265] [<ffffffff8123276e>] proc_reg_unlocked_ioctl+0xde/0x100
[ 241.489441] [<ffffffff811d6882>] vfs_ioctl+0x22/0xa0
[ 241.489612] [<ffffffff810c90bb>] ? ub_memory_uncharge+0x5b/0x90
[ 241.489785] [<ffffffff811d6d66>] do_vfs_ioctl+0x3c6/0x5b0
[ 241.489954] [<ffffffff811d6f9f>] sys_ioctl+0x4f/0x80
[ 241.490124] [<ffffffff8110657e>] ? __audit_syscall_exit+0x25e/0x290
[ 241.490298] [<ffffffff8100b1a2>] system_call_fastpath+0x16/0x1b
[ 241.490495] INFO: task upstart-udev-br:7363 blocked for more than 120 seconds.
[ 241.490780] Not tainted 2.6.32-042stab120.3 #1
[ 241.490948] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 241.491233] upstart-udev- D ffff8810390ca6c0 0 7363 4019 97569 0x00000084
[ 241.491614] ffff88102fe13e70 0000000000000086 0000000000000000 ffff8810637ebdf8
[ 241.492080] 0000000000000001 ffff8810637ebe10 0000000b26ec1297 0000000000000000
[ 241.492547] ffff88102fe13df0 0000000000000000 00000000fffc2664 ffffffff81061909
[ 241.492999] Call Trace:
[ 241.493161] [<ffffffff81061909>] ? __wake_up_common+0x59/0x90
[ 241.493328] [<ffffffffa050291b>] hook+0x11b/0x1af0 [vzrst]
[ 241.493497] [<ffffffff8115a26e>] ? free_pages+0x3e/0x40
[ 241.493662] [<ffffffff8108010e>] ? __mmdrop+0x4e/0x80
[ 241.493825] [<ffffffff81076eb8>] ? finish_task_switch+0xc8/0x120
[ 241.493993] [<ffffffffa0502800>] ? hook+0x0/0x1af0 [vzrst]
[ 241.494160] [<ffffffffa05007ff>] child_rip+0xa/0x1b [vzrst]
[ 241.494330] [<ffffffffa0502800>] ? hook+0x0/0x1af0 [vzrst]
[ 241.494503] [<ffffffffa0502800>] ? hook+0x0/0x1af0 [vzrst]
[ 241.494675] [<ffffffffa05007f5>] ? child_rip+0x0/0x1b [vzrst]
[ 241.494846] INFO: task udevd:7368 blocked for more than 120 seconds.
[ 241.495016] Not tainted 2.6.32-042stab120.3 #1
[ 241.495184] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 241.495473] udevd D ffff88102fe32480 0 7368 4019 97569 0x00000080
[ 241.495850] ffff88102fe3fe70 0000000000000082 00000000000001c8 ffff8810637ebdf8
[ 241.496313] 0000000000000001 ffff8810637ebe10 0000000000000000 0000000000000000
[ 241.496778] ffff88102fe3fdf0 ffffffff8106d432 ffff88102fe3fe40 ffffffff81061909
[ 241.497241] Call Trace:
[ 241.497407] [<ffffffff8106d432>] ? default_wake_function+0x12/0x20
[ 241.497584] [<ffffffff81061909>] ? __wake_up_common+0x59/0x90
[ 241.497758] [<ffffffffa050291b>] hook+0x11b/0x1af0 [vzrst]
[ 241.497929] [<ffffffff8108913a>] ? do_exit+0x64a/0x9e0
[ 241.498099] [<ffffffff81076eb8>] ? finish_task_switch+0xc8/0x120
[ 241.498274] [<ffffffffa0502800>] ? hook+0x0/0x1af0 [vzrst]
[ 241.498449] [<ffffffffa05007ff>] child_rip+0xa/0x1b [vzrst]
[ 241.498621] [<ffffffffa0502800>] ? hook+0x0/0x1af0 [vzrst]
[ 241.498790] [<ffffffffa0502800>] ? hook+0x0/0x1af0 [vzrst]
[ 241.498961] [<ffffffffa05007f5>] ? child_rip+0x0/0x1b [vzrst]
[ 241.499131] INFO: task udevd:7376 blocked for more than 120 seconds.
[ 241.499302] Not tainted 2.6.32-042stab120.3 #1
[ 241.499471] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 241.499761] udevd D ffff88102fea6540 0 7376 7368 97569 0x00000084
[ 241.500137] ffff88102feb7e70 0000000000000086 0000000000000000 ffff88102fe3fdf8
[ 241.500603] 0000000000000001 ffff88102fe3fe10 0000000b3b621526 0000000000000000
[ 241.501067] ffff88102feb7df0 0000000000000000 00000000fffc2664 ffffffff81061909
[ 241.501534] Call Trace:
[ 241.501698] [<ffffffff81061909>] ? __wake_up_common+0x59/0x90
[ 241.501869] [<ffffffffa050291b>] hook+0x11b/0x1af0 [vzrst]
[ 241.502037] [<ffffffff81076eb8>] ? finish_task_switch+0xc8/0x120
[ 241.502209] [<ffffffffa0502800>] ? hook+0x0/0x1af0 [vzrst]
[ 241.502385] [<ffffffffa05007ff>] child_rip+0xa/0x1b [vzrst]
[ 241.502558] [<ffffffffa0502800>] ? hook+0x0/0x1af0 [vzrst]
[ 241.502730] [<ffffffffa0502800>] ? hook+0x0/0x1af0 [vzrst]
[ 241.502901] [<ffffffffa05007f5>] ? child_rip+0x0/0x1b [vzrst]
[ 241.503072] INFO: task udevd:7377 blocked for more than 120 seconds.
[ 241.503243] Not tainted 2.6.32-042stab120.3 #1
[ 241.503414] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 241.506335] udevd D ffff88102fe32f80 0 7377 7368 97569 0x00000080
[ 241.506714] ffff88102fccfe70 0000000000000086 00000000000001c8 ffff88102fe3fdf8
[ 241.507166] 0000000000000001 ffff88102fe3fe10 0000000000000000 0000000000000000
[ 241.507623] ffff88102fccfdf0 ffffffff8106d432 ffff88102fccfe40 ffffffff81061909
[ 241.508076] Call Trace:
[ 241.508235] [<ffffffff8106d432>] ? default_wake_function+0x12/0x20
[ 241.508403] [<ffffffff81061909>] ? __wake_up_common+0x59/0x90
[ 241.508573] [<ffffffffa050291b>] hook+0x11b/0x1af0 [vzrst]
[ 241.508743] [<ffffffff810b65e4>] ? switch_task_namespaces+0x24/0x60
[ 241.508915] [<ffffffff8108913a>] ? do_exit+0x64a/0x9e0
[ 241.509084] [<ffffffff81076eb8>] ? finish_task_switch+0xc8/0x120
[ 241.509256] [<ffffffffa0502800>] ? hook+0x0/0x1af0 [vzrst]
[ 241.509429] [<ffffffffa05007ff>] child_rip+0xa/0x1b [vzrst]
[ 241.509602] [<ffffffffa0502800>] ? hook+0x0/0x1af0 [vzrst]
[ 241.509774] [<ffffffffa0502800>] ? hook+0x0/0x1af0 [vzrst]
[ 241.509945] [<ffffffffa05007f5>] ? child_rip+0x0/0x1b [vzrst]
[ 241.510114] INFO: task upstart-socket-:7378 blocked for more than 120 seconds.
[ 241.510401] Not tainted 2.6.32-042stab120.3 #1
[ 241.510567] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 241.510852] upstart-socke D ffff88102fcd1080 0 7378 4019 97569 0x00000084
[ 241.511228] ffff88102fcd7e70 0000000000000082 0000000000000000 ffff8810637ebdf8
[ 241.511696] 0000000000000001 ffff8810637ebe10 0000000b5ff04850 0000000000000000
[ 241.512159] ffff88102fcd7df0 0000000000000000 00000000fffc2a73 ffffffff81061909
[ 241.512624] Call Trace:
[ 241.512789] [<ffffffff81061909>] ? __wake_up_common+0x59/0x90
[ 241.512961] [<ffffffffa050291b>] hook+0x11b/0x1af0 [vzrst]
[ 241.513129] [<ffffffff8108913a>] ? do_exit+0x64a/0x9e0
[ 241.513301] [<ffffffff81076eb8>] ? finish_task_switch+0xc8/0x120
[ 241.513477] [<ffffffffa0502800>] ? hook+0x0/0x1af0 [vzrst]
[ 241.513648] [<ffffffffa05007ff>] child_rip+0xa/0x1b [vzrst]
[ 241.513818] [<ffffffffa0502800>] ? hook+0x0/0x1af0 [vzrst]
[ 241.513989] [<ffffffffa0502800>] ? hook+0x0/0x1af0 [vzrst]
[ 241.514161] [<ffffffffa05007f5>] ? child_rip+0x0/0x1b [vzrst]
[ 241.514331] INFO: task rsyslogd:7389 blocked for more than 120 seconds.
[ 241.514504] Not tainted 2.6.32-042stab120.3 #1
[ 241.514674] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 241.514962] rsyslogd D ffff881058d70440 0 7389 4019 97569 0x00000080
[ 241.515336] ffff88102b08fe70 0000000000000086 00000000000001c8 ffff8810637ebdf8
[ 241.515804] 0000000000000001 ffff8810637ebe10 0000000000000000 0000000000000000
[ 241.516268] ffff88102b08fdf0 ffffffff8106d432 ffff88102b08fe40 ffffffff81061909
[ 241.516736] Call Trace:
[ 241.516901] [<ffffffff8106d432>] ? default_wake_function+0x12/0x20
[ 241.517073] [<ffffffff81061909>] ? __wake_up_common+0x59/0x90
[ 241.517244] [<ffffffffa050291b>] hook+0x11b/0x1af0 [vzrst]
[ 241.517417] [<ffffffff81076eb8>] ? finish_task_switch+0xc8/0x120
[ 241.517590] [<ffffffffa0502800>] ? hook+0x0/0x1af0 [vzrst]
[ 241.517762] [<ffffffffa05007ff>] child_rip+0xa/0x1b [vzrst]
[ 241.517934] [<ffffffffa0502800>] ? hook+0x0/0x1af0 [vzrst]
[ 241.518106] [<ffffffffa0502800>] ? hook+0x0/0x1af0 [vzrst]
[ 241.518276] [<ffffffffa05007f5>] ? child_rip+0x0/0x1b [vzrst]
