Kernel hungup on i/o

Hello!

We have about ~7 faults for last half year on same hardware on different kernels.

This server do not hangup completely but disk system become broken and all applications do not work correctly.

<3>[257488.657334] INFO: task kswapd0:102 blocked for more than 120 seconds.
<3>[257488.657397] Not tainted 2.6.32-042stab108.2 #1
<3>[257488.657454] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
<6>[257488.657562] kswapd0 D ffff880816a36f00 0 102 2 0 0x00000000
<4>[257488.657673] ffff880816a397f0 0000000000000046 ffff88065ae4e000 ffff880802444840
<4>[257488.657785] ffff880815a1cf48 0000000000000008 ffff880816a36f00 ffff880816a36f00
<4>[257488.657897] ffff880816a39820 ffffffff8127d7c0 ffff88080ece0b40 ffffffff00000002
<4>[257488.658009] Call Trace:
<4>[257488.658066] [<ffffffff8127d7c0>] ? generic_make_request+0x240/0x550
<4>[257488.658126] [<ffffffff81189510>] ? wait_for_discard+0x0/0x20
<4>[257488.658184] [<ffffffff8118951e>] wait_for_discard+0xe/0x20
<4>[257488.658243] [<ffffffff8153646f>] __wait_on_bit+0x5f/0x90
<4>[257488.658305] [<ffffffff8117d961>] ? page_check_address+0x141/0x1c0
<4>[257488.658366] [<ffffffff81189510>] ? wait_for_discard+0x0/0x20
<4>[257488.658425] [<ffffffff81536518>] out_of_line_wait_on_bit+0x78/0x90
<4>[257488.658496] [<ffffffff810a88e0>] ? wake_bit_function+0x0/0x50
<4>[257488.658564] [<ffffffff8118993d>] scan_swap_map+0x40d/0x640
<4>[257488.658622] [<ffffffff81189cad>] get_swap_page+0x9d/0x140
<4>[257488.658680] [<ffffffff81186ec7>] add_to_swap+0x17/0x90
<4>[257488.658739] [<ffffffff8115a70e>] shrink_page_list.clone.0+0x2de/0x900
<4>[257488.658799] [<ffffffff8115bf0e>] shrink_inactive_list+0x3be/0xb10
<4>[257488.658859] [<ffffffff8115ca90>] shrink_lruvec+0x430/0x600
<4>[257488.658918] [<ffffffff81015059>] ? read_tsc+0x9/0x20
<4>[257488.658982] [<ffffffffa038d4be>] ? nfs_access_cache_shrinker+0x1ce/0x210 [nfs]
<4>[257488.659090] [<ffffffff8115cee7>] shrink_zone+0x287/0x3d0
<4>[257488.659148] [<ffffffff8115e173>] balance_pgdat+0xc23/0xda0
<4>[257488.659206] [<ffffffff8115e46f>] kswapd+0x17f/0x3f0
<4>[257488.659264] [<ffffffff810a8860>] ? autoremove_wake_function+0x0/0x40
<4>[257488.659324] [<ffffffff8115e2f0>] ? kswapd+0x0/0x3f0
<4>[257488.659382] [<ffffffff810a846e>] kthread+0x9e/0xc0
<4>[257488.659439] [<ffffffff8100c38a>] child_rip+0xa/0x20
<4>[257488.659497] [<ffffffff810a83d0>] ? kthread+0x0/0xc0
<4>[257488.659554] [<ffffffff8100c380>] ? child_rip+0x0/0x20
<3>[257488.659618] INFO: task flush-182:94339:1834 blocked for more than 120 seconds.
<3>[257488.659725] Not tainted 2.6.32-042stab108.2 #1
<3>[257488.659781] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
<6>[257488.659890] flush-182:943 D ffff880809201340 0 1834 2 0 0x00000080
<4>[257488.660000] ffff880809203830 0000000000000046 ffff880817b99088 ffff880809201ba8
<4>[257488.660110] ffff88081b236480 0000000000000040 ffff88019169dbc0 0000000000000010
<4>[257488.660221] ffff8808092037d0 000000007c988e0c ffff88081b236480 ffff88019169d8c0
<4>[257488.660335] Call Trace:
<4>[257488.660389] [<ffffffff81536b36>] __mutex_lock_slowpath+0x96/0x210
<4>[257488.660448] [<ffffffff8153665b>] mutex_lock+0x2b/0x50
<4>[257488.660514] [<ffffffffa00aa618>] __jbd2_log_wait_for_space+0xc8/0x1b0 [jbd2]
<4>[257488.660630] [<ffffffffa00a5f55>] start_this_handle+0x125/0x490 [jbd2]
<4>[257488.660699] [<ffffffffa00ce2be>] ? ext4_num_dirty_pages+0x1ee/0x270 [ext4]
<4>[257488.660768] [<ffffffffa00a64a5>] jbd2_journal_start+0xb5/0x100 [jbd2]
<4>[257488.660838] [<ffffffffa00e9636>] ext4_journal_start_sb+0x56/0xe0 [ext4]
<4>[257488.660904] [<ffffffffa00d4306>] ext4_da_writepages+0x286/0x660 [ext4]
<4>[257488.660965] [<ffffffff8100bc8e>] ? apic_timer_interrupt+0xe/0x20
<4>[257488.661028] [<ffffffff8100bc8e>] ? apic_timer_interrupt+0xe/0x20
<4>[257488.661089] [<ffffffff81154ce1>] do_writepages+0x21/0x40
<4>[257488.661149] [<ffffffff811e4e89>] __writeback_single_inode+0xf9/0x3d0
<4>[257488.661211] [<ffffffff811e51f8>] writeback_single_inode+0x98/0xd0
<4>[257488.661275] [<ffffffff811e549a>] writeback_sb_inodes+0xda/0x1b0
<4>[257488.661338] [<ffffffff811e5688>] writeback_inodes_wb+0x118/0x170
<4>[257488.661402] [<ffffffff811e5a1b>] wb_writeback+0x33b/0x460
<4>[257488.661461] [<ffffffff811e5cf5>] wb_do_writeback+0x1b5/0x260
<4>[257488.661519] [<ffffffff811e5e3c>] bdi_writeback_task+0x9c/0x1f0
<4>[257488.661579] [<ffffffff8116a240>] ? bdi_start_fn+0x0/0x110
<4>[257488.661637] [<ffffffff8116a240>] ? bdi_start_fn+0x0/0x110
<4>[257488.661696] [<ffffffff8116a2d5>] bdi_start_fn+0x95/0x110
<4>[257488.661754] [<ffffffff8116a240>] ? bdi_start_fn+0x0/0x110
<4>[257488.661812] [<ffffffff810a846e>] kthread+0x9e/0xc0
<4>[257488.661870] [<ffffffff8100c38a>] child_rip+0xa/0x20
<4>[257488.661927] [<ffffffff810a83d0>] ? kthread+0x0/0xc0
<4>[257488.661986] [<ffffffff8100c380>] ? child_rip+0x0/0x20
<3>[257488.662044] INFO: task jbd2/ploop58962:1887 blocked for more than 120 seconds.
<3>[257488.662153] Not tainted 2.6.32-042stab108.2 #1
<3>[257488.662209] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
<6>[257488.662320] jbd2/ploop589 D ffff88080467ebc0 0 1887 2 0 0x00000080
<4>[257488.662434] ffff88080248efb0 0000000000000046 ffff88080248ef78 ffff88039aa59000
<4>[257488.662546] 0005120000000000 0000000000100000 0000ea03ce85546e ffff88002820f7b0
<4>[257488.662658] ffff880800000000 ffff88002820f7a0 000000010f53da97 0000000100000082
<4>[257488.662769] Call Trace:
<4>[257488.662824] [<ffffffff815360b5>] schedule_timeout+0x215/0x2e0
<4>[257488.662883] [<ffffffff81015059>] ? read_tsc+0x9/0x20
<4>[257488.662943] [<ffffffff810b4801>] ? ktime_get_ts+0xb1/0xf0
<4>[257488.663002] [<ffffffff81015059>] ? read_tsc+0x9/0x20
<4>[257488.663059] [<ffffffff810b4801>] ? ktime_get_ts+0xb1/0xf0
<4>[257488.663119] [<ffffffff8153464f>] io_schedule_timeout+0x7f/0xd0
<4>[257488.663178] [<ffffffff81535a24>] wait_for_completion_io+0xe4/0x120
<4>[257488.663238] [<ffffffff81065c00>] ? default_wake_function+0x0/0x20
<4>[257488.663301] [<ffffffff81286166>] blkdev_issue_discard+0x216/0x230
<4>[257488.663365] [<ffffffff811898b5>] scan_swap_map+0x385/0x640
<4>[257488.663423] [<ffffffff81189cad>] get_swap_page+0x9d/0x140
<4>[257488.663481] [<ffffffff81186ec7>] add_to_swap+0x17/0x90
<4>[257488.663539] [<ffffffff8115a70e>] shrink_page_list.clone.0+0x2de/0x900
<4>[257488.663599] [<ffffffff8115bf0e>] shrink_inactive_list+0x3be/0xb10
<4>[257488.663658] [<ffffffff8115ca90>] shrink_lruvec+0x430/0x600
<4>[257488.663717] [<ffffffff8115cee7>] shrink_zone+0x287/0x3d0
<4>[257488.663774] [<ffffffff8115ec68>] do_try_to_free_pages+0x588/0xa60
<4>[257488.663833] [<ffffffff8115f35b>] try_to_free_pages+0x8b/0x120
<4>[257488.663893] [<ffffffff8114fe21>] __alloc_pages_nodemask+0x671/0xb50
<4>[257488.663953] [<ffffffff8119acb9>] kmem_getpages+0x59/0x140
<4>[257488.664011] [<ffffffff8119ce0b>] fallback_alloc+0x1bb/0x260
<4>[257488.664070] [<ffffffff8119cb99>] ____cache_alloc_node+0x99/0x150
<4>[257488.664129] [<ffffffff8119dba3>] kmem_cache_alloc+0x173/0x1e0
<4>[257488.664188] [<ffffffff81140e25>] mempool_alloc_slab+0x15/0x20
<4>[257488.664245] [<ffffffff81140fc7>] mempool_alloc+0x67/0x170
<4>[257488.664306] [<ffffffff81140fc7>] ? mempool_alloc+0x67/0x170
<4>[257488.664368] [<ffffffff811f4922>] bvec_alloc_bs+0xe2/0x110
<4>[257488.664426] [<ffffffff811f4a02>] bio_alloc_bioset+0xb2/0xf0
<4>[257488.664484] [<ffffffff811f4ae5>] bio_alloc+0x15/0x30
<4>[257488.664543] [<ffffffffa049e0e8>] ploop_make_request+0x5a8/0xa30 [ploop]
<4>[257488.664604] [<ffffffff8127d7c0>] generic_make_request+0x240/0x550
<4>[257488.664663] [<ffffffff81140e25>] ? mempool_alloc_slab+0x15/0x20
<4>[257488.664721] [<ffffffff8127db53>] submit_bio+0x83/0x1c0
<4>[257488.664779] [<ffffffff811f49ab>] ? bio_alloc_bioset+0x5b/0xf0
<4>[257488.664837] [<ffffffff811ee45d>] submit_bh+0x11d/0x1e0
<4>[257488.664898] [<ffffffffa00a7c18>] jbd2_journal_commit_transaction+0x5a8/0x1500 [jbd2]
<4>[257488.665009] [<ffffffff810097dd>] ? __switch_to+0x13d/0x320
<4>[257488.665068] [<ffffffff8108f90b>] ? try_to_del_timer_sync+0x7b/0xe0
<4>[257488.665129] [<ffffffffa00adbb8>] kjournald2+0xb8/0x220 [jbd2]
<4>[257488.665188] [<ffffffff810a8860>] ? autoremove_wake_function+0x0/0x40
<4>[257488.665249] [<ffffffffa00adb00>] ? kjournald2+0x0/0x220 [jbd2]
<4>[257488.665312] [<ffffffff810a846e>] kthread+0x9e/0xc0
<4>[257488.665373] [<ffffffff8100c38a>] child_rip+0xa/0x20
<4>[257488.665431] [<ffffffff810a83d0>] ? kthread+0x0/0xc0
<4>[257488.665488] [<ffffffff8100c380>] ? child_rip+0x0/0x20
<3>[257488.665551] INFO: task rs:main Q:Reg:4216 blocked for more than 120 seconds.
<3>[257488.665611] Not tainted 2.6.32-042stab108.2 #1
<3>[257488.667080] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
<6>[257488.667191] rs:main Q:Reg D ffff880815f98d40 0 4216 1902 104 0x00000080
<4>[257488.667304] ffff88080999ba38 0000000000000086 0000000000000000 ffff880815f98d40
<4>[257488.667417] ffff88080999b9f8 ffff88080999b9c8 0000ea041a08b082 ffff88002835dde8
<4>[257488.667528] 0000000000003fd0 0000000000000000 000000010f53df92 ffff88002835dde8
<4>[257488.667639] Call Trace:
<4>[257488.667693] [<ffffffff81536b36>] __mutex_lock_slowpath+0x96/0x210
<4>[257488.667752] [<ffffffff8153665b>] mutex_lock+0x2b/0x50
<4>[257488.667811] [<ffffffffa00aa618>] __jbd2_log_wait_for_space+0xc8/0x1b0 [jbd2]
<4>[257488.667920] [<ffffffffa00a5f55>] start_this_handle+0x125/0x490 [jbd2]
<4>[257488.667981] [<ffffffffa00a59b7>] ? jbd2_journal_stop+0x1e7/0x2b0 [jbd2]
<4>[257488.668042] [<ffffffffa00a64a5>] jbd2_journal_start+0xb5/0x100 [jbd2]
<4>[257488.668106] [<ffffffffa00e9636>] ext4_journal_start_sb+0x56/0xe0 [ext4]
<4>[257488.668169] [<ffffffffa00d01c2>] ext4_dirty_inode+0x32/0x70 [ext4]
<4>[257488.668229] [<ffffffff811e4b4a>] __mark_inode_dirty+0x5a/0x2a0
<4>[257488.668291] [<ffffffff811d2051>] file_update_time+0x121/0x1c0
<4>[257488.668354] [<ffffffff81140914>] __generic_file_write_iter+0x1f4/0x420
<4>[257488.668414] [<ffffffff81140bc5>] __generic_file_aio_write+0x85/0xa0
<4>[257488.668473] [<ffffffff81140c68>] generic_file_aio_write+0x88/0x100
<4>[257488.668534] [<ffffffffa00c9008>] ext4_file_write+0x58/0x190 [ext4]
<4>[257488.668594] [<ffffffff811b3cd2>] do_sync_write+0xf2/0x140
<4>[257488.668652] [<ffffffff811b3fb8>] vfs_write+0xb8/0x1a0
<4>[257488.668709] [<ffffffff811b4981>] sys_write+0x51/0x90
<4>[257488.668767] [<ffffffff810fd3ee>] ? __audit_syscall_exit+0x25e/0x290
<4>[257488.668826] [<ffffffff8100b142>] system_call_fastpath+0x16/0x1b
<3>[257488.668891] INFO: task nginx:5784 blocked for more than 120 seconds.
<3>[257488.668950] Not tainted 2.6.32-042stab108.2 #1
<3>[257488.669006] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
<6>[257488.669114] nginx D ffff880803703240 0 5784 5780 104 0x00000080
<4>[257488.669223] ffff88080a68ba38 0000000000000082 0000000000000000 ffff880803703240
<4>[257488.669341] 0000000000000000 0000000000b8213b ffff880803703240 ffffffff810a8860
<4>[257488.669456] ffff88080a68b9a8 ffff88080a68b9a8 ffff880807a8da08 ffff88080a68ba38
<4>[257488.669567] Call Trace:
<4>[257488.669621] [<ffffffff810a8860>] ? autoremove_wake_function+0x0/0x40
<4>[257488.669681] [<ffffffff81536b36>] __mutex_lock_slowpath+0x96/0x210
<4>[257488.669741] [<ffffffff814c44d7>] ? tcp_init_tso_segs+0x37/0x50
<4>[257488.669800] [<ffffffff8153665b>] mutex_lock+0x2b/0x50
<4>[257488.669860] [<ffffffffa00aa618>] __jbd2_log_wait_for_space+0xc8/0x1b0 [jbd2]
<4>[257488.669969] [<ffffffffa00a5f55>] start_this_handle+0x125/0x490 [jbd2]
<4>[257488.670030] [<ffffffffa00a64a5>] jbd2_journal_start+0xb5/0x100 [jbd2]
<4>[257488.670094] [<ffffffffa00e9636>] ext4_journal_start_sb+0x56/0xe0 [ext4]
<4>[257488.670157] [<ffffffffa00d01c2>] ext4_dirty_inode+0x32/0x70 [ext4]
<4>[257488.670217] [<ffffffff811e4b4a>] __mark_inode_dirty+0x5a/0x2a0
<4>[257488.670275] [<ffffffff811d2051>] file_update_time+0x121/0x1c0
<4>[257488.670338] [<ffffffff81457340>] ? sock_aio_write+0x0/0x180
<4>[257488.670400] [<ffffffff81140914>] __generic_file_write_iter+0x1f4/0x420
<4>[257488.670460] [<ffffffff811e6b70>] ? pipe_to_sendpage+0x0/0xa0
<4>[257488.670519] [<ffffffff81140bc5>] __generic_file_aio_write+0x85/0xa0
<4>[257488.670581] [<ffffffff81140c68>] generic_file_aio_write+0x88/0x100
<4>[257488.670644] [<ffffffffa00c9008>] ext4_file_write+0x58/0x190 [ext4]
<4>[257488.670704] [<ffffffff811b3cd2>] do_sync_write+0xf2/0x140
<4>[257488.670762] [<ffffffff811b3fb8>] vfs_write+0xb8/0x1a0
<4>[257488.670819] [<ffffffff811b4981>] sys_write+0x51/0x90
<4>[257488.670878] [<ffffffff810fd3ee>] ? __audit_syscall_exit+0x25e/0x290
<4>[257488.670937] [<ffffffff8100b142>] system_call_fastpath+0x16/0x1b
<3>[257488.671000] INFO: task master:6046 blocked for more than 120 seconds.
<3>[257488.671059] Not tainted 2.6.32-042stab108.2 #1
<3>[257488.671116] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
<6>[257488.671225] master D ffff8808088ff440 0 6046 1902 104 0x00000084
<4>[257488.671338] ffff8808050ddb08 0000000000000082 00000000000f774b 0000000000000000
<4>[257488.671454] ffff8808050dda78 ffff880807698800 0000000000000246 0000000000000246
<4>[257488.671565] ffff8808050ddaa8 0000000000000282 ffff8808050ddab8 ffffffff810a8c5c
<4>[257488.671676] Call Trace:
<4>[257488.671731] [<ffffffff810a8c5c>] ? remove_wait_queue+0x3c/0x50
<4>[257488.671790] [<ffffffff81536b36>] __mutex_lock_slowpath+0x96/0x210
<4>[257488.671849] [<ffffffff8153665b>] mutex_lock+0x2b/0x50
<4>[257488.671909] [<ffffffffa00aa618>] __jbd2_log_wait_for_space+0xc8/0x1b0 [jbd2]
<4>[257488.672017] [<ffffffffa00a5f55>] start_this_handle+0x125/0x490 [jbd2]
<4>[257488.672079] [<ffffffffa00a64a5>] jbd2_journal_start+0xb5/0x100 [jbd2]
<4>[257488.672142] [<ffffffffa00e9636>] ext4_journal_start_sb+0x56/0xe0 [ext4]
<4>[257488.672205] [<ffffffffa00d01c2>] ext4_dirty_inode+0x32/0x70 [ext4]
<4>[257488.672264] [<ffffffff811e4b4a>] __mark_inode_dirty+0x5a/0x2a0
<4>[257488.672326] [<ffffffff811d2051>] file_update_time+0x121/0x1c0
<4>[257488.672389] [<ffffffff811bfaa2>] pipe_write+0x2f2/0x690
<4>[257488.672447] [<ffffffff811b3cd2>] do_sync_write+0xf2/0x140
<4>[257488.672505] [<ffffffff810b4801>] ? ktime_get_ts+0xb1/0xf0
<4>[257488.672563] [<ffffffff811b3fb8>] vfs_write+0xb8/0x1a0
<4>[257488.672620] [<ffffffff811b4981>] sys_write+0x51/0x90
<4>[257488.672677] [<ffffffff810fd3ee>] ? __audit_syscall_exit+0x25e/0x290
<4>[257488.672736] [<ffffffff8100b142>] system_call_fastpath+0x16/0x1b
<3>[257488.672802] INFO: task pickup:453666 blocked for more than 120 seconds.
<3>[257488.672861] Not tainted 2.6.32-042stab108.2 #1
<3>[257488.672918] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
<6>[257488.673025] pickup D ffff8808008f85c0 0 453666 6046 104 0x00000080
<4>[257488.673135] ffff880807fcfa38 0000000000000082 0000000000000000 ffffffff811eefdc
<4>[257488.673246] ffff880807fcfb08 0000000000b82069 0000ea060f3fb08a 00000001008f85c0
<4>[257488.673362] 0000000000000000 0000000000000000 000000010f53ffef 0000000000000000
<4>[257488.673474] Call Trace:
<4>[257488.673527] [<ffffffff811eefdc>] ? __getblk+0x2c/0x2a0
<4>[257488.673585] [<ffffffff81536b36>] __mutex_lock_slowpath+0x96/0x210
<4>[257488.673644] [<ffffffff8153665b>] mutex_lock+0x2b/0x50
<4>[257488.673703] [<ffffffffa00aa618>] __jbd2_log_wait_for_space+0xc8/0x1b0 [jbd2]
<4>[257488.673812] [<ffffffffa00a5f55>] start_this_handle+0x125/0x490 [jbd2]
<4>[257488.673873] [<ffffffffa00a64a5>] jbd2_journal_start+0xb5/0x100 [jbd2]
<4>[257488.673933] [<ffffffff811b6955>] ? __sb_start_write+0xd5/0x1b0
<4>[257488.673995] [<ffffffffa00e9636>] ext4_journal_start_sb+0x56/0xe0 [ext4]
<4>[257488.674058] [<ffffffffa00d01c2>] ext4_dirty_inode+0x32/0x70 [ext4]
<4>[257488.674117] [<ffffffff811e4b4a>] __mark_inode_dirty+0x5a/0x2a0
<4>[257488.674176] [<ffffffff811d2285>] touch_atime+0x195/0x1a0
<4>[257488.674233] [<ffffffff811c0115>] pipe_read+0x2d5/0x4e0
<4>[257488.674293] [<ffffffff811b3e1a>] do_sync_read+0xfa/0x140
<4>[257488.674356] [<ffffffff810a8860>] ? autoremove_wake_function+0x0/0x40
<4>[257488.674415] [<ffffffff811b47b5>] vfs_read+0xb5/0x1a0
<4>[257488.674472] [<ffffffff811b48f1>] sys_read+0x51/0x90
<4>[257488.674529] [<ffffffff810fd3ee>] ? __audit_syscall_exit+0x25e/0x290
<4>[257488.674588] [<ffffffff8100b142>] system_call_fastpath+0x16/0x1b
<3>[257488.674649] INFO: task tar:457003 blocked for more than 120 seconds.
<3>[257488.674708] Not tainted 2.6.32-042stab108.2 #1
<3>[257488.674765] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
<6>[257488.674872] tar D ffff88080665ec00 0 457003 457002 104 0x00000080
<4>[257488.674982] ffff880816fd1958 0000000000000086 0000000000000002 0000000000000003
<4>[257488.675093] ffff880816fd18c8 ffffffff81061527 0000000000000003 ffff88002831dde8
<4>[257488.675204] ffff880816fd18f8 ffffffff81070978 ffff88002831dde8 0000000300000001
<4>[257488.675317] Call Trace:
<4>[257488.675374] [<ffffffff81061527>] ? account_entity_enqueue+0x47/0x80
<4>[257488.675434] [<ffffffff81070978>] ? update_cfs_shares+0x108/0x160
<4>[257488.675493] [<ffffffff810a8b8e>] ? prepare_to_wait+0x4e/0x80
<4>[257488.675554] [<ffffffffa00ad6f5>] jbd2_log_wait_commit+0xc5/0x140 [jbd2]
<4>[257488.675614] [<ffffffff8100bace>] ? common_interrupt+0xe/0x13
<4>[257488.675672] [<ffffffff810a8860>] ? autoremove_wake_function+0x0/0x40
<4>[257488.675733] [<ffffffffa00aa0d3>] jbd2_log_do_checkpoint+0xb3/0x530 [jbd2]
<4>[257488.675794] [<ffffffff8105a549>] ? __wake_up_common+0x59/0x90
<4>[257488.675854] [<ffffffffa00aa5e5>] __jbd2_log_wait_for_space+0x95/0x1b0 [jbd2]
<4>[257488.675963] [<ffffffffa00a5f55>] start_this_handle+0x125/0x490 [jbd2]
<4>[257488.676022] [<ffffffff8113e180>] ? sync_page_killable+0x0/0x40
<4>[257488.676081] [<ffffffff810a8860>] ? autoremove_wake_function+0x0/0x40
<4>[257488.676141] [<ffffffffa00a64a5>] jbd2_journal_start+0xb5/0x100 [jbd2]
<4>[257488.676201] [<ffffffff811b6955>] ? __sb_start_write+0xd5/0x1b0
<4>[257488.676263] [<ffffffffa00e9636>] ext4_journal_start_sb+0x56/0xe0 [ext4]
<4>[257488.676330] [<ffffffffa00d01c2>] ext4_dirty_inode+0x32/0x70 [ext4]
<4>[257488.676393] [<ffffffff811e4b4a>] __mark_inode_dirty+0x5a/0x2a0
<4>[257488.676452] [<ffffffff811d2285>] touch_atime+0x195/0x1a0
<4>[257488.676510] [<ffffffff81140087>] generic_file_read_iter+0x337/0x640
<4>[257488.676569] [<ffffffff8114041b>] generic_file_aio_read+0x8b/0xa0
<4>[257488.676628] [<ffffffff811b3e1a>] do_sync_read+0xfa/0x140
<4>[257488.676686] [<ffffffff811c73d4>] ? user_path_at+0x64/0xa0
<4>[257488.676744] [<ffffffff810a8860>] ? autoremove_wake_function+0x0/0x40
<4>[257488.676805] [<ffffffff811ba1d4>] ? cp_new_stat+0xe4/0x100
<4>[257488.676862] [<ffffffff811b47b5>] vfs_read+0xb5/0x1a0
<4>[257488.676919] [<ffffffff811b48f1>] sys_read+0x51/0x90
<4>[257488.676976] [<ffffffff810fd3ee>] ? __audit_syscall_exit+0x25e/0x290
<4>[257488.677035] [<ffffffff8100b142>] system_call_fastpath+0x16/0x1b
<3>[257608.607564] INFO: task kswapd0:102 blocked for more than 120 seconds.
<3>[257608.607626] Not tainted 2.6.32-042stab108.2 #1
<3>[257608.607683] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
<6>[257608.607791] kswapd0 D ffff880816a36f00 0 102 2 0 0x00000000
<4>[257608.607902] ffff880816a397f0 0000000000000046 ffff88065ae4e000 ffff880802444840
<4>[257608.609424] ffff880815a1cf48 0000000000000008 ffff880816a36f00 ffff880816a36f00
<4>[257608.609543] ffff880816a39820 ffffffff8127d7c0 ffff88080ece0b40 ffffffff00000002
<4>[257608.609656] Call Trace:
<4>[257608.609712] [<ffffffff8127d7c0>] ? generic_make_request+0x240/0x550
<4>[257608.609773] [<ffffffff81189510>] ? wait_for_discard+0x0/0x20
<4>[257608.609831] [<ffffffff8118951e>] wait_for_discard+0xe/0x20
<4>[257608.609890] [<ffffffff8153646f>] __wait_on_bit+0x5f/0x90
<4>[257608.609949] [<ffffffff8117d961>] ? page_check_address+0x141/0x1c0
<4>[257608.610007] [<ffffffff81189510>] ? wait_for_discard+0x0/0x20
<4>[257608.610066] [<ffffffff81536518>] out_of_line_wait_on_bit+0x78/0x90
<4>[257608.610127] [<ffffffff810a88e0>] ? wake_bit_function+0x0/0x50
<4>[257608.610185] [<ffffffff8118993d>] scan_swap_map+0x40d/0x640
<4>[257608.610242] [<ffffffff81189cad>] get_swap_page+0x9d/0x140
<4>[257608.610301] [<ffffffff81186ec7>] add_to_swap+0x17/0x90
<4>[257608.610361] [<ffffffff8115a70e>] shrink_page_list.clone.0+0x2de/0x900
<4>[257608.610421] [<ffffffff8115bf0e>] shrink_inactive_list+0x3be/0xb10
<4>[257608.610480] [<ffffffff8115ca90>] shrink_lruvec+0x430/0x600
<4>[257608.610542] [<ffffffff81015059>] ? read_tsc+0x9/0x20
<4>[257608.610612] [<ffffffffa038d4be>] ? nfs_access_cache_shrinker+0x1ce/0x210 [nfs]
<4>[257608.610721] [<ffffffff8115cee7>] shrink_zone+0x287/0x3d0
<4>[257608.610779] [<ffffffff8115e173>] balance_pgdat+0xc23/0xda0
<4>[257608.610837] [<ffffffff8115e46f>] kswapd+0x17f/0x3f0
<4>[257608.610895] [<ffffffff810a8860>] ? autoremove_wake_function+0x0/0x40
<4>[257608.610955] [<ffffffff8115e2f0>] ? kswapd+0x0/0x3f0
<4>[257608.611012] [<ffffffff810a846e>] kthread+0x9e/0xc0
<4>[257608.611070] [<ffffffff8100c38a>] child_rip+0xa/0x20
<4>[257608.611128] [<ffffffff810a83d0>] ? kthread+0x0/0xc0
<4>[257608.611185] [<ffffffff8100c380>] ? child_rip+0x0/0x20
<3>[257608.611249] INFO: task flush-182:94339:1834 blocked for more than 120 seconds.
<3>[257608.611356] Not tainted 2.6.32-042stab108.2 #1
<3>[257608.611413] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
<6>[257608.611523] flush-182:943 D ffff880809201340 0 1834 2 0 0x00000080
<4>[257608.611637] ffff880809203830 0000000000000046 ffff880817b99088 ffff880809201ba8
<4>[257608.611748] ffff88081b236480 0000000000000040 ffff88019169dbc0 0000000000000010
<4>[257608.611859] ffff8808092037d0 000000007c988e0c ffff88081b236480 ffff88019169d8c0
<4>[257608.611970] Call Trace:
<4>[257608.612024] [<ffffffff81536b36>] __mutex_lock_slowpath+0x96/0x210
<4>[257608.612083] [<ffffffff8153665b>] mutex_lock+0x2b/0x50
<4>[257608.612148] [<ffffffffa00aa618>] __jbd2_log_wait_for_space+0xc8/0x1b0 [jbd2]
<4>[257608.612257] [<ffffffffa00a5f55>] start_this_handle+0x125/0x490 [jbd2]
<4>[257608.612321] [<ffffffffa00ce2be>] ? ext4_num_dirty_pages+0x1ee/0x270 [ext4]
<4>[257608.612384] [<ffffffffa00a64a5>] jbd2_journal_start+0xb5/0x100 [jbd2]
<4>[257608.612448] [<ffffffffa00e9636>] ext4_journal_start_sb+0x56/0xe0 [ext4]
<4>[257608.612511] [<ffffffffa00d4306>] ext4_da_writepages+0x286/0x660 [ext4]
<4>[257608.612574] [<ffffffff8100bc8e>] ? apic_timer_interrupt+0xe/0x20
<4>[257608.612637] [<ffffffff8100bc8e>] ? apic_timer_interrupt+0xe/0x20
<4>[257608.612696] [<ffffffff81154ce1>] do_writepages+0x21/0x40
<4>[257608.612756] [<ffffffff811e4e89>] __writeback_single_inode+0xf9/0x3d0
<4>[257608.612816] [<ffffffff811e51f8>] writeback_single_inode+0x98/0xd0
<4>[257608.612875] [<ffffffff811e549a>] writeback_sb_inodes+0xda/0x1b0
<4>[257608.612934] [<ffffffff811e5688>] writeback_inodes_wb+0x118/0x170
<4>[257608.612993] [<ffffffff811e5a1b>] wb_writeback+0x33b/0x460
<4>[257608.613051] [<ffffffff811e5cf5>] wb_do_writeback+0x1b5/0x260
<4>[257608.613110] [<ffffffff811e5e3c>] bdi_writeback_task+0x9c/0x1f0
<4>[257608.613169] [<ffffffff8116a240>] ? bdi_start_fn+0x0/0x110
<4>[257608.613227] [<ffffffff8116a240>] ? bdi_start_fn+0x0/0x110
<4>[257608.613285] [<ffffffff8116a2d5>] bdi_start_fn+0x95/0x110
<4>[257608.613343] [<ffffffff8116a240>] ? bdi_start_fn+0x0/0x110
<4>[257608.613401] [<ffffffff810a846e>] kthread+0x9e/0xc0
<4>[257608.613458] [<ffffffff8100c38a>] child_rip+0xa/0x20
<4>[257608.613515] [<ffffffff810a83d0>] ? kthread+0x0/0xc0
<4>[257608.613574] [<ffffffff8100c380>] ? child_rip+0x0/0x20
<4>[259197.027216] Trigger a crash
<1>[259197.027283] BUG: unable to handle kernel NULL pointer dereference at (null)
<1>[259197.027345] IP: [<ffffffff8135afb6>] sysrq_handle_crash+0x16/0x20
<4>[259197.027409] PGD 8043ee067 PUD 8026f0067 PMD 0
<4>[259197.027468] Oops: 0002 [#1] SMP
<4>[259197.027523] last sysfs file: /sys/kernel/kexec_crash_loaded
<4>[259197.027581] CPU 4
<4>[259197.027585] Modules linked in: xt_state xt_conntrack vzethdev pio_kaio pio_nfs pio_direct pfmt_raw pfmt_ploop1 ploop simfs vzrst nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 vzcpt nfs lockd fscache auth_rpcgss nfs_acl sunrpc nf_conntrack vziolimit vzdquota ip6t_REJECT ip6table_mangle ip6table_filter ip6_tables xt_length xt_hl xt_tcpmss xt_TCPMSS iptable_mangle iptable_filter xt_multiport xt_limit xt_dscp ipt_REJECT ip_tables vzevent netconsole configfs vznetdev vzmon vzdev ipv6 cpufreq_ondemand acpi_cpufreq freq_table mperf ppdev iTCO_wdt iTCO_vendor_support parport_pc parport e1000e ptp pps_core r8169 mii serio_raw i2c_i801 i2c_core sg lpc_ich mfd_core shpchp ext4 jbd2 mbcache raid1 sd_mod crc_t10dif ahci video output wmi xhci_hcd dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]
<4>[259197.028352]
<4>[259197.028404] Pid: 460552, comm: bash veid: 0 Not tainted 2.6.32-042stab108.2 #1 042stab108_2 MSI MS-7816/H87-G43 (MS-7816)
<4>[259197.028520] RIP: 0010:[<ffffffff8135afb6>] [<ffffffff8135afb6>] sysrq_handle_crash+0x16/0x20
<4>[259197.028632] RSP: 0018:ffff88024d397dd8 EFLAGS: 00010096
<4>[259197.028689] RAX: 0000000000000023 RBX: 0000000000000063 RCX: 00000000000018a4
<4>[259197.028796] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000063
<4>[259197.028903] RBP: ffff88024d397dd8 R08: 0000000000000100 R09: ffff8808190f0860
<4>[259197.029010] R10: 0000000000000000 R11: 0000000000000003 R12: 0000000000000000
<4>[259197.029117] R13: ffffffff81b09f00 R14: 0000000000000286 R15: 0000000000000007
<4>[259197.029224] FS: 00007f77232be700(0000) GS:ffff880028300000(0000) knlGS:0000000000000000
<4>[259197.029333] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
<4>[259197.029390] CR2: 0000000000000000 CR3: 000000080244c000 CR4: 00000000001407e0
<4>[259197.029497] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
<4>[259197.029603] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
<4>[259197.029711] Process bash (pid: 460552, veid: 0, threadinfo ffff88024d396000, task ffff8808197b8480)
<4>[259197.029822] Stack:
<4>[259197.029873] ffff88024d397e28 ffffffff8135b4cc ffff8801d562d638 ffff880800000000
<4>[259197.029937] <d> ffff88024d397f18 0000000000000002 ffffffff81aad400 0000000000000000
<4>[259197.030049] <d> ffff88024d397e39 0000000000000002 ffff88024d397e98 ffffffff8135b6bf
<4>[259197.030209] Call Trace:
<4>[259197.030263] [<ffffffff8135b4cc>] __handle_sysrq+0x7c/0x170
<4>[259197.030321] [<ffffffff8135b6bf>] write_sysrq_trigger+0xff/0x130
<4>[259197.030381] [<ffffffff8122572e>] proc_reg_write+0x7e/0xc0
<4>[259197.030439] [<ffffffff811b3fb8>] vfs_write+0xb8/0x1a0
<4>[259197.030496] [<ffffffff811b4981>] sys_write+0x51/0x90
<4>[259197.030554] [<ffffffff8100b142>] system_call_fastpath+0x16/0x1b
<4>[259197.030612] Code: d0 88 81 a3 bb 3f 82 c9 c3 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 0f 1f 44 00 00 c7 05 dd 42 74 00 01 00 00 00 0f ae f8 <c6> 04 25 00 00 00 00 01 c9 c3 55 48 89 e5 0f 1f 44 00 00 48 c7
<1>[259197.030841] RIP [<ffffffff8135afb6>] sysrq_handle_crash+0x16/0x20
<4>[259197.030901] RSP <ffff88024d397dd8>
<4>[259197.030955] CR2: 0000000000000000


We have kdump and could share it with you.