Details
-
Type:
Bug
-
Status: Resolved
-
Priority:
Minor
-
Resolution: Fixed
-
Fix Version/s: OpenVZ-legacy
-
Component/s: Containers::Kernel
-
Security Level: Public
-
Environment:Operating System: Other
Platform: Other
-
External issue URL:
-
External issue ID:3253
Description
[1294400.795342] INFO: task kswapd0:102 blocked for more than 120 seconds.
[1294400.795403] Not tainted 2.6.32-042stab108.1 #1
[1294400.795459] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[1294400.795569] kswapd0 D ffff880816a36f00 0 102 2 0 0x00000000
[1294400.795682] ffff880816a39700 0000000000000046 0000000000000000 ffffea0020139300
[1294400.795792] ffff880000023600 0000000815402cc0 00049915aa9ddfb9 ffff8808162da800
[1294400.795900] 0000000000000001 0000000000000000 000000014d2b374a ffff880816a36f00
[1294400.796009] Call Trace:
[1294400.796064] [<ffffffff811894d0>] ? wait_for_discard+0x0/0x20
[1294400.796122] [<ffffffff811894de>] wait_for_discard+0xe/0x20
[1294400.796181] [<ffffffff8153642f>] __wait_on_bit+0x5f/0x90
[1294400.796239] [<ffffffff811894d0>] ? wait_for_discard+0x0/0x20
[1294400.796297] [<ffffffff815364d8>] out_of_line_wait_on_bit+0x78/0x90
[1294400.796357] [<ffffffff810a88e0>] ? wake_bit_function+0x0/0x50
[1294400.796415] [<ffffffff811898fd>] scan_swap_map+0x40d/0x640
[1294400.796472] [<ffffffff81189c6d>] get_swap_page+0x9d/0x140
[1294400.796530] [<ffffffff811863fe>] ? swap_writepage+0xde/0x120
[1294400.796592] [<ffffffff8116291c>] shmem_writepage+0xcc/0x230
[1294400.796653] [<ffffffff8115a22b>] pageout.clone.2+0x12b/0x2f0
[1294400.796711] [<ffffffff8115aa46>] shrink_page_list.clone.0+0x656/0x900
[1294400.796771] [<ffffffff8115bece>] shrink_inactive_list+0x3be/0xb10
[1294400.796830] [<ffffffff8115ca50>] shrink_lruvec+0x430/0x600
[1294400.796889] [<ffffffff81015059>] ? read_tsc+0x9/0x20
[1294400.796952] [<ffffffffa03774be>] ? nfs_access_cache_shrinker+0x1ce/0x210 [nfs]
[1294400.797060] [<ffffffff8115cea7>] shrink_zone+0x287/0x3d0
[1294400.797117] [<ffffffff8115e133>] balance_pgdat+0xc23/0xda0
[1294400.797175] [<ffffffff8115e42f>] kswapd+0x17f/0x3f0
[1294400.797232] [<ffffffff810a8860>] ? autoremove_wake_function+0x0/0x40
[1294400.797292] [<ffffffff8115e2b0>] ? kswapd+0x0/0x3f0
[1294400.797348] [<ffffffff810a846e>] kthread+0x9e/0xc0
[1294400.797406] [<ffffffff8100c38a>] child_rip+0xa/0x20
[1294400.797462] [<ffffffff810a83d0>] ? kthread+0x0/0xc0
[1294400.797519] [<ffffffff8100c380>] ? child_rip+0x0/0x20
[1294400.797583] INFO: task flush-182:44681:1686 blocked for more than 120 seconds.
[1294400.797692] Not tainted 2.6.32-042stab108.1 #1
[1294400.797751] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[1294400.797862] flush-182:446 D ffff8808165c2e40 0 1686 2 0 0x00000080
[1294400.797976] ffff88080d027830 0000000000000046 0000000000000000 ffff8808165c36a8
[1294400.798084] ffff88081b236480 0000000000000040 00049916bdd7aabb ffff88002831a538
[1294400.798200] ffff88080d027820 0000000000000000 000000014d2b4965 0000000000000001
[1294400.798309] Call Trace:
[1294400.798362] [<ffffffff81536af6>] __mutex_lock_slowpath+0x96/0x210
[1294400.798421] [<ffffffff8153661b>] mutex_lock+0x2b/0x50
[1294400.798486] [<ffffffffa00ab618>] __jbd2_log_wait_for_space+0xc8/0x1b0 [jbd2]
[1294400.798598] [<ffffffffa00a6f55>] start_this_handle+0x125/0x490 [jbd2]
[1294400.798666] [<ffffffffa00cf2be>] ? ext4_num_dirty_pages+0x1ee/0x270 [ext4]
[1294400.798728] [<ffffffffa00a74a5>] jbd2_journal_start+0xb5/0x100 [jbd2]
[1294400.798792] [<ffffffffa00ea636>] ext4_journal_start_sb+0x56/0xe0 [ext4]
[1294400.798855] [<ffffffffa00d5306>] ext4_da_writepages+0x286/0x660 [ext4]
[1294400.798915] [<ffffffff8100bc8e>] ? apic_timer_interrupt+0xe/0x20
[1294400.798974] [<ffffffff81154ca1>] do_writepages+0x21/0x40
[1294400.799032] [<ffffffff811e4e49>] __writeback_single_inode+0xf9/0x3d0
[1294400.799092] [<ffffffff811e51b8>] writeback_single_inode+0x98/0xd0
[1294400.799151] [<ffffffff811e545a>] writeback_sb_inodes+0xda/0x1b0
[1294400.799209] [<ffffffff811e5648>] writeback_inodes_wb+0x118/0x170
[1294400.799268] [<ffffffff811e59db>] wb_writeback+0x33b/0x460
[1294400.799327] [<ffffffffa0143f80>] ? do_dbs_timer+0x0/0x3bc [cpufreq_ondemand]
[1294400.799434] [<ffffffff811e5cb5>] wb_do_writeback+0x1b5/0x260
[1294400.799493] [<ffffffff811e5dfc>] bdi_writeback_task+0x9c/0x1f0
[1294400.799554] [<ffffffff8116a200>] ? bdi_start_fn+0x0/0x110
[1294400.799614] [<ffffffff8116a200>] ? bdi_start_fn+0x0/0x110
[1294400.799672] [<ffffffff8116a295>] bdi_start_fn+0x95/0x110
[1294400.799729] [<ffffffff8116a200>] ? bdi_start_fn+0x0/0x110
[1294400.799787] [<ffffffff810a846e>] kthread+0x9e/0xc0
[1294400.799844] [<ffffffff8100c38a>] child_rip+0xa/0x20
[1294400.799901] [<ffffffff810a83d0>] ? kthread+0x0/0xc0
[1294400.799957] [<ffffffff8100c380>] ? child_rip+0x0/0x20
[1294400.800014] INFO: task jbd2/ploop27926:1692 blocked for more than 120 seconds.
[1294400.800121] Not tainted 2.6.32-042stab108.1 #1
[1294400.800177] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[1294400.800285] jbd2/ploop279 D ffff88081950e400 0 1692 2 0 0x00000080
[1294400.800393] ffff88080dde4fe0 0000000000000046 0000000000000000 ffff88060dac1000
[1294400.800502] 0005120000000000 ffff88081b2c4ae0 0004991706322912 0000000000000002
[1294400.800615] ffff88080dde4fb8 0000000000000000 000000014d2b4e18 0000000000000000
[1294400.800724] Call Trace:
[1294400.800777] [<ffffffff81536075>] schedule_timeout+0x215/0x2e0
[1294400.800836] [<ffffffff81015059>] ? read_tsc+0x9/0x20
[1294400.800893] [<ffffffff810b4801>] ? ktime_get_ts+0xb1/0xf0
[1294400.800951] [<ffffffff81015059>] ? read_tsc+0x9/0x20
[1294400.801008] [<ffffffff810b4801>] ? ktime_get_ts+0xb1/0xf0
[1294400.801066] [<ffffffff8153460f>] io_schedule_timeout+0x7f/0xd0
[1294400.801125] [<ffffffff815359e4>] wait_for_completion_io+0xe4/0x120
[1294400.801185] [<ffffffff81065c00>] ? default_wake_function+0x0/0x20
[1294400.801246] [<ffffffff81286126>] blkdev_issue_discard+0x216/0x230
[1294400.801305] [<ffffffff81189875>] scan_swap_map+0x385/0x640
[1294400.801362] [<ffffffff81189c6d>] get_swap_page+0x9d/0x140
[1294400.801420] [<ffffffff81186e87>] add_to_swap+0x17/0x90
[1294400.801478] [<ffffffff8115a6ce>] shrink_page_list.clone.0+0x2de/0x900
[1294400.801537] [<ffffffff8115bece>] shrink_inactive_list+0x3be/0xb10
[1294400.801600] [<ffffffff81015059>] ? read_tsc+0x9/0x20
[1294400.801660] [<ffffffff8115ca50>] shrink_lruvec+0x430/0x600
[1294400.801718] [<ffffffff8115cea7>] shrink_zone+0x287/0x3d0
[1294400.801775] [<ffffffff8115ec28>] do_try_to_free_pages+0x588/0xa60
[1294400.801834] [<ffffffff8115f31b>] try_to_free_pages+0x8b/0x120
[1294400.801894] [<ffffffff8114fde1>] __alloc_pages_nodemask+0x671/0xb50
[1294400.801954] [<ffffffff8119ac79>] kmem_getpages+0x59/0x140
[1294400.802011] [<ffffffff8119cdcb>] fallback_alloc+0x1bb/0x260
[1294400.802069] [<ffffffff8119cb59>] ____cache_alloc_node+0x99/0x150
[1294400.802128] [<ffffffff8119db63>] kmem_cache_alloc+0x173/0x1e0
[1294400.802186] [<ffffffff81140de5>] mempool_alloc_slab+0x15/0x20
[1294400.802244] [<ffffffff81140f87>] mempool_alloc+0x67/0x170
[1294400.802302] [<ffffffff8119ac79>] ? kmem_getpages+0x59/0x140
[1294400.802361] [<ffffffff811f494e>] bio_alloc_bioset+0x3e/0xf0
[1294400.802419] [<ffffffff811f4aa5>] bio_alloc+0x15/0x30
[1294400.802478] [<ffffffffa04880e8>] ploop_make_request+0x5a8/0xa30 [ploop]
[1294400.802538] [<ffffffff8127d780>] generic_make_request+0x240/0x550
[1294400.802599] [<ffffffff8127db13>] submit_bio+0x83/0x1c0
[1294400.802659] [<ffffffff811f496b>] ? bio_alloc_bioset+0x5b/0xf0
[1294400.802719] [<ffffffff811ee41d>] submit_bh+0x11d/0x1e0
[1294400.802778] [<ffffffffa00a8c18>] jbd2_journal_commit_transaction+0x5a8/0x1500 [jbd2]
[1294400.802888] [<ffffffff810097dd>] ? __switch_to+0x13d/0x320
[1294400.802946] [<ffffffff8108f90b>] ? try_to_del_timer_sync+0x7b/0xe0
[1294400.803008] [<ffffffffa00aebb8>] kjournald2+0xb8/0x220 [jbd2]
[1294400.803066] [<ffffffff810a8860>] ? autoremove_wake_function+0x0/0x40
[1294400.803127] [<ffffffffa00aeb00>] ? kjournald2+0x0/0x220 [jbd2]
[1294400.803185] [<ffffffff810a846e>] kthread+0x9e/0xc0
[1294400.803242] [<ffffffff8100c38a>] child_rip+0xa/0x20
[1294400.803298] [<ffffffff810a83d0>] ? kthread+0x0/0xc0
[1294400.803355] [<ffffffff8100c380>] ? child_rip+0x0/0x20
[1294400.803414] INFO: task rs:main Q:Reg:2160 blocked for more than 120 seconds.
[1294400.803520] Not tainted 2.6.32-042stab108.1 #1
[1294400.803579] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[1294400.803691] rs:main Q:Reg D ffff8808002f1080 0 2160 1707 104 0x00000080
[1294400.803799] ffff88080c51da38 0000000000000086 0000000000000000 0000000000000000
[1294400.803908] ffff88080c51db18 ffffffff81061eec 0004991615595016 ffff88002820f7b0
[1294400.804017] ffff88080c51dba8 ffff88002820f7a0 000000014d2b3e58 0000000200000000
[1294400.804126] Call Trace:
[1294400.804179] [<ffffffff81061eec>] ? find_busiest_group+0x27c/0xa70
[1294400.804238] [<ffffffff81536af6>] __mutex_lock_slowpath+0x96/0x210
[1294400.804297] [<ffffffff8153661b>] mutex_lock+0x2b/0x50
[1294400.804356] [<ffffffffa00ab618>] __jbd2_log_wait_for_space+0xc8/0x1b0 [jbd2]
[1294400.804464] [<ffffffffa00a6f55>] start_this_handle+0x125/0x490 [jbd2]
[1294400.804524] [<ffffffffa00a69b7>] ? jbd2_journal_stop+0x1e7/0x2b0 [jbd2]
[1294400.804588] [<ffffffffa00a74a5>] jbd2_journal_start+0xb5/0x100 [jbd2]
[1294400.804657] [<ffffffffa00ea636>] ext4_journal_start_sb+0x56/0xe0 [ext4]
[1294400.804719] [<ffffffffa00d11c2>] ext4_dirty_inode+0x32/0x70 [ext4]
[1294400.804778] [<ffffffff811e4b0a>] __mark_inode_dirty+0x5a/0x2a0
[1294400.804838] [<ffffffff811d2011>] file_update_time+0x121/0x1c0
[1294400.804896] [<ffffffff811408d4>] __generic_file_write_iter+0x1f4/0x420
[1294400.804955] [<ffffffff81140b85>] __generic_file_aio_write+0x85/0xa0
[1294400.805014] [<ffffffff81140c28>] generic_file_aio_write+0x88/0x100
[1294400.805075] [<ffffffffa00ca008>] ext4_file_write+0x58/0x190 [ext4]
[1294400.805134] [<ffffffff8106f3a8>] ? finish_task_switch+0xc8/0x120
[1294400.805193] [<ffffffff811b3c92>] do_sync_write+0xf2/0x140
[1294400.805250] [<ffffffff811b3f78>] vfs_write+0xb8/0x1a0
[1294400.805307] [<ffffffff811b4941>] sys_write+0x51/0x90
[1294400.805364] [<ffffffff810fd3ae>] ? __audit_syscall_exit+0x25e/0x290
[1294400.805423] [<ffffffff8100b142>] system_call_fastpath+0x16/0x1b
[1294400.806899] INFO: task master:4288 blocked for more than 120 seconds.
[1294400.806961] Not tainted 2.6.32-042stab108.1 #1
[1294400.807017] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[1294400.807125] master D ffff880816530040 0 4288 1707 104 0x00000080
[1294400.807233] ffff8808011fdb08 0000000000000082 0000000000000001 0000000000000282
[1294400.807342] 0000000000000001 ffff880801509c00 0000000000000246 0000000000000246
[1294400.807451] ffff8808011fdaa8 0000000000000282 ffff8808011fdab8 ffffffff810a8c5c
[1294400.807562] Call Trace:
[1294400.807617] [<ffffffff810a8c5c>] ? remove_wait_queue+0x3c/0x50
[1294400.807676] [<ffffffff81536af6>] __mutex_lock_slowpath+0x96/0x210
[1294400.807735] [<ffffffff8153661b>] mutex_lock+0x2b/0x50
[1294400.807794] [<ffffffffa00ab618>] __jbd2_log_wait_for_space+0xc8/0x1b0 [jbd2]
[1294400.807902] [<ffffffffa00a6f55>] start_this_handle+0x125/0x490 [jbd2]
[1294400.807962] [<ffffffffa00a74a5>] jbd2_journal_start+0xb5/0x100 [jbd2]
[1294400.808025] [<ffffffffa00ea636>] ext4_journal_start_sb+0x56/0xe0 [ext4]
[1294400.808087] [<ffffffffa00d11c2>] ext4_dirty_inode+0x32/0x70 [ext4]
[1294400.808146] [<ffffffff811e4b0a>] __mark_inode_dirty+0x5a/0x2a0
[1294400.808205] [<ffffffff811d2011>] file_update_time+0x121/0x1c0
[1294400.808263] [<ffffffff811bfa62>] pipe_write+0x2f2/0x690
[1294400.808320] [<ffffffff811b3c92>] do_sync_write+0xf2/0x140
[1294400.808378] [<ffffffff810b4801>] ? ktime_get_ts+0xb1/0xf0
[1294400.808435] [<ffffffff811b3f78>] vfs_write+0xb8/0x1a0
[1294400.808492] [<ffffffff811b4941>] sys_write+0x51/0x90
[1294400.808551] [<ffffffff810fd3ae>] ? __audit_syscall_exit+0x25e/0x290
[1294400.808614] [<ffffffff8100b142>] system_call_fastpath+0x16/0x1b
[1294400.808688] INFO: task pickup:695593 blocked for more than 120 seconds.
[1294400.808746] Not tainted 2.6.32-042stab108.1 #1
[1294400.808802] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[1294400.808910] pickup D ffff8808165d2300 0 695593 4288 104 0x00000080
[1294400.809018] ffff88080b891a38 0000000000000086 0000000000000000 ffff88080b8919d8
[1294400.809127] ffff88080b891b08 ffff88012a838240 00049917ba70169f ffff8808165d2300
[1294400.809236] ffff88080b891a18 0000000000000000 000000014d2b599a ffffffff8113defe
[1294400.809345] Call Trace:
[1294400.809397] [<ffffffff8113defe>] ? find_get_page+0x1e/0xa0
[1294400.809455] [<ffffffff81536af6>] __mutex_lock_slowpath+0x96/0x210
[1294400.809514] [<ffffffff8153661b>] mutex_lock+0x2b/0x50
[1294400.809575] [<ffffffffa00ab618>] __jbd2_log_wait_for_space+0xc8/0x1b0 [jbd2]
[1294400.809687] [<ffffffffa00a6f55>] start_this_handle+0x125/0x490 [jbd2]
[1294400.809748] [<ffffffffa00a74a5>] jbd2_journal_start+0xb5/0x100 [jbd2]
[1294400.809807] [<ffffffff811b6915>] ? __sb_start_write+0xd5/0x1b0
[1294400.809869] [<ffffffffa00ea636>] ext4_journal_start_sb+0x56/0xe0 [ext4]
[1294400.809931] [<ffffffffa00d11c2>] ext4_dirty_inode+0x32/0x70 [ext4]
[1294400.809990] [<ffffffff811e4b0a>] __mark_inode_dirty+0x5a/0x2a0
[1294400.810048] [<ffffffff811d2245>] touch_atime+0x195/0x1a0
[1294400.810105] [<ffffffff811c00d5>] pipe_read+0x2d5/0x4e0
[1294400.810162] [<ffffffff811b3dda>] do_sync_read+0xfa/0x140
[1294400.810220] [<ffffffff810a8860>] ? autoremove_wake_function+0x0/0x40
[1294400.810279] [<ffffffff811b4775>] vfs_read+0xb5/0x1a0
[1294400.810335] [<ffffffff811b48b1>] sys_read+0x51/0x90
[1294400.810392] [<ffffffff810fd3ae>] ? __audit_syscall_exit+0x25e/0x290
[1294400.810451] [<ffffffff8100b142>] system_call_fastpath+0x16/0x1b
[1294400.810510] INFO: task tar:698096 blocked for more than 120 seconds.
[1294400.810571] Not tainted 2.6.32-042stab108.1 #1
[1294400.810630] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[1294400.810738] tar D ffff8807fffba5c0 0 698096 698095 104 0x00000080
[1294400.810846] ffff8807ffee59c8 0000000000000086 ffff8807ffee59e8 ffffffff81067e53
[1294400.810955] ffff88002831dd80 ffff88080ea19800 ffff88080ea19800 ffff88002831dde8
[1294400.811064] 0000000000000003 0000000000000001 ffff88081950e448 ffff88002831dde8
[1294400.811172] Call Trace:
[1294400.811225] [<ffffffff81067e53>] ? perf_event_task_sched_out+0x33/0x70
[1294400.811285] [<ffffffff81536af6>] __mutex_lock_slowpath+0x96/0x210
[1294400.811344] [<ffffffff8105a549>] ? __wake_up_common+0x59/0x90
[1294400.811402] [<ffffffff8153661b>] mutex_lock+0x2b/0x50
[1294400.811461] [<ffffffffa00ab618>] __jbd2_log_wait_for_space+0xc8/0x1b0 [jbd2]
[1294400.811572] [<ffffffffa00a6f55>] start_this_handle+0x125/0x490 [jbd2]
[1294400.811634] [<ffffffff8113e140>] ? sync_page_killable+0x0/0x40
[1294400.811693] [<ffffffff810a8860>] ? autoremove_wake_function+0x0/0x40
[1294400.811753] [<ffffffffa00a74a5>] jbd2_journal_start+0xb5/0x100 [jbd2]
[1294400.811812] [<ffffffff811b6915>] ? __sb_start_write+0xd5/0x1b0
[1294400.811874] [<ffffffffa00ea636>] ext4_journal_start_sb+0x56/0xe0 [ext4]
[1294400.811936] [<ffffffffa00d11c2>] ext4_dirty_inode+0x32/0x70 [ext4]
[1294400.811995] [<ffffffff811e4b0a>] __mark_inode_dirty+0x5a/0x2a0
[1294400.812053] [<ffffffff811d2245>] touch_atime+0x195/0x1a0
[1294400.812110] [<ffffffff81140047>] generic_file_read_iter+0x337/0x640
[1294400.812170] [<ffffffff811403db>] generic_file_aio_read+0x8b/0xa0
[1294400.812228] [<ffffffff811b3dda>] do_sync_read+0xfa/0x140
[1294400.812285] [<ffffffff811c7394>] ? user_path_at+0x64/0xa0
[1294400.812343] [<ffffffff810a8860>] ? autoremove_wake_function+0x0/0x40
[1294400.812402] [<ffffffff811ba194>] ? cp_new_stat+0xe4/0x100
[1294400.812459] [<ffffffff811b4775>] vfs_read+0xb5/0x1a0
[1294400.812516] [<ffffffff811b48b1>] sys_read+0x51/0x90
[1294400.812575] [<ffffffff810fd3ae>] ? __audit_syscall_exit+0x25e/0x290
[1294400.812638] [<ffffffff8100b142>] system_call_fastpath+0x16/0x1b
[1294400.812696] INFO: task pigz:698098 blocked for more than 120 seconds.
[1294400.812754] Not tainted 2.6.32-042stab108.1 #1
[1294400.812810] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[1294400.812918] pigz D ffff88015f53b1c0 0 698098 698096 104 0x00000080
[1294400.813026] ffff88013229d9c8 0000000000000086 0000000000000000 0000000000000002
[1294400.813135] ffff88013229d938 ffffffff810a884f 00049915a5702803 ffff88002820f7b0
[1294400.813244] 0000000000000000 ffff88002820f7a0 000000014d2b36d9 0000000700000001
[1294400.813352] Call Trace:
[1294400.813405] [<ffffffff810a884f>] ? wake_up_bit+0x2f/0x40
[1294400.813465] [<ffffffffa00ae6f5>] jbd2_log_wait_commit+0xc5/0x140 [jbd2]
[1294400.813524] [<ffffffff810a8860>] ? autoremove_wake_function+0x0/0x40
[1294400.813588] [<ffffffffa00ab0d3>] jbd2_log_do_checkpoint+0xb3/0x530 [jbd2]
[1294400.813652] [<ffffffffa00ab5e5>] __jbd2_log_wait_for_space+0x95/0x1b0 [jbd2]
[1294400.813760] [<ffffffffa00a6f55>] start_this_handle+0x125/0x490 [jbd2]
[1294400.813821] [<ffffffffa00a69b7>] ? jbd2_journal_stop+0x1e7/0x2b0 [jbd2]
[1294400.813881] [<ffffffffa00a74a5>] jbd2_journal_start+0xb5/0x100 [jbd2]
[1294400.813944] [<ffffffffa00ea636>] ext4_journal_start_sb+0x56/0xe0 [ext4]
[1294400.814006] [<ffffffffa00d11c2>] ext4_dirty_inode+0x32/0x70 [ext4]
[1294400.814065] [<ffffffff811e4b0a>] __mark_inode_dirty+0x5a/0x2a0
[1294400.814123] [<ffffffff811d2011>] file_update_time+0x121/0x1c0
[1294400.814182] [<ffffffff811408d4>] __generic_file_write_iter+0x1f4/0x420
[1294400.814241] [<ffffffff81140b85>] __generic_file_aio_write+0x85/0xa0
[1294400.814300] [<ffffffff81140c28>] generic_file_aio_write+0x88/0x100
[1294400.814361] [<ffffffffa00ca008>] ext4_file_write+0x58/0x190 [ext4]
[1294400.814420] [<ffffffff8106f3a8>] ? finish_task_switch+0xc8/0x120
[1294400.814478] [<ffffffff811b3c92>] do_sync_write+0xf2/0x140
[1294400.814535] [<ffffffff811b3f78>] vfs_write+0xb8/0x1a0
[1294400.814594] [<ffffffff811b4941>] sys_write+0x51/0x90
[1294400.814655] [<ffffffff810fd3ae>] ? __audit_syscall_exit+0x25e/0x290
[1294400.814714] [<ffffffff8100b142>] system_call_fastpath+0x16/0x1b
[1294520.743913] INFO: task kswapd0:102 blocked for more than 120 seconds.
[1294520.743974] Not tainted 2.6.32-042stab108.1 #1
[1294520.744030] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[1294520.744142] kswapd0 D ffff880816a36f00 0 102 2 0 0x00000000
[1294520.744256] ffff880816a39700 0000000000000046 0000000000000000 ffffea0020139300
[1294520.744365] ffff880000023600 0000000815402cc0 00049915aa9ddfb9 ffff8808162da800
[1294520.744474] 0000000000000001 0000000000000000 000000014d2b374a ffff880816a36f00
[1294520.744584] Call Trace:
[1294520.744640] [<ffffffff811894d0>] ? wait_for_discard+0x0/0x20
[1294520.744697] [<ffffffff811894de>] wait_for_discard+0xe/0x20
[1294520.744756] [<ffffffff8153642f>] __wait_on_bit+0x5f/0x90
[1294520.744813] [<ffffffff811894d0>] ? wait_for_discard+0x0/0x20
[1294520.744872] [<ffffffff815364d8>] out_of_line_wait_on_bit+0x78/0x90
[1294520.744931] [<ffffffff810a88e0>] ? wake_bit_function+0x0/0x50
[1294520.744989] [<ffffffff811898fd>] scan_swap_map+0x40d/0x640
[1294520.745047] [<ffffffff81189c6d>] get_swap_page+0x9d/0x140
[1294520.745105] [<ffffffff811863fe>] ? swap_writepage+0xde/0x120
[1294520.745168] [<ffffffff8116291c>] shmem_writepage+0xcc/0x230
[1294520.745230] [<ffffffff8115a22b>] pageout.clone.2+0x12b/0x2f0
[1294520.745288] [<ffffffff8115aa46>] shrink_page_list.clone.0+0x656/0x900
[1294520.745348] [<ffffffff8115bece>] shrink_inactive_list+0x3be/0xb10
[1294520.745407] [<ffffffff8115ca50>] shrink_lruvec+0x430/0x600
[1294520.745466] [<ffffffff81015059>] ? read_tsc+0x9/0x20
[1294520.745530] [<ffffffffa03774be>] ? nfs_access_cache_shrinker+0x1ce/0x210 [nfs]
[1294520.745638] [<ffffffff8115cea7>] shrink_zone+0x287/0x3d0
[1294520.745695] [<ffffffff8115e133>] balance_pgdat+0xc23/0xda0
[1294520.745753] [<ffffffff8115e42f>] kswapd+0x17f/0x3f0
[1294520.745811] [<ffffffff810a8860>] ? autoremove_wake_function+0x0/0x40
[1294520.745870] [<ffffffff8115e2b0>] ? kswapd+0x0/0x3f0
[1294520.745927] [<ffffffff810a846e>] kthread+0x9e/0xc0
[1294520.745983] [<ffffffff8100c38a>] child_rip+0xa/0x20
[1294520.746040] [<ffffffff810a83d0>] ? kthread+0x0/0xc0
[1294520.746097] [<ffffffff8100c380>] ? child_rip+0x0/0x20
[1294520.746163] INFO: task flush-182:44681:1686 blocked for more than 120 seconds.
[1294520.746274] Not tainted 2.6.32-042stab108.1 #1
[1294520.746330] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[1294520.747849] flush-182:446 D ffff8808165c2e40 0 1686 2 0 0x00000080
[1294520.747959] ffff88080d027830 0000000000000046 0000000000000000 ffff8808165c36a8
[1294520.748068] ffff88081b236480 0000000000000040 00049916bdd7aabb ffff88002831a538
[1294520.748182] ffff88080d027820 0000000000000000 000000014d2b4965 0000000000000001
[1294520.748291] Call Trace:
[1294520.748344] [<ffffffff81536af6>] __mutex_lock_slowpath+0x96/0x210
[1294520.748403] [<ffffffff8153661b>] mutex_lock+0x2b/0x50
[1294520.748468] [<ffffffffa00ab618>] __jbd2_log_wait_for_space+0xc8/0x1b0 [jbd2]
[1294520.748575] [<ffffffffa00a6f55>] start_this_handle+0x125/0x490 [jbd2]
[1294520.748639] [<ffffffffa00cf2be>] ? ext4_num_dirty_pages+0x1ee/0x270 [ext4]
[1294520.748701] [<ffffffffa00a74a5>] jbd2_journal_start+0xb5/0x100 [jbd2]
[1294520.748765] [<ffffffffa00ea636>] ext4_journal_start_sb+0x56/0xe0 [ext4]
[1294520.748828] [<ffffffffa00d5306>] ext4_da_writepages+0x286/0x660 [ext4]
[1294520.748888] [<ffffffff8100bc8e>] ? apic_timer_interrupt+0xe/0x20
[1294520.748946] [<ffffffff81154ca1>] do_writepages+0x21/0x40
[1294520.749005] [<ffffffff811e4e49>] __writeback_single_inode+0xf9/0x3d0
[1294520.749065] [<ffffffff811e51b8>] writeback_single_inode+0x98/0xd0
[1294520.749126] [<ffffffff811e545a>] writeback_sb_inodes+0xda/0x1b0
[1294520.749189] [<ffffffff811e5648>] writeback_inodes_wb+0x118/0x170
[1294520.749248] [<ffffffff811e59db>] wb_writeback+0x33b/0x460
[1294520.749307] [<ffffffffa0143f80>] ? do_dbs_timer+0x0/0x3bc [cpufreq_ondemand]
[1294520.749415] [<ffffffff811e5cb5>] wb_do_writeback+0x1b5/0x260
[1294520.749473] [<ffffffff811e5dfc>] bdi_writeback_task+0x9c/0x1f0
[1294520.749532] [<ffffffff8116a200>] ? bdi_start_fn+0x0/0x110
[1294520.749589] [<ffffffff8116a200>] ? bdi_start_fn+0x0/0x110
[1294520.749646] [<ffffffff8116a295>] bdi_start_fn+0x95/0x110
[1294520.749704] [<ffffffff8116a200>] ? bdi_start_fn+0x0/0x110
[1294520.749762] [<ffffffff810a846e>] kthread+0x9e/0xc0
[1294520.749818] [<ffffffff8100c38a>] child_rip+0xa/0x20
[1294520.749875] [<ffffffff810a83d0>] ? kthread+0x0/0xc0
[1294520.749932] [<ffffffff8100c380>] ? child_rip+0x0/0x20
Only reboot could fix issue. Repeated multiple times. No hardware raid. Only software raid here with SSD drives.
[1294400.795403] Not tainted 2.6.32-042stab108.1 #1
[1294400.795459] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[1294400.795569] kswapd0 D ffff880816a36f00 0 102 2 0 0x00000000
[1294400.795682] ffff880816a39700 0000000000000046 0000000000000000 ffffea0020139300
[1294400.795792] ffff880000023600 0000000815402cc0 00049915aa9ddfb9 ffff8808162da800
[1294400.795900] 0000000000000001 0000000000000000 000000014d2b374a ffff880816a36f00
[1294400.796009] Call Trace:
[1294400.796064] [<ffffffff811894d0>] ? wait_for_discard+0x0/0x20
[1294400.796122] [<ffffffff811894de>] wait_for_discard+0xe/0x20
[1294400.796181] [<ffffffff8153642f>] __wait_on_bit+0x5f/0x90
[1294400.796239] [<ffffffff811894d0>] ? wait_for_discard+0x0/0x20
[1294400.796297] [<ffffffff815364d8>] out_of_line_wait_on_bit+0x78/0x90
[1294400.796357] [<ffffffff810a88e0>] ? wake_bit_function+0x0/0x50
[1294400.796415] [<ffffffff811898fd>] scan_swap_map+0x40d/0x640
[1294400.796472] [<ffffffff81189c6d>] get_swap_page+0x9d/0x140
[1294400.796530] [<ffffffff811863fe>] ? swap_writepage+0xde/0x120
[1294400.796592] [<ffffffff8116291c>] shmem_writepage+0xcc/0x230
[1294400.796653] [<ffffffff8115a22b>] pageout.clone.2+0x12b/0x2f0
[1294400.796711] [<ffffffff8115aa46>] shrink_page_list.clone.0+0x656/0x900
[1294400.796771] [<ffffffff8115bece>] shrink_inactive_list+0x3be/0xb10
[1294400.796830] [<ffffffff8115ca50>] shrink_lruvec+0x430/0x600
[1294400.796889] [<ffffffff81015059>] ? read_tsc+0x9/0x20
[1294400.796952] [<ffffffffa03774be>] ? nfs_access_cache_shrinker+0x1ce/0x210 [nfs]
[1294400.797060] [<ffffffff8115cea7>] shrink_zone+0x287/0x3d0
[1294400.797117] [<ffffffff8115e133>] balance_pgdat+0xc23/0xda0
[1294400.797175] [<ffffffff8115e42f>] kswapd+0x17f/0x3f0
[1294400.797232] [<ffffffff810a8860>] ? autoremove_wake_function+0x0/0x40
[1294400.797292] [<ffffffff8115e2b0>] ? kswapd+0x0/0x3f0
[1294400.797348] [<ffffffff810a846e>] kthread+0x9e/0xc0
[1294400.797406] [<ffffffff8100c38a>] child_rip+0xa/0x20
[1294400.797462] [<ffffffff810a83d0>] ? kthread+0x0/0xc0
[1294400.797519] [<ffffffff8100c380>] ? child_rip+0x0/0x20
[1294400.797583] INFO: task flush-182:44681:1686 blocked for more than 120 seconds.
[1294400.797692] Not tainted 2.6.32-042stab108.1 #1
[1294400.797751] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[1294400.797862] flush-182:446 D ffff8808165c2e40 0 1686 2 0 0x00000080
[1294400.797976] ffff88080d027830 0000000000000046 0000000000000000 ffff8808165c36a8
[1294400.798084] ffff88081b236480 0000000000000040 00049916bdd7aabb ffff88002831a538
[1294400.798200] ffff88080d027820 0000000000000000 000000014d2b4965 0000000000000001
[1294400.798309] Call Trace:
[1294400.798362] [<ffffffff81536af6>] __mutex_lock_slowpath+0x96/0x210
[1294400.798421] [<ffffffff8153661b>] mutex_lock+0x2b/0x50
[1294400.798486] [<ffffffffa00ab618>] __jbd2_log_wait_for_space+0xc8/0x1b0 [jbd2]
[1294400.798598] [<ffffffffa00a6f55>] start_this_handle+0x125/0x490 [jbd2]
[1294400.798666] [<ffffffffa00cf2be>] ? ext4_num_dirty_pages+0x1ee/0x270 [ext4]
[1294400.798728] [<ffffffffa00a74a5>] jbd2_journal_start+0xb5/0x100 [jbd2]
[1294400.798792] [<ffffffffa00ea636>] ext4_journal_start_sb+0x56/0xe0 [ext4]
[1294400.798855] [<ffffffffa00d5306>] ext4_da_writepages+0x286/0x660 [ext4]
[1294400.798915] [<ffffffff8100bc8e>] ? apic_timer_interrupt+0xe/0x20
[1294400.798974] [<ffffffff81154ca1>] do_writepages+0x21/0x40
[1294400.799032] [<ffffffff811e4e49>] __writeback_single_inode+0xf9/0x3d0
[1294400.799092] [<ffffffff811e51b8>] writeback_single_inode+0x98/0xd0
[1294400.799151] [<ffffffff811e545a>] writeback_sb_inodes+0xda/0x1b0
[1294400.799209] [<ffffffff811e5648>] writeback_inodes_wb+0x118/0x170
[1294400.799268] [<ffffffff811e59db>] wb_writeback+0x33b/0x460
[1294400.799327] [<ffffffffa0143f80>] ? do_dbs_timer+0x0/0x3bc [cpufreq_ondemand]
[1294400.799434] [<ffffffff811e5cb5>] wb_do_writeback+0x1b5/0x260
[1294400.799493] [<ffffffff811e5dfc>] bdi_writeback_task+0x9c/0x1f0
[1294400.799554] [<ffffffff8116a200>] ? bdi_start_fn+0x0/0x110
[1294400.799614] [<ffffffff8116a200>] ? bdi_start_fn+0x0/0x110
[1294400.799672] [<ffffffff8116a295>] bdi_start_fn+0x95/0x110
[1294400.799729] [<ffffffff8116a200>] ? bdi_start_fn+0x0/0x110
[1294400.799787] [<ffffffff810a846e>] kthread+0x9e/0xc0
[1294400.799844] [<ffffffff8100c38a>] child_rip+0xa/0x20
[1294400.799901] [<ffffffff810a83d0>] ? kthread+0x0/0xc0
[1294400.799957] [<ffffffff8100c380>] ? child_rip+0x0/0x20
[1294400.800014] INFO: task jbd2/ploop27926:1692 blocked for more than 120 seconds.
[1294400.800121] Not tainted 2.6.32-042stab108.1 #1
[1294400.800177] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[1294400.800285] jbd2/ploop279 D ffff88081950e400 0 1692 2 0 0x00000080
[1294400.800393] ffff88080dde4fe0 0000000000000046 0000000000000000 ffff88060dac1000
[1294400.800502] 0005120000000000 ffff88081b2c4ae0 0004991706322912 0000000000000002
[1294400.800615] ffff88080dde4fb8 0000000000000000 000000014d2b4e18 0000000000000000
[1294400.800724] Call Trace:
[1294400.800777] [<ffffffff81536075>] schedule_timeout+0x215/0x2e0
[1294400.800836] [<ffffffff81015059>] ? read_tsc+0x9/0x20
[1294400.800893] [<ffffffff810b4801>] ? ktime_get_ts+0xb1/0xf0
[1294400.800951] [<ffffffff81015059>] ? read_tsc+0x9/0x20
[1294400.801008] [<ffffffff810b4801>] ? ktime_get_ts+0xb1/0xf0
[1294400.801066] [<ffffffff8153460f>] io_schedule_timeout+0x7f/0xd0
[1294400.801125] [<ffffffff815359e4>] wait_for_completion_io+0xe4/0x120
[1294400.801185] [<ffffffff81065c00>] ? default_wake_function+0x0/0x20
[1294400.801246] [<ffffffff81286126>] blkdev_issue_discard+0x216/0x230
[1294400.801305] [<ffffffff81189875>] scan_swap_map+0x385/0x640
[1294400.801362] [<ffffffff81189c6d>] get_swap_page+0x9d/0x140
[1294400.801420] [<ffffffff81186e87>] add_to_swap+0x17/0x90
[1294400.801478] [<ffffffff8115a6ce>] shrink_page_list.clone.0+0x2de/0x900
[1294400.801537] [<ffffffff8115bece>] shrink_inactive_list+0x3be/0xb10
[1294400.801600] [<ffffffff81015059>] ? read_tsc+0x9/0x20
[1294400.801660] [<ffffffff8115ca50>] shrink_lruvec+0x430/0x600
[1294400.801718] [<ffffffff8115cea7>] shrink_zone+0x287/0x3d0
[1294400.801775] [<ffffffff8115ec28>] do_try_to_free_pages+0x588/0xa60
[1294400.801834] [<ffffffff8115f31b>] try_to_free_pages+0x8b/0x120
[1294400.801894] [<ffffffff8114fde1>] __alloc_pages_nodemask+0x671/0xb50
[1294400.801954] [<ffffffff8119ac79>] kmem_getpages+0x59/0x140
[1294400.802011] [<ffffffff8119cdcb>] fallback_alloc+0x1bb/0x260
[1294400.802069] [<ffffffff8119cb59>] ____cache_alloc_node+0x99/0x150
[1294400.802128] [<ffffffff8119db63>] kmem_cache_alloc+0x173/0x1e0
[1294400.802186] [<ffffffff81140de5>] mempool_alloc_slab+0x15/0x20
[1294400.802244] [<ffffffff81140f87>] mempool_alloc+0x67/0x170
[1294400.802302] [<ffffffff8119ac79>] ? kmem_getpages+0x59/0x140
[1294400.802361] [<ffffffff811f494e>] bio_alloc_bioset+0x3e/0xf0
[1294400.802419] [<ffffffff811f4aa5>] bio_alloc+0x15/0x30
[1294400.802478] [<ffffffffa04880e8>] ploop_make_request+0x5a8/0xa30 [ploop]
[1294400.802538] [<ffffffff8127d780>] generic_make_request+0x240/0x550
[1294400.802599] [<ffffffff8127db13>] submit_bio+0x83/0x1c0
[1294400.802659] [<ffffffff811f496b>] ? bio_alloc_bioset+0x5b/0xf0
[1294400.802719] [<ffffffff811ee41d>] submit_bh+0x11d/0x1e0
[1294400.802778] [<ffffffffa00a8c18>] jbd2_journal_commit_transaction+0x5a8/0x1500 [jbd2]
[1294400.802888] [<ffffffff810097dd>] ? __switch_to+0x13d/0x320
[1294400.802946] [<ffffffff8108f90b>] ? try_to_del_timer_sync+0x7b/0xe0
[1294400.803008] [<ffffffffa00aebb8>] kjournald2+0xb8/0x220 [jbd2]
[1294400.803066] [<ffffffff810a8860>] ? autoremove_wake_function+0x0/0x40
[1294400.803127] [<ffffffffa00aeb00>] ? kjournald2+0x0/0x220 [jbd2]
[1294400.803185] [<ffffffff810a846e>] kthread+0x9e/0xc0
[1294400.803242] [<ffffffff8100c38a>] child_rip+0xa/0x20
[1294400.803298] [<ffffffff810a83d0>] ? kthread+0x0/0xc0
[1294400.803355] [<ffffffff8100c380>] ? child_rip+0x0/0x20
[1294400.803414] INFO: task rs:main Q:Reg:2160 blocked for more than 120 seconds.
[1294400.803520] Not tainted 2.6.32-042stab108.1 #1
[1294400.803579] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[1294400.803691] rs:main Q:Reg D ffff8808002f1080 0 2160 1707 104 0x00000080
[1294400.803799] ffff88080c51da38 0000000000000086 0000000000000000 0000000000000000
[1294400.803908] ffff88080c51db18 ffffffff81061eec 0004991615595016 ffff88002820f7b0
[1294400.804017] ffff88080c51dba8 ffff88002820f7a0 000000014d2b3e58 0000000200000000
[1294400.804126] Call Trace:
[1294400.804179] [<ffffffff81061eec>] ? find_busiest_group+0x27c/0xa70
[1294400.804238] [<ffffffff81536af6>] __mutex_lock_slowpath+0x96/0x210
[1294400.804297] [<ffffffff8153661b>] mutex_lock+0x2b/0x50
[1294400.804356] [<ffffffffa00ab618>] __jbd2_log_wait_for_space+0xc8/0x1b0 [jbd2]
[1294400.804464] [<ffffffffa00a6f55>] start_this_handle+0x125/0x490 [jbd2]
[1294400.804524] [<ffffffffa00a69b7>] ? jbd2_journal_stop+0x1e7/0x2b0 [jbd2]
[1294400.804588] [<ffffffffa00a74a5>] jbd2_journal_start+0xb5/0x100 [jbd2]
[1294400.804657] [<ffffffffa00ea636>] ext4_journal_start_sb+0x56/0xe0 [ext4]
[1294400.804719] [<ffffffffa00d11c2>] ext4_dirty_inode+0x32/0x70 [ext4]
[1294400.804778] [<ffffffff811e4b0a>] __mark_inode_dirty+0x5a/0x2a0
[1294400.804838] [<ffffffff811d2011>] file_update_time+0x121/0x1c0
[1294400.804896] [<ffffffff811408d4>] __generic_file_write_iter+0x1f4/0x420
[1294400.804955] [<ffffffff81140b85>] __generic_file_aio_write+0x85/0xa0
[1294400.805014] [<ffffffff81140c28>] generic_file_aio_write+0x88/0x100
[1294400.805075] [<ffffffffa00ca008>] ext4_file_write+0x58/0x190 [ext4]
[1294400.805134] [<ffffffff8106f3a8>] ? finish_task_switch+0xc8/0x120
[1294400.805193] [<ffffffff811b3c92>] do_sync_write+0xf2/0x140
[1294400.805250] [<ffffffff811b3f78>] vfs_write+0xb8/0x1a0
[1294400.805307] [<ffffffff811b4941>] sys_write+0x51/0x90
[1294400.805364] [<ffffffff810fd3ae>] ? __audit_syscall_exit+0x25e/0x290
[1294400.805423] [<ffffffff8100b142>] system_call_fastpath+0x16/0x1b
[1294400.806899] INFO: task master:4288 blocked for more than 120 seconds.
[1294400.806961] Not tainted 2.6.32-042stab108.1 #1
[1294400.807017] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[1294400.807125] master D ffff880816530040 0 4288 1707 104 0x00000080
[1294400.807233] ffff8808011fdb08 0000000000000082 0000000000000001 0000000000000282
[1294400.807342] 0000000000000001 ffff880801509c00 0000000000000246 0000000000000246
[1294400.807451] ffff8808011fdaa8 0000000000000282 ffff8808011fdab8 ffffffff810a8c5c
[1294400.807562] Call Trace:
[1294400.807617] [<ffffffff810a8c5c>] ? remove_wait_queue+0x3c/0x50
[1294400.807676] [<ffffffff81536af6>] __mutex_lock_slowpath+0x96/0x210
[1294400.807735] [<ffffffff8153661b>] mutex_lock+0x2b/0x50
[1294400.807794] [<ffffffffa00ab618>] __jbd2_log_wait_for_space+0xc8/0x1b0 [jbd2]
[1294400.807902] [<ffffffffa00a6f55>] start_this_handle+0x125/0x490 [jbd2]
[1294400.807962] [<ffffffffa00a74a5>] jbd2_journal_start+0xb5/0x100 [jbd2]
[1294400.808025] [<ffffffffa00ea636>] ext4_journal_start_sb+0x56/0xe0 [ext4]
[1294400.808087] [<ffffffffa00d11c2>] ext4_dirty_inode+0x32/0x70 [ext4]
[1294400.808146] [<ffffffff811e4b0a>] __mark_inode_dirty+0x5a/0x2a0
[1294400.808205] [<ffffffff811d2011>] file_update_time+0x121/0x1c0
[1294400.808263] [<ffffffff811bfa62>] pipe_write+0x2f2/0x690
[1294400.808320] [<ffffffff811b3c92>] do_sync_write+0xf2/0x140
[1294400.808378] [<ffffffff810b4801>] ? ktime_get_ts+0xb1/0xf0
[1294400.808435] [<ffffffff811b3f78>] vfs_write+0xb8/0x1a0
[1294400.808492] [<ffffffff811b4941>] sys_write+0x51/0x90
[1294400.808551] [<ffffffff810fd3ae>] ? __audit_syscall_exit+0x25e/0x290
[1294400.808614] [<ffffffff8100b142>] system_call_fastpath+0x16/0x1b
[1294400.808688] INFO: task pickup:695593 blocked for more than 120 seconds.
[1294400.808746] Not tainted 2.6.32-042stab108.1 #1
[1294400.808802] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[1294400.808910] pickup D ffff8808165d2300 0 695593 4288 104 0x00000080
[1294400.809018] ffff88080b891a38 0000000000000086 0000000000000000 ffff88080b8919d8
[1294400.809127] ffff88080b891b08 ffff88012a838240 00049917ba70169f ffff8808165d2300
[1294400.809236] ffff88080b891a18 0000000000000000 000000014d2b599a ffffffff8113defe
[1294400.809345] Call Trace:
[1294400.809397] [<ffffffff8113defe>] ? find_get_page+0x1e/0xa0
[1294400.809455] [<ffffffff81536af6>] __mutex_lock_slowpath+0x96/0x210
[1294400.809514] [<ffffffff8153661b>] mutex_lock+0x2b/0x50
[1294400.809575] [<ffffffffa00ab618>] __jbd2_log_wait_for_space+0xc8/0x1b0 [jbd2]
[1294400.809687] [<ffffffffa00a6f55>] start_this_handle+0x125/0x490 [jbd2]
[1294400.809748] [<ffffffffa00a74a5>] jbd2_journal_start+0xb5/0x100 [jbd2]
[1294400.809807] [<ffffffff811b6915>] ? __sb_start_write+0xd5/0x1b0
[1294400.809869] [<ffffffffa00ea636>] ext4_journal_start_sb+0x56/0xe0 [ext4]
[1294400.809931] [<ffffffffa00d11c2>] ext4_dirty_inode+0x32/0x70 [ext4]
[1294400.809990] [<ffffffff811e4b0a>] __mark_inode_dirty+0x5a/0x2a0
[1294400.810048] [<ffffffff811d2245>] touch_atime+0x195/0x1a0
[1294400.810105] [<ffffffff811c00d5>] pipe_read+0x2d5/0x4e0
[1294400.810162] [<ffffffff811b3dda>] do_sync_read+0xfa/0x140
[1294400.810220] [<ffffffff810a8860>] ? autoremove_wake_function+0x0/0x40
[1294400.810279] [<ffffffff811b4775>] vfs_read+0xb5/0x1a0
[1294400.810335] [<ffffffff811b48b1>] sys_read+0x51/0x90
[1294400.810392] [<ffffffff810fd3ae>] ? __audit_syscall_exit+0x25e/0x290
[1294400.810451] [<ffffffff8100b142>] system_call_fastpath+0x16/0x1b
[1294400.810510] INFO: task tar:698096 blocked for more than 120 seconds.
[1294400.810571] Not tainted 2.6.32-042stab108.1 #1
[1294400.810630] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[1294400.810738] tar D ffff8807fffba5c0 0 698096 698095 104 0x00000080
[1294400.810846] ffff8807ffee59c8 0000000000000086 ffff8807ffee59e8 ffffffff81067e53
[1294400.810955] ffff88002831dd80 ffff88080ea19800 ffff88080ea19800 ffff88002831dde8
[1294400.811064] 0000000000000003 0000000000000001 ffff88081950e448 ffff88002831dde8
[1294400.811172] Call Trace:
[1294400.811225] [<ffffffff81067e53>] ? perf_event_task_sched_out+0x33/0x70
[1294400.811285] [<ffffffff81536af6>] __mutex_lock_slowpath+0x96/0x210
[1294400.811344] [<ffffffff8105a549>] ? __wake_up_common+0x59/0x90
[1294400.811402] [<ffffffff8153661b>] mutex_lock+0x2b/0x50
[1294400.811461] [<ffffffffa00ab618>] __jbd2_log_wait_for_space+0xc8/0x1b0 [jbd2]
[1294400.811572] [<ffffffffa00a6f55>] start_this_handle+0x125/0x490 [jbd2]
[1294400.811634] [<ffffffff8113e140>] ? sync_page_killable+0x0/0x40
[1294400.811693] [<ffffffff810a8860>] ? autoremove_wake_function+0x0/0x40
[1294400.811753] [<ffffffffa00a74a5>] jbd2_journal_start+0xb5/0x100 [jbd2]
[1294400.811812] [<ffffffff811b6915>] ? __sb_start_write+0xd5/0x1b0
[1294400.811874] [<ffffffffa00ea636>] ext4_journal_start_sb+0x56/0xe0 [ext4]
[1294400.811936] [<ffffffffa00d11c2>] ext4_dirty_inode+0x32/0x70 [ext4]
[1294400.811995] [<ffffffff811e4b0a>] __mark_inode_dirty+0x5a/0x2a0
[1294400.812053] [<ffffffff811d2245>] touch_atime+0x195/0x1a0
[1294400.812110] [<ffffffff81140047>] generic_file_read_iter+0x337/0x640
[1294400.812170] [<ffffffff811403db>] generic_file_aio_read+0x8b/0xa0
[1294400.812228] [<ffffffff811b3dda>] do_sync_read+0xfa/0x140
[1294400.812285] [<ffffffff811c7394>] ? user_path_at+0x64/0xa0
[1294400.812343] [<ffffffff810a8860>] ? autoremove_wake_function+0x0/0x40
[1294400.812402] [<ffffffff811ba194>] ? cp_new_stat+0xe4/0x100
[1294400.812459] [<ffffffff811b4775>] vfs_read+0xb5/0x1a0
[1294400.812516] [<ffffffff811b48b1>] sys_read+0x51/0x90
[1294400.812575] [<ffffffff810fd3ae>] ? __audit_syscall_exit+0x25e/0x290
[1294400.812638] [<ffffffff8100b142>] system_call_fastpath+0x16/0x1b
[1294400.812696] INFO: task pigz:698098 blocked for more than 120 seconds.
[1294400.812754] Not tainted 2.6.32-042stab108.1 #1
[1294400.812810] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[1294400.812918] pigz D ffff88015f53b1c0 0 698098 698096 104 0x00000080
[1294400.813026] ffff88013229d9c8 0000000000000086 0000000000000000 0000000000000002
[1294400.813135] ffff88013229d938 ffffffff810a884f 00049915a5702803 ffff88002820f7b0
[1294400.813244] 0000000000000000 ffff88002820f7a0 000000014d2b36d9 0000000700000001
[1294400.813352] Call Trace:
[1294400.813405] [<ffffffff810a884f>] ? wake_up_bit+0x2f/0x40
[1294400.813465] [<ffffffffa00ae6f5>] jbd2_log_wait_commit+0xc5/0x140 [jbd2]
[1294400.813524] [<ffffffff810a8860>] ? autoremove_wake_function+0x0/0x40
[1294400.813588] [<ffffffffa00ab0d3>] jbd2_log_do_checkpoint+0xb3/0x530 [jbd2]
[1294400.813652] [<ffffffffa00ab5e5>] __jbd2_log_wait_for_space+0x95/0x1b0 [jbd2]
[1294400.813760] [<ffffffffa00a6f55>] start_this_handle+0x125/0x490 [jbd2]
[1294400.813821] [<ffffffffa00a69b7>] ? jbd2_journal_stop+0x1e7/0x2b0 [jbd2]
[1294400.813881] [<ffffffffa00a74a5>] jbd2_journal_start+0xb5/0x100 [jbd2]
[1294400.813944] [<ffffffffa00ea636>] ext4_journal_start_sb+0x56/0xe0 [ext4]
[1294400.814006] [<ffffffffa00d11c2>] ext4_dirty_inode+0x32/0x70 [ext4]
[1294400.814065] [<ffffffff811e4b0a>] __mark_inode_dirty+0x5a/0x2a0
[1294400.814123] [<ffffffff811d2011>] file_update_time+0x121/0x1c0
[1294400.814182] [<ffffffff811408d4>] __generic_file_write_iter+0x1f4/0x420
[1294400.814241] [<ffffffff81140b85>] __generic_file_aio_write+0x85/0xa0
[1294400.814300] [<ffffffff81140c28>] generic_file_aio_write+0x88/0x100
[1294400.814361] [<ffffffffa00ca008>] ext4_file_write+0x58/0x190 [ext4]
[1294400.814420] [<ffffffff8106f3a8>] ? finish_task_switch+0xc8/0x120
[1294400.814478] [<ffffffff811b3c92>] do_sync_write+0xf2/0x140
[1294400.814535] [<ffffffff811b3f78>] vfs_write+0xb8/0x1a0
[1294400.814594] [<ffffffff811b4941>] sys_write+0x51/0x90
[1294400.814655] [<ffffffff810fd3ae>] ? __audit_syscall_exit+0x25e/0x290
[1294400.814714] [<ffffffff8100b142>] system_call_fastpath+0x16/0x1b
[1294520.743913] INFO: task kswapd0:102 blocked for more than 120 seconds.
[1294520.743974] Not tainted 2.6.32-042stab108.1 #1
[1294520.744030] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[1294520.744142] kswapd0 D ffff880816a36f00 0 102 2 0 0x00000000
[1294520.744256] ffff880816a39700 0000000000000046 0000000000000000 ffffea0020139300
[1294520.744365] ffff880000023600 0000000815402cc0 00049915aa9ddfb9 ffff8808162da800
[1294520.744474] 0000000000000001 0000000000000000 000000014d2b374a ffff880816a36f00
[1294520.744584] Call Trace:
[1294520.744640] [<ffffffff811894d0>] ? wait_for_discard+0x0/0x20
[1294520.744697] [<ffffffff811894de>] wait_for_discard+0xe/0x20
[1294520.744756] [<ffffffff8153642f>] __wait_on_bit+0x5f/0x90
[1294520.744813] [<ffffffff811894d0>] ? wait_for_discard+0x0/0x20
[1294520.744872] [<ffffffff815364d8>] out_of_line_wait_on_bit+0x78/0x90
[1294520.744931] [<ffffffff810a88e0>] ? wake_bit_function+0x0/0x50
[1294520.744989] [<ffffffff811898fd>] scan_swap_map+0x40d/0x640
[1294520.745047] [<ffffffff81189c6d>] get_swap_page+0x9d/0x140
[1294520.745105] [<ffffffff811863fe>] ? swap_writepage+0xde/0x120
[1294520.745168] [<ffffffff8116291c>] shmem_writepage+0xcc/0x230
[1294520.745230] [<ffffffff8115a22b>] pageout.clone.2+0x12b/0x2f0
[1294520.745288] [<ffffffff8115aa46>] shrink_page_list.clone.0+0x656/0x900
[1294520.745348] [<ffffffff8115bece>] shrink_inactive_list+0x3be/0xb10
[1294520.745407] [<ffffffff8115ca50>] shrink_lruvec+0x430/0x600
[1294520.745466] [<ffffffff81015059>] ? read_tsc+0x9/0x20
[1294520.745530] [<ffffffffa03774be>] ? nfs_access_cache_shrinker+0x1ce/0x210 [nfs]
[1294520.745638] [<ffffffff8115cea7>] shrink_zone+0x287/0x3d0
[1294520.745695] [<ffffffff8115e133>] balance_pgdat+0xc23/0xda0
[1294520.745753] [<ffffffff8115e42f>] kswapd+0x17f/0x3f0
[1294520.745811] [<ffffffff810a8860>] ? autoremove_wake_function+0x0/0x40
[1294520.745870] [<ffffffff8115e2b0>] ? kswapd+0x0/0x3f0
[1294520.745927] [<ffffffff810a846e>] kthread+0x9e/0xc0
[1294520.745983] [<ffffffff8100c38a>] child_rip+0xa/0x20
[1294520.746040] [<ffffffff810a83d0>] ? kthread+0x0/0xc0
[1294520.746097] [<ffffffff8100c380>] ? child_rip+0x0/0x20
[1294520.746163] INFO: task flush-182:44681:1686 blocked for more than 120 seconds.
[1294520.746274] Not tainted 2.6.32-042stab108.1 #1
[1294520.746330] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[1294520.747849] flush-182:446 D ffff8808165c2e40 0 1686 2 0 0x00000080
[1294520.747959] ffff88080d027830 0000000000000046 0000000000000000 ffff8808165c36a8
[1294520.748068] ffff88081b236480 0000000000000040 00049916bdd7aabb ffff88002831a538
[1294520.748182] ffff88080d027820 0000000000000000 000000014d2b4965 0000000000000001
[1294520.748291] Call Trace:
[1294520.748344] [<ffffffff81536af6>] __mutex_lock_slowpath+0x96/0x210
[1294520.748403] [<ffffffff8153661b>] mutex_lock+0x2b/0x50
[1294520.748468] [<ffffffffa00ab618>] __jbd2_log_wait_for_space+0xc8/0x1b0 [jbd2]
[1294520.748575] [<ffffffffa00a6f55>] start_this_handle+0x125/0x490 [jbd2]
[1294520.748639] [<ffffffffa00cf2be>] ? ext4_num_dirty_pages+0x1ee/0x270 [ext4]
[1294520.748701] [<ffffffffa00a74a5>] jbd2_journal_start+0xb5/0x100 [jbd2]
[1294520.748765] [<ffffffffa00ea636>] ext4_journal_start_sb+0x56/0xe0 [ext4]
[1294520.748828] [<ffffffffa00d5306>] ext4_da_writepages+0x286/0x660 [ext4]
[1294520.748888] [<ffffffff8100bc8e>] ? apic_timer_interrupt+0xe/0x20
[1294520.748946] [<ffffffff81154ca1>] do_writepages+0x21/0x40
[1294520.749005] [<ffffffff811e4e49>] __writeback_single_inode+0xf9/0x3d0
[1294520.749065] [<ffffffff811e51b8>] writeback_single_inode+0x98/0xd0
[1294520.749126] [<ffffffff811e545a>] writeback_sb_inodes+0xda/0x1b0
[1294520.749189] [<ffffffff811e5648>] writeback_inodes_wb+0x118/0x170
[1294520.749248] [<ffffffff811e59db>] wb_writeback+0x33b/0x460
[1294520.749307] [<ffffffffa0143f80>] ? do_dbs_timer+0x0/0x3bc [cpufreq_ondemand]
[1294520.749415] [<ffffffff811e5cb5>] wb_do_writeback+0x1b5/0x260
[1294520.749473] [<ffffffff811e5dfc>] bdi_writeback_task+0x9c/0x1f0
[1294520.749532] [<ffffffff8116a200>] ? bdi_start_fn+0x0/0x110
[1294520.749589] [<ffffffff8116a200>] ? bdi_start_fn+0x0/0x110
[1294520.749646] [<ffffffff8116a295>] bdi_start_fn+0x95/0x110
[1294520.749704] [<ffffffff8116a200>] ? bdi_start_fn+0x0/0x110
[1294520.749762] [<ffffffff810a846e>] kthread+0x9e/0xc0
[1294520.749818] [<ffffffff8100c38a>] child_rip+0xa/0x20
[1294520.749875] [<ffffffff810a83d0>] ? kthread+0x0/0xc0
[1294520.749932] [<ffffffff8100c380>] ? child_rip+0x0/0x20
Only reboot could fix issue. Repeated multiple times. No hardware raid. Only software raid here with SSD drives.
Attachments
Issue Links
- duplicates
-
-
- Resolved
-
