Uploaded image for project: 'OpenVZ'
  1. OpenVZ
  2. OVZ-7135

EXT4-fs error in ext4_free_blocks:4879: Out of memory

    XMLWordPrintable

    Details

      Description

      reported via https://forum.openvz.org/index.php?t=msg&th=13617

      We are seeing lot of cases where fsck is needed after out of memory situations (different hardwares/ves/etc)


      ============================================================ ============================================
      Sep 30 10:18:34 kernel: EXT4-fs error (device ploop43739p1) in ext4_free_blocks:4879: Out of memory
      Sep 30 10:18:34 kernel: Aborting journal on device ploop43739p1-8.
      Sep 30 10:18:34 kernel: EXT4-fs (ploop43739p1): Remounting filesystem read-only
      Sep 30 10:18:34 kernel: EXT4-fs error (device ploop43739p1) in ext4_reserve_inode_write:5354: Journal has aborted
      Sep 30 10:18:34 kernel: EXT4-fs error (device ploop43739p1) in ext4_reserve_inode_write:5354: Journal has aborted
      Sep 30 10:18:34 kernel: EXT4-fs error (device ploop43739p1) in ext4_ext_remove_space:3060: Journal has aborted
      Sep 30 10:18:35 kernel: EXT4-fs error (device ploop43739p1) in ext4_ext_truncate:4698: Journal has aborted
      Sep 30 10:18:35 kernel: EXT4-fs error (device ploop43739p1) in ext4_reserve_inode_write:5354: Journal has aborted
      Sep 30 10:18:35 kernel: EXT4-fs error (device ploop43739p1) in ext4_truncate:4139: Journal has aborted
      Sep 30 10:18:35 kernel: EXT4-fs error (device ploop43739p1) in ext4_reserve_inode_write:5354: Journal has aborted
      Sep 30 10:18:35 kernel: httpd invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
      Sep 30 10:18:35 kernel: httpd cpuset=3657 mems_allowed=0
      Sep 30 10:18:35 kernel: CPU: 1 PID: 828588 Comm: httpd ve: 3657 Kdump: loaded Not tainted 3.10.0-957.12.2.vz7.96.21 #1 96.21
      Sep 30 10:18:35 kernel: Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.0b 09/17/2012
      Sep 30 10:18:35 kernel: Call Trace:
      Sep 30 10:18:35 kernel: [<ffffffff8b594677>] dump_stack+0x19/0x1b
      Sep 30 10:18:35 kernel: [<ffffffff8b58ef52>] dump_header+0x90/0x229
      Sep 30 10:18:35 kernel: [<ffffffff8afcdcf8>] oom_kill_process+0x5e8/0x640
      Sep 30 10:18:35 kernel: [<ffffffff8aff82ce>] ? get_task_oom_score_adj+0xee/0x100
      Sep 30 10:18:35 kernel: [<ffffffff8b0514e9>] mem_cgroup_oom_synchronize+0x4a9/0x500
      Sep 30 10:18:35 kernel: [<ffffffff8afce2a3>] pagefault_out_of_memory+0x13/0x50
      Sep 30 10:18:35 kernel: [<ffffffff8b58d027>] mm_fault_error+0x6a/0x157
      Sep 30 10:18:35 kernel: [<ffffffff8b5a28a8>] __do_page_fault+0x3c8/0x4f0
      Sep 30 10:18:35 kernel: [<ffffffff8b5a2a05>] do_page_fault+0x35/0x90
      Sep 30 10:18:35 kernel: [<ffffffff8b59e768>] page_fault+0x28/0x30
      Sep 30 10:18:35 kernel: Task in /machine.slice/3657 killed as a result of limit of /machine.slice/3657
      Sep 30 10:18:35 kernel: memory: usage 2097056kB, limit 2097152kB, failcnt 45598816
      Sep 30 10:18:35 kernel: memory+swap: usage 2621444kB, limit 2621440kB, failcnt 13806662
      Sep 30 10:18:35 kernel: kmem: usage 407384kB, limit 9007199254740988kB, failcnt 0
      Sep 30 10:18:35 kernel: Memory cgroup stats for /machine.slice/3657: rss_huge:0KB mapped_file:338136KB shmem:406204KB slab_unreclaimable:13824KB swap:524388KB cache:406476KB rss:1283196KB slab_reclaimable:355864KB inactive_anon:432588KB active_anon:1256812KB inactive_file:40KB active_file:12KB unevictable:0KB
      Sep 30 10:18:35 kernel: Memory cgroup out of memory: Kill process 828656 (clamd) score 283 or sacrifice child
      Sep 30 10:18:35 kernel: Killed process 794108 (clamd) in VE "3657" total-vm:1032160kB, anon-rss:628748kB, file-rss:0kB, shmem-rss:0kB
      Sep 30 10:18:35 kernel: rsyslogd invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
      Sep 30 10:18:35 kernel: rsyslogd cpuset=3657 mems_allowed=0
      Sep 30 10:18:35 kernel: CPU: 1 PID: 72938 Comm: rsyslogd ve: 3657 Kdump: loaded Not tainted 3.10.0-957.12.2.vz7.96.21 #1 96.21
      Sep 30 10:18:35 kernel: Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.0b 09/17/2012
      Sep 30 10:18:35 kernel: Call Trace:
      Sep 30 10:18:35 kernel: [<ffffffff8b594677>] dump_stack+0x19/0x1b
      Sep 30 10:18:35 kernel: [<ffffffff8b58ef52>] dump_header+0x90/0x229
      Sep 30 10:18:35 kernel: [<ffffffff8afcdcf8>] oom_kill_process+0x5e8/0x640
      Sep 30 10:18:35 kernel: [<ffffffff8aff82ce>] ? get_task_oom_score_adj+0xee/0x100
      Sep 30 10:18:35 kernel: [<ffffffff8b0514e9>] mem_cgroup_oom_synchronize+0x4a9/0x500
      Sep 30 10:18:35 kernel: [<ffffffff8afce2a3>] pagefault_out_of_memory+0x13/0x50
      Sep 30 10:18:35 kernel: [<ffffffff8b58d027>] mm_fault_error+0x6a/0x157
      Sep 30 10:18:35 kernel: [<ffffffff8b5a28a8>] __do_page_fault+0x3c8/0x4f0
      Sep 30 10:18:35 kernel: [<ffffffff8b5a2a05>] do_page_fault+0x35/0x90
      Sep 30 10:18:35 kernel: [<ffffffff8b59e768>] page_fault+0x28/0x30
      Sep 30 10:18:35 kernel: Task in /machine.slice/3657 killed as a result of limit of /machine.slice/3657
      Sep 30 10:18:35 kernel: memory: usage 2097056kB, limit 2097152kB, failcnt 45598831
      Sep 30 10:18:35 kernel: memory+swap: usage 2621444kB, limit 2621440kB, failcnt 13806683
      Sep 30 10:18:35 kernel: kmem: usage 407384kB, limit 9007199254740988kB, failcnt 0
      Sep 30 10:18:35 kernel: Memory cgroup stats for /machine.slice/3657: rss_huge:0KB mapped_file:338136KB shmem:406204KB slab_unreclaimable:13824KB swap:524388KB cache:406476KB rss:1283196KB slab_reclaimable:355864KB inactive_anon:432588KB active_anon:1256812KB inactive_file:40KB active_file:12KB unevictable:0KB
      Sep 30 10:18:35 kernel: Memory cgroup out of memory: Kill process 828656 (clamd) score 283 or sacrifice child
      Sep 30 10:18:35 kernel: Killed process 828656 (clamd) in VE "3657" total-vm:1032160kB, anon-rss:628756kB, file-rss:120kB, shmem-rss:0kB
      Sep 30 10:18:35 kernel: EXT4-fs error (device ploop43739p1) in ext4_orphan_del:2723: Journal has aborted
      Sep 30 10:18:35 kernel: EXT4-fs error (device ploop43739p1) in ext4_reserve_inode_write:5354: Journal has aborted
      ============================================================ ============================================

      After that, VE will only mount with an fsck.

      Looks like a bug.

        Attachments

          Activity

            People

            Assignee:
            vvs Vasily Averin
            Reporter:
            vvs Vasily Averin
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: