Details
-
Type: Bug
-
Status: Resolved
-
Priority: Minor
-
Resolution: Invalid
-
Fix Version/s: OpenVZ-legacy
-
Component/s: Containers::Kernel
-
Security Level: Public
-
Environment:Operating System: RHEL/CentOS 6
Platform: Other
-
External issue URL:
-
External issue ID:3092
Description
Hello!
We got this messages on one heavy loaded nodes:
cat /var/log/messages|grep nf_conntrack
Oct 9 07:01:50 evo12 kernel: [662670.048839] nf_conntrack: falling back to vmalloc.
Oct 9 08:52:05 evo12 kernel: [669281.064677] nf_conntrack: falling back to vmalloc.
Oct 9 14:00:14 evo12 kernel: [687757.891278] nf_conntrack: falling back to vmalloc.
Oct 9 15:15:14 evo12 kernel: [692254.440065] nf_conntrack: falling back to vmalloc.
Oct 9 19:10:09 evo12 kernel: [706341.132158] nf_conntrack: falling back to vmalloc.
Oct 9 21:46:24 evo12 kernel: [715709.743746] nf_conntrack: falling back to vmalloc.
Oct 9 21:46:24 evo12 kernel: [715709.765083] nf_conntrack: falling back to vmalloc.
Oct 9 22:29:27 evo12 kernel: [718290.751622] nf_conntrack: falling back to vmalloc.
Oct 9 22:29:27 evo12 kernel: [718290.784883] nf_conntrack: falling back to vmalloc.
Oct 10 00:37:19 evo12 kernel: [725957.516900] nf_conntrack: falling back to vmalloc.
Oct 10 00:37:19 evo12 kernel: [725957.525601] nf_conntrack: falling back to vmalloc.
Oct 10 03:04:36 evo12 kernel: [734789.492685] nf_conntrack: falling back to vmalloc.
Oct 10 03:04:37 evo12 kernel: [734789.745141] nf_conntrack: falling back to vmalloc.
Oct 10 11:40:08 evo12 kernel: [765700.968148] nf_conntrack: falling back to vmalloc.
Oct 10 11:40:08 evo12 kernel: [765700.987912] nf_conntrack: falling back to vmalloc.
Oct 10 12:13:22 evo12 kernel: [767693.634866] nf_conntrack: falling back to vmalloc.
Oct 10 12:42:47 evo12 kernel: [769457.687523] nf_conntrack: falling back to vmalloc.
Oct 10 13:22:55 evo12 kernel: [771863.840371] nf_conntrack: falling back to vmalloc.
Oct 10 20:03:15 evo12 kernel: [795868.692005] nf_conntrack: falling back to vmalloc.
Oct 11 09:20:10 evo12 kernel: [843652.263531] nf_conntrack: falling back to vmalloc.
Oct 11 09:20:10 evo12 kernel: [843652.285770] nf_conntrack: falling back to vmalloc.
Oct 11 12:15:59 evo12 kernel: [854194.623251] nf_conntrack: falling back to vmalloc.
Oct 11 14:07:34 evo12 kernel: [860884.352007] nf_conntrack: falling back to vmalloc.
Oct 11 14:07:34 evo12 kernel: [860884.362122] nf_conntrack: falling back to vmalloc.
cat /etc/modprobe.d/nf_conntrack.conf
options nf_conntrack ip_conntrack_disable_ve0=1
I found this code https://github.com/pavel-odintsov/openvz_rhel6_kernel_mirror/blob/c33442f2693bbcc5888e7027f77c1849e4024eab/net/netfilter/nf_conntrack_core.c but can't understand why __get_free_pages(GFP_KERNEL_UBC | __GFP_NOWARN | __GFP_ZERO,get_order(sz)) can't allocate memory because we have bunch of free memory:
cat /proc/meminfo
MemTotal: 263963704 kB
MemFree: 14928504 kB
Buffers: 5749136 kB
Cached: 84719852 kB
SwapCached: 5290772 kB
MemCommitted: 735648384 kB
VirtualSwap: 0 kB
Active: 104220612 kB
Inactive: 122695412 kB
Active(anon): 68279040 kB
Inactive(anon): 72773180 kB
Active(file): 35941572 kB
Inactive(file): 49922232 kB
Unevictable: 729480 kB
Mlocked: 729480 kB
SwapTotal: 20971516 kB
SwapFree: 14935036 kB
Dirty: 308060 kB
Writeback: 0 kB
AnonPages: 132163212 kB
Mapped: 12636472 kB
Shmem: 4075040 kB
Slab: 14664700 kB
SReclaimable: 11361964 kB
SUnreclaim: 3302736 kB
KernelStack: 346488 kB
PageTables: 3831676 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 152953368 kB
Committed_AS: 333618296 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 1967732 kB
VmallocChunk: 34224678832 kB
HardwareCorrupted: 0 kB
AnonHugePages: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 6140 kB
DirectMap2M: 2054144 kB
DirectMap1G: 266338304 kB
uname -r
2.6.32-042stab093.5
We got this messages on one heavy loaded nodes:
cat /var/log/messages|grep nf_conntrack
Oct 9 07:01:50 evo12 kernel: [662670.048839] nf_conntrack: falling back to vmalloc.
Oct 9 08:52:05 evo12 kernel: [669281.064677] nf_conntrack: falling back to vmalloc.
Oct 9 14:00:14 evo12 kernel: [687757.891278] nf_conntrack: falling back to vmalloc.
Oct 9 15:15:14 evo12 kernel: [692254.440065] nf_conntrack: falling back to vmalloc.
Oct 9 19:10:09 evo12 kernel: [706341.132158] nf_conntrack: falling back to vmalloc.
Oct 9 21:46:24 evo12 kernel: [715709.743746] nf_conntrack: falling back to vmalloc.
Oct 9 21:46:24 evo12 kernel: [715709.765083] nf_conntrack: falling back to vmalloc.
Oct 9 22:29:27 evo12 kernel: [718290.751622] nf_conntrack: falling back to vmalloc.
Oct 9 22:29:27 evo12 kernel: [718290.784883] nf_conntrack: falling back to vmalloc.
Oct 10 00:37:19 evo12 kernel: [725957.516900] nf_conntrack: falling back to vmalloc.
Oct 10 00:37:19 evo12 kernel: [725957.525601] nf_conntrack: falling back to vmalloc.
Oct 10 03:04:36 evo12 kernel: [734789.492685] nf_conntrack: falling back to vmalloc.
Oct 10 03:04:37 evo12 kernel: [734789.745141] nf_conntrack: falling back to vmalloc.
Oct 10 11:40:08 evo12 kernel: [765700.968148] nf_conntrack: falling back to vmalloc.
Oct 10 11:40:08 evo12 kernel: [765700.987912] nf_conntrack: falling back to vmalloc.
Oct 10 12:13:22 evo12 kernel: [767693.634866] nf_conntrack: falling back to vmalloc.
Oct 10 12:42:47 evo12 kernel: [769457.687523] nf_conntrack: falling back to vmalloc.
Oct 10 13:22:55 evo12 kernel: [771863.840371] nf_conntrack: falling back to vmalloc.
Oct 10 20:03:15 evo12 kernel: [795868.692005] nf_conntrack: falling back to vmalloc.
Oct 11 09:20:10 evo12 kernel: [843652.263531] nf_conntrack: falling back to vmalloc.
Oct 11 09:20:10 evo12 kernel: [843652.285770] nf_conntrack: falling back to vmalloc.
Oct 11 12:15:59 evo12 kernel: [854194.623251] nf_conntrack: falling back to vmalloc.
Oct 11 14:07:34 evo12 kernel: [860884.352007] nf_conntrack: falling back to vmalloc.
Oct 11 14:07:34 evo12 kernel: [860884.362122] nf_conntrack: falling back to vmalloc.
cat /etc/modprobe.d/nf_conntrack.conf
options nf_conntrack ip_conntrack_disable_ve0=1
I found this code https://github.com/pavel-odintsov/openvz_rhel6_kernel_mirror/blob/c33442f2693bbcc5888e7027f77c1849e4024eab/net/netfilter/nf_conntrack_core.c but can't understand why __get_free_pages(GFP_KERNEL_UBC | __GFP_NOWARN | __GFP_ZERO,get_order(sz)) can't allocate memory because we have bunch of free memory:
cat /proc/meminfo
MemTotal: 263963704 kB
MemFree: 14928504 kB
Buffers: 5749136 kB
Cached: 84719852 kB
SwapCached: 5290772 kB
MemCommitted: 735648384 kB
VirtualSwap: 0 kB
Active: 104220612 kB
Inactive: 122695412 kB
Active(anon): 68279040 kB
Inactive(anon): 72773180 kB
Active(file): 35941572 kB
Inactive(file): 49922232 kB
Unevictable: 729480 kB
Mlocked: 729480 kB
SwapTotal: 20971516 kB
SwapFree: 14935036 kB
Dirty: 308060 kB
Writeback: 0 kB
AnonPages: 132163212 kB
Mapped: 12636472 kB
Shmem: 4075040 kB
Slab: 14664700 kB
SReclaimable: 11361964 kB
SUnreclaim: 3302736 kB
KernelStack: 346488 kB
PageTables: 3831676 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 152953368 kB
Committed_AS: 333618296 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 1967732 kB
VmallocChunk: 34224678832 kB
HardwareCorrupted: 0 kB
AnonHugePages: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 6140 kB
DirectMap2M: 2054144 kB
DirectMap1G: 266338304 kB
uname -r
2.6.32-042stab093.5