Details
-
Type: Bug
-
Status: Open
-
Priority: Critical
-
Resolution: Incomplete
-
Fix Version/s: Vz6.0-Update-next, OpenVZ-legacy, CU-rh6-next-stable
-
Component/s: Containers::Kernel
-
Security Level: Public
-
Environment:CentOS 6, 2.6.32-042stab117.6
Description
>Description of problem:
Can't resume container after suspend
>Steps to Reproduce:
1. Create new container with last CentOS 7 template (centos-7-x86_64.tar.gz )
2. Install MariaDB server from it repository and start it.
https://downloads.mariadb.org/mariadb/repositories/#mirror=best&distro=CentOS&distro_release=centos7-amd64--centos7&version=10.1
> yum install MariaDB-server MariaDB-client
> systemctl start mariadb
3. Suspend container
vzctl suspend <CTID>
4. Resume container
vzctl resume <CTID>
>Actual results:
[root@localhost]# vzctl suspend 111
Setting up checkpoint...
suspend...
dump...
kill...
Checkpointing completed successfully
Unmounting file system at /vz/root/111
Unmounting device /dev/ploop60715
Container is unmounted
[root@localhost]# vzctl resume 111
Restoring container ...
Opening delta /vz/private/111/root.hdd/root.hdd
Adding delta dev=/dev/ploop60715 img=/vz/private/111/root.hdd/root.hdd (rw)
Mounting /dev/ploop60715p1 at /vz/root/111 fstype=ext4 data='balloon_ino=12,'
Container is mounted
undump...
Adding IP address(es): 10.1.0.1
Setting CPU units: 1000
Restore error, undump failed: No such file or directory
Error: bindmount lookup failed: @14648 /dev/log
Error: Failed to restore mount point @135072 dev 'devtmpfs', type 'devtmpfs', path '/dev/log'
Error: rst_namespace: -2
Error: BUG: ctx->objcount=1
Container restore failed
Unmounting file system at /vz/root/111
Unmounting device /dev/ploop60715
Container is unmounted
>Host OS:
CentOS 6, 042stab117.6
>Guest OS:
CentOS 7, centos-7-x86_64.tar.gz
In my opinion, problem in mount namespaces, that created by systemd for mysqld process.
[root@localhost /]# ps aux|grep mysql
mysql 148 0.6 34.4 835972 90248 ? Ssl 07:06 0:00 /usr/sbin/mysqld
root 461 0.0 0.3 8992 812 pts/0 S+ 07:07 0:00 grep --color=auto mysql
[root@localhost /]# cat /proc/148/mountinfo
242 241 0:63 / /proc rw,relatime shared:73 - proc proc rw
243 241 0:61 / /sys rw,relatime shared:74 - sysfs sysfs rw
244 243 0:68 / /sys/fs/cgroup ro,nosuid,nodev,noexec shared:75 - tmpfs tmpfs ro,mode=755
245 244 0:31 / /sys/fs/cgroup/systemd rw,nosuid,nodev,noexec,relatime shared:76 - cgroup cgroup rw,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd
246 244 0:14 / /sys/fs/cgroup/blkio rw,nosuid,nodev,noexec,relatime shared:77 - cgroup cgroup rw,blkio,name=beancounter
247 244 0:15 / /sys/fs/cgroup/memory rw,nosuid,nodev,noexec,relatime shared:78 - cgroup cgroup rw,memory
248 241 0:60 / /dev rw,nosuid shared:79 - devtmpfs devtmpfs rw,mode=755
249 248 0:66 / /dev/shm rw,nosuid,nodev shared:80 - tmpfs tmpfs rw
250 248 0:65 / /dev/pts rw,nosuid,noexec,relatime shared:81 - devpts devpts rw,gid=5,mode=620,ptmxmode=000
251 248 0:62 / /dev/mqueue rw,relatime shared:82 - mqueue mqueue rw
252 241 0:67 / /run rw,nosuid,nodev shared:89 - tmpfs tmpfs rw,mode=755
241 240 182:971441 / / rw,relatime shared:72 - ext4 /dev/ploop60715p1 rw,barrier=1,data=ordered,balloon_ino=12
253 248 0:69 / /dev rw,nosuid,relatime shared:84 - tmpfs tmpfs rw,mode=755,gid=27
254 253 0:65 / /dev/pts rw,nosuid,noexec,relatime shared:85 - devpts devpts rw,gid=5,mode=620,ptmxmode=000
255 253 0:66 / /dev/shm rw,nosuid,nodev,relatime shared:86 - tmpfs tmpfs rw
256 253 0:62 / /dev/mqueue rw,relatime shared:87 - mqueue mqueue rw
257 253 0:60 /log /dev/log rw,nosuid,relatime shared:88 - devtmpfs devtmpfs rw,mode=755
258 241 0:67 /systemd/inaccessible /home ro,nosuid,nodev,relatime shared:91 - tmpfs tmpfs rw,mode=755
259 241 0:67 /systemd/inaccessible /root ro,nosuid,nodev,relatime shared:92 - tmpfs tmpfs rw,mode=755
260 252 0:67 /systemd/inaccessible /run/user ro,nosuid,nodev,relatime shared:90 - tmpfs tmpfs rw,mode=755
261 241 182:971441 /boot /boot ro,relatime shared:93 - ext4 /dev/ploop60715p1 rw,barrier=1,data=ordered,balloon_ino=12
262 241 182:971441 /etc /etc ro,relatime shared:94 - ext4 /dev/ploop60715p1 rw,barrier=1,data=ordered,balloon_ino=12
263 241 182:971441 /usr /usr ro,relatime shared:95 - ext4 /dev/ploop60715p1 rw,barrier=1,data=ordered,balloon_ino=12
Mount with a problem at this line:
257 253 0:60 /log /dev/log rw,nosuid,relatime shared:88 - devtmpfs devtmpfs rw,mode=755
For older version MariaDB from official centos repository problem does not occurs. This mounts appends by systemd. If disable PrivateDevices=true option in mysqld.service (/etc/systemd/system/mysqld.service), resuming completed fine.
Also if make snapshot several times for this container and stop it, kernel hung with soft lockup. I described this at https://bugs.openvz.org/browse/OVZ-6779
Can't resume container after suspend
>Steps to Reproduce:
1. Create new container with last CentOS 7 template (centos-7-x86_64.tar.gz )
2. Install MariaDB server from it repository and start it.
https://downloads.mariadb.org/mariadb/repositories/#mirror=best&distro=CentOS&distro_release=centos7-amd64--centos7&version=10.1
> yum install MariaDB-server MariaDB-client
> systemctl start mariadb
3. Suspend container
vzctl suspend <CTID>
4. Resume container
vzctl resume <CTID>
>Actual results:
[root@localhost]# vzctl suspend 111
Setting up checkpoint...
suspend...
dump...
kill...
Checkpointing completed successfully
Unmounting file system at /vz/root/111
Unmounting device /dev/ploop60715
Container is unmounted
[root@localhost]# vzctl resume 111
Restoring container ...
Opening delta /vz/private/111/root.hdd/root.hdd
Adding delta dev=/dev/ploop60715 img=/vz/private/111/root.hdd/root.hdd (rw)
Mounting /dev/ploop60715p1 at /vz/root/111 fstype=ext4 data='balloon_ino=12,'
Container is mounted
undump...
Adding IP address(es): 10.1.0.1
Setting CPU units: 1000
Restore error, undump failed: No such file or directory
Error: bindmount lookup failed: @14648 /dev/log
Error: Failed to restore mount point @135072 dev 'devtmpfs', type 'devtmpfs', path '/dev/log'
Error: rst_namespace: -2
Error: BUG: ctx->objcount=1
Container restore failed
Unmounting file system at /vz/root/111
Unmounting device /dev/ploop60715
Container is unmounted
>Host OS:
CentOS 6, 042stab117.6
>Guest OS:
CentOS 7, centos-7-x86_64.tar.gz
In my opinion, problem in mount namespaces, that created by systemd for mysqld process.
[root@localhost /]# ps aux|grep mysql
mysql 148 0.6 34.4 835972 90248 ? Ssl 07:06 0:00 /usr/sbin/mysqld
root 461 0.0 0.3 8992 812 pts/0 S+ 07:07 0:00 grep --color=auto mysql
[root@localhost /]# cat /proc/148/mountinfo
242 241 0:63 / /proc rw,relatime shared:73 - proc proc rw
243 241 0:61 / /sys rw,relatime shared:74 - sysfs sysfs rw
244 243 0:68 / /sys/fs/cgroup ro,nosuid,nodev,noexec shared:75 - tmpfs tmpfs ro,mode=755
245 244 0:31 / /sys/fs/cgroup/systemd rw,nosuid,nodev,noexec,relatime shared:76 - cgroup cgroup rw,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd
246 244 0:14 / /sys/fs/cgroup/blkio rw,nosuid,nodev,noexec,relatime shared:77 - cgroup cgroup rw,blkio,name=beancounter
247 244 0:15 / /sys/fs/cgroup/memory rw,nosuid,nodev,noexec,relatime shared:78 - cgroup cgroup rw,memory
248 241 0:60 / /dev rw,nosuid shared:79 - devtmpfs devtmpfs rw,mode=755
249 248 0:66 / /dev/shm rw,nosuid,nodev shared:80 - tmpfs tmpfs rw
250 248 0:65 / /dev/pts rw,nosuid,noexec,relatime shared:81 - devpts devpts rw,gid=5,mode=620,ptmxmode=000
251 248 0:62 / /dev/mqueue rw,relatime shared:82 - mqueue mqueue rw
252 241 0:67 / /run rw,nosuid,nodev shared:89 - tmpfs tmpfs rw,mode=755
241 240 182:971441 / / rw,relatime shared:72 - ext4 /dev/ploop60715p1 rw,barrier=1,data=ordered,balloon_ino=12
253 248 0:69 / /dev rw,nosuid,relatime shared:84 - tmpfs tmpfs rw,mode=755,gid=27
254 253 0:65 / /dev/pts rw,nosuid,noexec,relatime shared:85 - devpts devpts rw,gid=5,mode=620,ptmxmode=000
255 253 0:66 / /dev/shm rw,nosuid,nodev,relatime shared:86 - tmpfs tmpfs rw
256 253 0:62 / /dev/mqueue rw,relatime shared:87 - mqueue mqueue rw
257 253 0:60 /log /dev/log rw,nosuid,relatime shared:88 - devtmpfs devtmpfs rw,mode=755
258 241 0:67 /systemd/inaccessible /home ro,nosuid,nodev,relatime shared:91 - tmpfs tmpfs rw,mode=755
259 241 0:67 /systemd/inaccessible /root ro,nosuid,nodev,relatime shared:92 - tmpfs tmpfs rw,mode=755
260 252 0:67 /systemd/inaccessible /run/user ro,nosuid,nodev,relatime shared:90 - tmpfs tmpfs rw,mode=755
261 241 182:971441 /boot /boot ro,relatime shared:93 - ext4 /dev/ploop60715p1 rw,barrier=1,data=ordered,balloon_ino=12
262 241 182:971441 /etc /etc ro,relatime shared:94 - ext4 /dev/ploop60715p1 rw,barrier=1,data=ordered,balloon_ino=12
263 241 182:971441 /usr /usr ro,relatime shared:95 - ext4 /dev/ploop60715p1 rw,barrier=1,data=ordered,balloon_ino=12
Mount with a problem at this line:
257 253 0:60 /log /dev/log rw,nosuid,relatime shared:88 - devtmpfs devtmpfs rw,mode=755
For older version MariaDB from official centos repository problem does not occurs. This mounts appends by systemd. If disable PrivateDevices=true option in mysqld.service (/etc/systemd/system/mysqld.service), resuming completed fine.
Also if make snapshot several times for this container and stop it, kernel hung with soft lockup. I described this at https://bugs.openvz.org/browse/OVZ-6779