Details
-
Type:
Dev Task
-
Status: Closed
-
Priority:
Critical
-
Resolution: Fixed
-
Fix Version/s: Vz7.0-Update15
-
Component/s: CRIU
-
Security Level: Public
-
Environment:vz7
Description
According to Andrey, deleted overmounts are not supported jet in CRIU.
Please see parent bug for original reproducer (on OpenVZ6),
IIRC it works on vz7 too and creates container where CRIU fails on dump.
>Description of problem:
Can't resume container after suspend
>Steps to Reproduce:
1. Create new container with last CentOS 7 template (centos-7-x86_64.tar.gz )
2. Install MariaDB server from it repository and start it.
https://downloads.mariadb.org/mariadb/repositories/#mirror=best&distro=CentOS&distro_release=centos7-amd64--centos7&version=10.1
> yum install MariaDB-server MariaDB-client
> systemctl start mariadb
3. Suspend container
vzctl suspend <CTID>
4. Resume container
vzctl resume <CTID>
>Actual results:
[root@localhost]# vzctl suspend 111
Setting up checkpoint...
suspend...
dump...
kill...
Checkpointing completed successfully
Unmounting file system at /vz/root/111
Unmounting device /dev/ploop60715
Container is unmounted
[root@localhost]# vzctl resume 111
Restoring container ...
Opening delta /vz/private/111/root.hdd/root.hdd
Adding delta dev=/dev/ploop60715 img=/vz/private/111/root.hdd/root.hdd (rw)
Mounting /dev/ploop60715p1 at /vz/root/111 fstype=ext4 data='balloon_ino=12,'
Container is mounted
undump...
Adding IP address(es): 10.1.0.1
Setting CPU units: 1000
Restore error, undump failed: No such file or directory
Error: bindmount lookup failed: @14648 /dev/log
Error: Failed to restore mount point @135072 dev 'devtmpfs', type 'devtmpfs', path '/dev/log'
Error: rst_namespace: -2
Error: BUG: ctx->objcount=1
Container restore failed
Unmounting file system at /vz/root/111
Unmounting device /dev/ploop60715
Container is unmounted
>Host OS:
CentOS 6, 042stab117.6
>Guest OS:
CentOS 7, centos-7-x86_64.tar.gz
In my opinion, problem in mount namespaces, that created by systemd for mysqld process.
[root@localhost /]# ps aux|grep mysql
mysql 148 0.6 34.4 835972 90248 ? Ssl 07:06 0:00 /usr/sbin/mysqld
root 461 0.0 0.3 8992 812 pts/0 S+ 07:07 0:00 grep --color=auto mysql
[root@localhost /]# cat /proc/148/mountinfo
242 241 0:63 / /proc rw,relatime shared:73 - proc proc rw
243 241 0:61 / /sys rw,relatime shared:74 - sysfs sysfs rw
244 243 0:68 / /sys/fs/cgroup ro,nosuid,nodev,noexec shared:75 - tmpfs tmpfs ro,mode=755
245 244 0:31 / /sys/fs/cgroup/systemd rw,nosuid,nodev,noexec,relatime shared:76 - cgroup cgroup rw,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd
246 244 0:14 / /sys/fs/cgroup/blkio rw,nosuid,nodev,noexec,relatime shared:77 - cgroup cgroup rw,blkio,name=beancounter
247 244 0:15 / /sys/fs/cgroup/memory rw,nosuid,nodev,noexec,relatime shared:78 - cgroup cgroup rw,memory
248 241 0:60 / /dev rw,nosuid shared:79 - devtmpfs devtmpfs rw,mode=755
249 248 0:66 / /dev/shm rw,nosuid,nodev shared:80 - tmpfs tmpfs rw
250 248 0:65 / /dev/pts rw,nosuid,noexec,relatime shared:81 - devpts devpts rw,gid=5,mode=620,ptmxmode=000
251 248 0:62 / /dev/mqueue rw,relatime shared:82 - mqueue mqueue rw
252 241 0:67 / /run rw,nosuid,nodev shared:89 - tmpfs tmpfs rw,mode=755
241 240 182:971441 / / rw,relatime shared:72 - ext4 /dev/ploop60715p1 rw,barrier=1,data=ordered,balloon_ino=12
253 248 0:69 / /dev rw,nosuid,relatime shared:84 - tmpfs tmpfs rw,mode=755,gid=27
254 253 0:65 / /dev/pts rw,nosuid,noexec,relatime shared:85 - devpts devpts rw,gid=5,mode=620,ptmxmode=000
255 253 0:66 / /dev/shm rw,nosuid,nodev,relatime shared:86 - tmpfs tmpfs rw
256 253 0:62 / /dev/mqueue rw,relatime shared:87 - mqueue mqueue rw
257 253 0:60 /log /dev/log rw,nosuid,relatime shared:88 - devtmpfs devtmpfs rw,mode=755
258 241 0:67 /systemd/inaccessible /home ro,nosuid,nodev,relatime shared:91 - tmpfs tmpfs rw,mode=755
259 241 0:67 /systemd/inaccessible /root ro,nosuid,nodev,relatime shared:92 - tmpfs tmpfs rw,mode=755
260 252 0:67 /systemd/inaccessible /run/user ro,nosuid,nodev,relatime shared:90 - tmpfs tmpfs rw,mode=755
261 241 182:971441 /boot /boot ro,relatime shared:93 - ext4 /dev/ploop60715p1 rw,barrier=1,data=ordered,balloon_ino=12
262 241 182:971441 /etc /etc ro,relatime shared:94 - ext4 /dev/ploop60715p1 rw,barrier=1,data=ordered,balloon_ino=12
263 241 182:971441 /usr /usr ro,relatime shared:95 - ext4 /dev/ploop60715p1 rw,barrier=1,data=ordered,balloon_ino=12
Mount with a problem at this line:
257 253 0:60 /log /dev/log rw,nosuid,relatime shared:88 - devtmpfs devtmpfs rw,mode=755
For older version MariaDB from official centos repository problem does not occurs. This mounts appends by systemd. If disable PrivateDevices=true option in mysqld.service (/etc/systemd/system/mysqld.service), resuming completed fine.
Also if make snapshot several times for this container and stop it, kernel hung with soft lockup. I described this at https://bugs.openvz.org/browse/OVZ-6779
Please see parent bug for original reproducer (on OpenVZ6),
IIRC it works on vz7 too and creates container where CRIU fails on dump.
>Description of problem:
Can't resume container after suspend
>Steps to Reproduce:
1. Create new container with last CentOS 7 template (centos-7-x86_64.tar.gz )
2. Install MariaDB server from it repository and start it.
https://downloads.mariadb.org/mariadb/repositories/#mirror=best&distro=CentOS&distro_release=centos7-amd64--centos7&version=10.1
> yum install MariaDB-server MariaDB-client
> systemctl start mariadb
3. Suspend container
vzctl suspend <CTID>
4. Resume container
vzctl resume <CTID>
>Actual results:
[root@localhost]# vzctl suspend 111
Setting up checkpoint...
suspend...
dump...
kill...
Checkpointing completed successfully
Unmounting file system at /vz/root/111
Unmounting device /dev/ploop60715
Container is unmounted
[root@localhost]# vzctl resume 111
Restoring container ...
Opening delta /vz/private/111/root.hdd/root.hdd
Adding delta dev=/dev/ploop60715 img=/vz/private/111/root.hdd/root.hdd (rw)
Mounting /dev/ploop60715p1 at /vz/root/111 fstype=ext4 data='balloon_ino=12,'
Container is mounted
undump...
Adding IP address(es): 10.1.0.1
Setting CPU units: 1000
Restore error, undump failed: No such file or directory
Error: bindmount lookup failed: @14648 /dev/log
Error: Failed to restore mount point @135072 dev 'devtmpfs', type 'devtmpfs', path '/dev/log'
Error: rst_namespace: -2
Error: BUG: ctx->objcount=1
Container restore failed
Unmounting file system at /vz/root/111
Unmounting device /dev/ploop60715
Container is unmounted
>Host OS:
CentOS 6, 042stab117.6
>Guest OS:
CentOS 7, centos-7-x86_64.tar.gz
In my opinion, problem in mount namespaces, that created by systemd for mysqld process.
[root@localhost /]# ps aux|grep mysql
mysql 148 0.6 34.4 835972 90248 ? Ssl 07:06 0:00 /usr/sbin/mysqld
root 461 0.0 0.3 8992 812 pts/0 S+ 07:07 0:00 grep --color=auto mysql
[root@localhost /]# cat /proc/148/mountinfo
242 241 0:63 / /proc rw,relatime shared:73 - proc proc rw
243 241 0:61 / /sys rw,relatime shared:74 - sysfs sysfs rw
244 243 0:68 / /sys/fs/cgroup ro,nosuid,nodev,noexec shared:75 - tmpfs tmpfs ro,mode=755
245 244 0:31 / /sys/fs/cgroup/systemd rw,nosuid,nodev,noexec,relatime shared:76 - cgroup cgroup rw,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd
246 244 0:14 / /sys/fs/cgroup/blkio rw,nosuid,nodev,noexec,relatime shared:77 - cgroup cgroup rw,blkio,name=beancounter
247 244 0:15 / /sys/fs/cgroup/memory rw,nosuid,nodev,noexec,relatime shared:78 - cgroup cgroup rw,memory
248 241 0:60 / /dev rw,nosuid shared:79 - devtmpfs devtmpfs rw,mode=755
249 248 0:66 / /dev/shm rw,nosuid,nodev shared:80 - tmpfs tmpfs rw
250 248 0:65 / /dev/pts rw,nosuid,noexec,relatime shared:81 - devpts devpts rw,gid=5,mode=620,ptmxmode=000
251 248 0:62 / /dev/mqueue rw,relatime shared:82 - mqueue mqueue rw
252 241 0:67 / /run rw,nosuid,nodev shared:89 - tmpfs tmpfs rw,mode=755
241 240 182:971441 / / rw,relatime shared:72 - ext4 /dev/ploop60715p1 rw,barrier=1,data=ordered,balloon_ino=12
253 248 0:69 / /dev rw,nosuid,relatime shared:84 - tmpfs tmpfs rw,mode=755,gid=27
254 253 0:65 / /dev/pts rw,nosuid,noexec,relatime shared:85 - devpts devpts rw,gid=5,mode=620,ptmxmode=000
255 253 0:66 / /dev/shm rw,nosuid,nodev,relatime shared:86 - tmpfs tmpfs rw
256 253 0:62 / /dev/mqueue rw,relatime shared:87 - mqueue mqueue rw
257 253 0:60 /log /dev/log rw,nosuid,relatime shared:88 - devtmpfs devtmpfs rw,mode=755
258 241 0:67 /systemd/inaccessible /home ro,nosuid,nodev,relatime shared:91 - tmpfs tmpfs rw,mode=755
259 241 0:67 /systemd/inaccessible /root ro,nosuid,nodev,relatime shared:92 - tmpfs tmpfs rw,mode=755
260 252 0:67 /systemd/inaccessible /run/user ro,nosuid,nodev,relatime shared:90 - tmpfs tmpfs rw,mode=755
261 241 182:971441 /boot /boot ro,relatime shared:93 - ext4 /dev/ploop60715p1 rw,barrier=1,data=ordered,balloon_ino=12
262 241 182:971441 /etc /etc ro,relatime shared:94 - ext4 /dev/ploop60715p1 rw,barrier=1,data=ordered,balloon_ino=12
263 241 182:971441 /usr /usr ro,relatime shared:95 - ext4 /dev/ploop60715p1 rw,barrier=1,data=ordered,balloon_ino=12
Mount with a problem at this line:
257 253 0:60 /log /dev/log rw,nosuid,relatime shared:88 - devtmpfs devtmpfs rw,mode=755
For older version MariaDB from official centos repository problem does not occurs. This mounts appends by systemd. If disable PrivateDevices=true option in mysqld.service (/etc/systemd/system/mysqld.service), resuming completed fine.
Also if make snapshot several times for this container and stop it, kernel hung with soft lockup. I described this at https://bugs.openvz.org/browse/OVZ-6779
