Transfer sparse files under Linux

Hello,

The main idea of this is :

We have bare-metal server A and we want to copy the virtual machine from A to bare-metal server B. If the virtual machine is 80GB and the connectivity between the two servers is 1GB the transfer will be ready for 20 minutes.

If our virtual machine is 1GB big, the transfer will still take so much because we will transfer all 80GB – 79GB of which zeros.

In this post I will share a way with tar how to avoid this annoying transfer of redundant data to sparse files.

Create a sparse file

For the test, I created a virtual machine with a size of 80GB hard drive :

stat vm-101-disk-1.qcow2
File: vm-101-disk-1.qcow2
Size: 85912715264 Blocks: 26008 IO Block: 4096 regular file
Device: fd02h/64770d Inode: 46792713 Links: 1
Access: (0640/-rw-r-----) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2018-09-12 09:30:28.543834708 +0300
Modify: 2018-09-12 09:29:06.787111495 +0300
Change: 2018-09-12 09:29:06.787111495 +0300
Birth: -

ls -lah shows me this :

ls -lha vm-101-disk-1.qcow2
-rw-r----- 1 root root 81G Sep 12 09:29 vm-101-disk-1.qcow2

But du -sh shows the real size :

du -sh vm-101-disk-1.qcow2
13M vm-101-disk-1.qcow2

Transfer sparse file

Standard copy modes – cp or scp will copy all 80GB :

scp vm-101-disk-1.qcow2 192.168.1.2:/var/lib/vz/
vm-101-disk-1.qcow2 0% 603MB 83.5MB/s 16:13 ETA

ETA (Estimated time of arrival) is 16-17 minutes.

However, the actual file size is 13M ?!??!? – 16minutes??? 🙂

Here’s how we can avoid this with the help of tar!

tar cvSzf - vm-101-disk-1.qcow2 | ssh 192.168.1.2 'cd /var/lib/vz/ ; tar xzf -'

The main idea is :

• c – create archive
• v – verbose
• S – sparse – the solution of our problem! – (Handle sparse files efficiently)
• z – gzip
• f – name of the file
• x – extract the archive

Now our file vm-101-disk-1.qcow2 which is 80GB with real size of 13MB will be copied to the bare-metal server B for a second 🙂

This is very useful for distant bare-metal servers with big virtual machines!

PS. it is very important to follow the tar sequence to avoid deleting our file.

Thanks to @Румен for the idea!