Howto - VRangesync 'intelligent' move feature save time, save bandwidth and extend lifespan of backup storage device
1) New work
photographer
graphic designer
moviemaker
musician
It is start of the new month.
You started to do your routine research on internet and save alot new files onto your workcomputer over the month. You also have an automatic scheduled job that run once a day which backup your workcomputer to your secondary storage device.
A large list of new files get accumulate on your workcomputer. Your backup secondary storage device also get an exact copy everyday as well.
# mp3 - 100 items; each file is 6.2mb size
# mp4 - 50 items; each file is 55mb size
# png - 100 items; each file is 3.2mb size
# jpg - 100 items; each file is 4.2mb size
# psd - 30 items; # each file is 32mb size
# zip - 5 items; each file is 100mb size
# pdf - 30 items; each file is 32mb size
# iso - 5 items; each file is 2.2G size
# a total of 315 items
$ ls src|wc -l
315
# a total size of 16GB
$ du -sh src
16G src
$ ls src/
i1.iso j19.jpg j32.jpg j46.jpg j5.jpg j73.jpg j87.jpg p10.pdf p17.psd p24.pdf p5.pdf v15.mp4 v29.mp4 v42.mp4 z100.mp3 z22.mp3 z35.mp3 z48.mp3 z5.zip z73.mp3 z87.mp3
i2.iso j1.jpg j33.jpg j47.jpg j60.jpg j74.jpg j88.jpg p10.psd p18.pdf p24.psd p5.psd v16.mp4 v2.mp4 v43.mp4 z10.mp3 z23.mp3 z36.mp3 z49.mp3 z60.mp3 z74.mp3 z88.mp3
i3.iso j20.jpg j34.jpg j48.jpg j61.jpg j75.jpg j89.jpg p11.pdf p18.psd p25.pdf p6.pdf v17.mp4 v30.mp4 v44.mp4 z11.mp3 z24.mp3 z37.mp3 z4.mp3 z61.mp3 z75.mp3 z89.mp3
i4.iso j21.jpg j35.jpg j49.jpg j62.jpg j76.jpg j8.jpg p11.psd p19.pdf p25.psd p6.psd v18.mp4 v31.mp4 v45.mp4 z12.mp3 z25.mp3 z38.mp3 z4.zip z62.mp3 z76.mp3 z8.mp3
i5.iso j22.jpg j36.jpg j4.jpg j63.jpg j77.jpg j90.jpg p12.pdf p19.psd p26.pdf p7.pdf v19.mp4 v32.mp4 v46.mp4 z13.mp3 z26.mp3 z39.mp3 z50.mp3 z63.mp3 z77.mp3 z90.mp3
j100.jpg j23.jpg j37.jpg j50.jpg j64.jpg j78.jpg j91.jpg p12.psd p1.pdf p27.pdf p7.psd v1.mp4 v33.mp4 v47.mp4 z14.mp3 z27.mp3 z3.mp3 z51.mp3 z64.mp3 z78.mp3 z91.mp3
j10.jpg j24.jpg j38.jpg j51.jpg j65.jpg j79.jpg j92.jpg p13.pdf p1.psd p28.pdf p8.pdf v20.mp4 v34.mp4 v48.mp4 z15.mp3 z28.mp3 z3.zip z52.mp3 z65.mp3 z79.mp3 z92.mp3
j11.jpg j25.jpg j39.jpg j52.jpg j66.jpg j7.jpg j93.jpg p13.psd p20.pdf p29.pdf p8.psd v21.mp4 v35.mp4 v49.mp4 z16.mp3 z29.mp3 z40.mp3 z53.mp3 z66.mp3 z7.mp3 z93.mp3
j12.jpg j26.jpg j3.jpg j53.jpg j67.jpg j80.jpg j94.jpg p14.pdf p20.psd p2.pdf p9.pdf v22.mp4 v36.mp4 v4.mp4 z17.mp3 z2.mp3 z41.mp3 z54.mp3 z67.mp3 z80.mp3 z94.mp3
j13.jpg j27.jpg j40.jpg j54.jpg j68.jpg j81.jpg j95.jpg p14.psd p21.pdf p2.psd p9.psd v23.mp4 v37.mp4 v50.mp4 z18.mp3 z2.zip z42.mp3 z55.mp3 z68.mp3 z81.mp3 z95.mp3
j14.jpg j28.jpg j41.jpg j55.jpg j69.jpg j82.jpg j96.jpg p15.pdf p21.psd p30.pdf v10.mp4 v24.mp4 v38.mp4 v5.mp4 z19.mp3 z30.mp3 z43.mp3 z56.mp3 z69.mp3 z82.mp3 z96.mp3
j15.jpg j29.jpg j42.jpg j56.jpg j6.jpg j83.jpg j97.jpg p15.psd p22.pdf p3.pdf v11.mp4 v25.mp4 v39.mp4 v6.mp4 z1.mp3 z31.mp3 z44.mp3 z57.mp3 z6.mp3 z83.mp3 z97.mp3
j16.jpg j2.jpg j43.jpg j57.jpg j70.jpg j84.jpg j98.jpg p16.pdf p22.psd p3.psd v12.mp4 v26.mp4 v3.mp4 v7.mp4 z1.zip z32.mp3 z45.mp3 z58.mp3 z70.mp3 z84.mp3 z98.mp3
j17.jpg j30.jpg j44.jpg j58.jpg j71.jpg j85.jpg j99.jpg p16.psd p23.pdf p4.pdf v13.mp4 v27.mp4 v40.mp4 v8.mp4 z20.mp3 z33.mp3 z46.mp3 z59.mp3 z71.mp3 z85.mp3 z99.mp3
j18.jpg j31.jpg j45.jpg j59.jpg j72.jpg j86.jpg j9.jpg p17.pdf p23.psd p4.psd v14.mp4 v28.mp4 v41.mp4 v9.mp4 z21.mp3 z34.mp3 z47.mp3 z5.mp3 z72.mp3 z86.mp3 z9.mp3
2) on last 1-2 days of the month, you decided to have a short break and use the time to do 'spring cleaning', organize the files into respective folders.
$ ls src/
iso jpg mp3 mp4 pdf psd zip
Vrangesync will move files and not copy the same files from 'src' to 'dst' again.
==============================================================================================================================
Rsync
************
# and start to sync to secondary storage device again
$ time rsync -avz --delete --progress "/data/virtual/src/" "/data/virtual/dst/"
deleting z99.mp3
deleting z98.mp3
deleting z97.mp3
deleting z96.mp3
.
.
.
iso/i1.iso
2,252,800,000 100% 145.51MB/s 0:00:14 (xfr#1, to-chk=314/323)
iso/i2.iso
2,252,800,000 100% 140.58MB/s 0:00:15 (xfr#2, to-chk=313/323)
iso/i3.iso
sent 16,578,234,034 bytes received 9,468 bytes 152,794,870.99 bytes/sec
total size is 16,574,464,000 speedup is 1.00
real 1m48.124s <-- total time to complete
user 0m26.007s
sys 0m26.839s
NOTE:
what rsync did is 1) delete files not found on src 2) sync the new files to new folder (again)
=====================================================================================================
VRangeSync
************
# Clear kernel cache to ensure more accurate test
$ sync; sudo sh -c "echo 3 > /proc/sys/vm/drop_caches"
# and start to sync to secondary storage device again
time ./vr -i "/data/virtual/src" -o "/data/virtual/dst"
real 0m5.089s <-- total time to complete
user 0m3.785s
sys 0m2.748s
NOTE:
vrangesync reduce the need to involve any 'disk i/o' activity. It do not copy files (again). It just move the files within the destination space.
1)Extend the lifespan of the disk.
2)complete file sync in shortest time
3)Save on bandwidth
[313/315] Processing data copy file for /data/virtual/src/mp3/z68.mp3
-- move '/data/virtual/dst/z68.mp3' '/data/virtual/dst/mp3/z68.mp3'
[314/315] Processing data copy file for /data/virtual/src/mp3/z52.mp3
-- move '/data/virtual/dst/z52.mp3' '/data/virtual/dst/mp3/z52.mp3'
[315/315] Processing data copy file for /data/virtual/src/mp3/z47.mp3
-- move '/data/virtual/dst/z47.mp3' '/data/virtual/dst/mp3/z47.mp3'
===========================================
Vrangesync offer free and paid license.
Registered Free license - you need to renew the license once a month
Registered paid license - you need to renew the license once a year
Next time, when you want to backup and sync data to storage device, do check out vrangesync. It is convenient and efficient data file copier.
Other similar software are : rsync , robocopy
No comments:
Post a Comment