Q-RAID 1 on TS-109 Pro creates two copies of Qmultimedia dir

Questions about SNMP, Power, System, Logs, disk, & RAID.
pakman
New here
Posts: 5
Joined: Wed Oct 03, 2007 10:07 am

Q-RAID 1 on TS-109 Pro creates two copies of Qmultimedia dir

Post by pakman »

Edit: My first speculation about hard links being responsible for this problem was wrong. Please read the whole of this thread

Hi,

I loaded about 30G of data onto my new TS-109 Pro and thought it about time to make a backup. I plugged in an eSATA disk and set it up as Q-RAID 1. When the backup was complete, I found that the eSATA disk contained a lot more data than the internal SATA. I logged on to the NAS and found this:

Code: Select all

[~] # df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/ram0                 8.7M      6.7M      2.0M  76% /
/dev/sda4                58.1M     41.5M     16.6M  72% /mnt/ext
/dev/sda1               509.5M     68.1M    441.4M  13% /mnt/HDA_ROOT
/dev/sda3               457.4G     29.8G    427.5G   7% /share/HDA_DATA
/dev/ram0                 8.7M      6.7M      2.0M  76% /share/HDA_DATA/etch/dev
/dev/sda3               457.4G     29.8G    427.5G   7% /share/HDA_DATA/etch/ext/updates
/dev/sda3               457.4G     29.8G    427.5G   7% /share/HDA_DATA/etch/ext/Qmultimedia
/dev/sda3               457.4G     29.8G    427.5G   7% /share/HDA_DATA/Public/plugins
/dev/sdb1               509.5M     68.1M    441.4M  13% /mnt/HDB_ROOT
/dev/sdb3               457.4G     58.4G    399.0G  13% /share/HDB_DATA
Note that /share/HDB_DATA has twice the volume of data of /share/HDA_DATA. A bit more digging found this:

Code: Select all

[~] # ls -id /share/HDA_DATA/Qmultimedia /share/HDA_DATA/etch/ext/Qmultimedia
14303233 /share/HDA_DATA/Qmultimedia/         14303233 /share/HDA_DATA/etch/ext/Qmultimedia/
[~] # ls -id /share/HDB_DATA/Qmultimedia /share/HDB_DATA/etch/ext/Qmultimedia
49397761 /share/HDB_DATA/Qmultimedia/         26592012 /share/HDB_DATA/etch/ext/Qmultimedia/
On /share/HDA_DATA, Qmultimedia and etch/ext/Qmultimedia have the same i-node number, so are hard-linked. On /share/HDB_DATA, the two directories are independent copies, so take twice the space.

Is this meant to be a hard link? If yes, the Q-RAID 1 process should be using 'rsync -H' to preserve the hard link.

I am running firmware 1.1.1 Build 0831T. The enabled services are:

Apple Networking
NFS
Web File Manager
FTP service
Download station
MySQL server: TCP/IP neworking

Multimedia Station and UPnP MediaServer are presently off, but have been enabled previously.

Any suggestions on how to get Q-RAID 1 to work properly?

Regards,
P.
Last edited by pakman on Tue Oct 16, 2007 9:08 pm, edited 1 time in total.
User avatar
kent
QNAP Staff
Posts: 198
Joined: Fri Aug 31, 2007 5:21 pm
Location: Taipei, Taiwan

Post by kent »

From your "df" information, it is not normal.
I guess you might manually install some program, didn't you?

Normally, without plug-in eSATA, you might get below:
Filesystem Size Used Available Use% Mounted on
/dev/ram0 8.7M 6.7M 2.0M 76% /
/dev/sda4 58.1M 41.5M 16.6M 72% /mnt/ext
/dev/sda1 509.5M 68.1M 441.4M 13% /mnt/HDA_ROOT
/dev/sda3 457.4G 29.8G 427.5G 7% /share/HDA_DATA

Regarding to your suggestion "rsync -H", we will consider it.
Thanks for your feedback.
I am Kent. Nice to meet you!
pakman
New here
Posts: 5
Joined: Wed Oct 03, 2007 10:07 am

Post by pakman »

kent wrote:From your "df" information, it is not normal.
I guess you might manually install some program, didn't you?
Ah, not me, but the people I bought it from (http://www.ultimatestorage.com) install SlimServer as standard. I'll ask them about this.
Normally, without plug-in eSATA, you might get below:
Filesystem Size Used Available Use% Mounted on
/dev/ram0 8.7M 6.7M 2.0M 76% /
/dev/sda4 58.1M 41.5M 16.6M 72% /mnt/ext
/dev/sda1 509.5M 68.1M 441.4M 13% /mnt/HDA_ROOT
/dev/sda3 457.4G 29.8G 427.5G 7% /share/HDA_DATA

Regarding to your suggestion "rsync -H", we will consider it.
Thanks for your feedback.
rsync -H may give you a performance hit. If you do decide to implement it, I would suggest allowing it to be turned on or off through the management interface, so people who aren't being affected by this kind of issue can do Q-RAID 1 without it.

Thanks for your comments,
P.
pakman
New here
Posts: 5
Joined: Wed Oct 03, 2007 10:07 am

Post by pakman »

Hi again,

Ultimate Storage haven't got back to me yet: I'll try to telephone them tomorrow if I can.

Looking at this again, I see now that it isn't a hard link that is causing the problem, but remounting various directories using 'mount --bind':

Code: Select all

/proc on /proc type proc (rw)
none on /dev/pts type devpts (rw,gid=5,mode=620)
usbfs on /proc/bus/usb type usbfs (rw)
/dev/sda4 on /mnt/ext type ext2 (rw)
/dev/sda1 on /mnt/HDA_ROOT type ext3 (rw)
/dev/sda3 on /share/HDA_DATA type ext3 (rw,data=writeback,extents,mballoc,delalloc)
/dev on /share/HDA_DATA/etch/dev type none (rw,bind)
/proc on /share/HDA_DATA/etch/proc type none (rw,bind)
/proc/bus/usb on /share/HDA_DATA/etch/proc/bus/usb type none (rw,bind)
/share/HDA_DATA/Public/updates on /share/HDA_DATA/etch/ext/updates type none (rw,bind)
/share/HDA_DATA/Qmultimedia on /share/HDA_DATA/etch/ext/Qmultimedia type none (rw,bind)
/share/HDA_DATA/etch/opt/slimserver/Plugins on /share/HDA_DATA/Public/plugins type none (rw,bind)
Going by the fact that /dev and /proc are mounted in this way, it looks as if a chroot-ed system is created under /share/HDA_DATA/etch. I don't know if 'rsync -H' would cope with this: a better strategy would be for the Q-RAID procedure to ignore filesystems mounted with --bind.

I'll keep investigating (although I had hoped that buying a NAS ready-built rather than making my own would save me this kind of hassle).

Regards,
P.
nicodemuz
Starting out
Posts: 11
Joined: Tue Dec 18, 2007 6:03 pm

Re: Q-RAID 1 on TS-109 Pro creates two copies of Qmultimedia dir

Post by nicodemuz »

Any news on this subject?
When will this problem occur (i.e. what do I as a possible qnap-buyer have to avoid to get this problem)?
pakman
New here
Posts: 5
Joined: Wed Oct 03, 2007 10:07 am

Re: Q-RAID 1 on TS-109 Pro creates two copies of Qmultimedia dir

Post by pakman »

Hi nicodemuz,

Good to see someone else taking an interest in this problem.
Any news on this subject?
Going by the (lack of) response on the Slimserver and QNAP forums, there is no sign that anyone at Progressive or QNAP has any interest in getting this fixed.
When will this problem occur (i.e. what do I as a possible qnap-buyer have to avoid to get this problem)?
I hit this problem with Slimserver: you will get this problem if you have Slimserver installed on your QNAP and you use QRAID-1 (or indeed any backup method that uses rsync). To avoid getting this problem, you should either:

(1) Not install Slimserver on the QNAP, or

(2) Uninstall Slimserver if installed, or

(3) Find a method of backing up the QNAP that doesn't use rsync, or

(4) Not back up the QNAP at all (!)

Obviously, I don't recommend (4). I went for option (2), so I never investigated option (3).

I don't know whether or not there are other third-party products like Slimserver that could be installed on the QNAP that might cause the same problem. It is easy enough to diagnose if you have access to a QNAP that has the third-party software installed: ssh into the QNAP unit and type the command:

Code: Select all

cat /etc/mtab
If the output contains any lines that has the word "bind" in the options field like this:
/share/HDA_DATA/Public/updates /share/HDA_DATA/etch/ext/updates none rw,bind 0 0
then you will be affected by this problem. There is no way of predicting in a general way if this is going to happen with any particular software: if you want to know before buying you will need to find someone who already has access to such a unit and ask them to check.

I'm not optimistic about the problem being fixed, though. Anyone from QNAP care to comment? (Better late than never.)

Regards,
P.
nicodemuz
Starting out
Posts: 11
Joined: Tue Dec 18, 2007 6:03 pm

Re: Q-RAID 1 on TS-109 Pro creates two copies of Qmultimedia dir

Post by nicodemuz »

Hi Pakman,

Many thanks for your thourough answer. Really appreciated!

My intentions with QNAP (or any competitor unit) is to do the following:
* Install a TS-109Pro at our house and store digital photo album and music album on it. Share digital photo album with my "parents in law" through the we interface.
* Install a TS-101 (or TS-109 to get encryption support) at my parents house and store their digital photo album on that one.
* Have my parents TS-101 to do weekly replication of their photo album to my TS-109Pro
* Have my TS-109Pro to do weekly replication of our digital photo album to my parents TS-101
* Have my TS-109Pro running QRAID 1 or "local" replication to get "instant" or "daily" backup. Using the replication function allows me to only backup the critical parts and save space but QRAID1 would give "instant" backup.

Other features I have been looking into but aren't as important are:
* File server to store TV recordings from my networked TV receiver and my HTPC
* Domain master to get simple password management, but requires upgrading the WinXP/Vista Home editions...
* FTP server
* Mail server (requires modding the TS-109Pro)
* Mediaserver (DLNA/UPnP) for streaming music and TV recordings to diskless mediaplayer clients
* Possibillity for modding

I know nothing about media streaming. So Slimserver tells me nothing. But from what I have understood it is a extra media streaming server as their is already one included with the unit?
But is Slimserver so much better than the one included so you "must" install it?

I have also found a competitor (Thecus) with a cheaper 2-disk solution (N299) but it has less features and I don't know if it is moddable. And I don't know about the performance. With one 500GB disk included costs about the same as a diskless TS-101. I beleive the Thecus is running rsync and ssh so it would have been great if QNAP and Thecus could replicate/backup between eachother as the Thecus would be the best solution for my parents.
guitch
Starting out
Posts: 32
Joined: Mon Aug 11, 2008 7:57 pm

Re: Q-RAID 1 on TS-109 Pro creates two copies of Qmultimedia dir

Post by guitch »

Ok, I should have started to look at the 1st posts :)
So, it looks I've this very same problem : all ids (Qmultimedia, Qusb, Public ...) are different in HDB_DATA, while they're equal on HDA_DATA. I indeed have a few binds too...

This problem first occured when I restarted the NAS. It never re-occured after 2 other reboots, but I still have 2 copies of each folders.
What can I do to resolve, even temporaly, this issue ? How can I reformat my backup HDD, and restart a fresh QRaid 1 copy ?

Regards
Guitch
-Model name: TS-109 II
-Firmware Version: v3.0.0 Build0430
-1 internal + 1 eSata HDDs
-HDDs capacity: 500GB
-QRAID1
- Services enabled: MS Networking, Download Station, Network Recycle Bin
- QPKG packages installed and running: SSOTS, Optware-ipkg, MLDonkey, phpMyAdmin
-desktop OS: Windows Vista
-desktop OS Language: French
guitch
Starting out
Posts: 32
Joined: Mon Aug 11, 2008 7:57 pm

Re: Q-RAID 1 on TS-109 Pro creates two copies of Qmultimedia dir

Post by guitch »

Hi

Okay, here are my findings this week-end:
  • - The 1st sync goes very well, and the backup HDD is a "perfect" image of the main HDD, mounts and links being created as what they are
    - After a reboot, the 2nd sync re-copies every mounts as distincts files => all data are doubled...
    - All further reboots make other syncs to start, but all files are left untouched (no mounts re-doubled, ...)
Now, after each reboot, the only solution I found is to format my backup HDD, and start a 1st sync right afterwards (1st sync = 23-24hrs)
I really think that **. QRaid-1 is really unusable, and this feature shouldn't be advertized at all.


I'm getting really upset with my QNap, as I bought it for the Multimedia Station, Easy-FTP server and QRaid 1 features... All of these are buggy, and/or useless (FTP), and made me search for hours for other, third-party solutions (Gallery 2, pure-FTP ...).

Are real solutions, real fixes being looked at for future Firmware releases ?

Thanks in advance
Rgds
Guillaume
-Model name: TS-109 II
-Firmware Version: v3.0.0 Build0430
-1 internal + 1 eSata HDDs
-HDDs capacity: 500GB
-QRAID1
- Services enabled: MS Networking, Download Station, Network Recycle Bin
- QPKG packages installed and running: SSOTS, Optware-ipkg, MLDonkey, phpMyAdmin
-desktop OS: Windows Vista
-desktop OS Language: French
mnemonic
Starting out
Posts: 33
Joined: Sat Dec 29, 2007 10:57 pm

Re: Q-RAID 1 on TS-109 Pro creates two copies of Qmultimedia dir

Post by mnemonic »

I got the same problem with it filling my external satadisk in Qraid 1 on 109 with the lateest firmware. I didn't check what makes it but I know it finished the sync and then I reboot and it fill the log with error about disk full. and its 750 Gb to sync so it takes a week to do. Not a usefull raid system. So I stopped use it even if thats why I bought it to have safe storage of my data.
So I just have to find some other backup solotion instead, if they dont fix this soon.
guitch
Starting out
Posts: 32
Joined: Mon Aug 11, 2008 7:57 pm

Re: Q-RAID 1 on TS-109 Pro creates two copies of Qmultimedia dir

Post by guitch »

Hi

I've installed the latest stable firmware last week-end, and the issue is still occuring.
It's a very annoying one, as I must format and re-sync my whole HDD each time I just restart my NAS server.......
Isn't there any other software than rsync that would do the expected job ?

Rgds
Guitch
-Model name: TS-109 II
-Firmware Version: v3.0.0 Build0430
-1 internal + 1 eSata HDDs
-HDDs capacity: 500GB
-QRAID1
- Services enabled: MS Networking, Download Station, Network Recycle Bin
- QPKG packages installed and running: SSOTS, Optware-ipkg, MLDonkey, phpMyAdmin
-desktop OS: Windows Vista
-desktop OS Language: French
Ronson1909
Starting out
Posts: 36
Joined: Sun Nov 25, 2007 9:10 pm

Re: Q-RAID 1 on TS-109 Pro creates two copies of Qmultimedia dir

Post by Ronson1909 »

/share/HDA_DATA/Qdownload is mounted on /mnt/HDA_ROOT/rootfs_2_3_6/share/Qdownload as well (in my opinion even without additional software), such that it well be rsync-ed twice. The mount command is located in /etc/rcS.d/S65btd which is started if the download station is activated. Will try to replace it with a soft link. That's really a serious bug.
mnemonic
Starting out
Posts: 33
Joined: Sat Dec 29, 2007 10:57 pm

Re: Q-RAID 1 on TS-109 Pro creates two copies of Qmultimedia dir

Post by mnemonic »

I uninstalled my optware in QPKG and that seem to fix the problem for me.
marcz
New here
Posts: 5
Joined: Thu Jul 02, 2009 4:13 am

Re: Q-RAID 1 on TS-109 Pro creates two copies of Qmultimedia dir

Post by marcz »

If I look at my mtab I find

Code: Select all

# cat /etc/mtab|grep bind
/bin /mnt/HDA_ROOT/rootfs_2_3_6/bin none rw,bind 0 0
/dev /mnt/HDA_ROOT/rootfs_2_3_6/dev none rw,bind 0 0
/mnt/HDA_ROOT/.config /mnt/HDA_ROOT/rootfs_2_3_6/etc/config none rw,bind 0 0
/proc /mnt/HDA_ROOT/rootfs_2_3_6/proc none rw,bind 0 0
/tmp /mnt/HDA_ROOT/rootfs_2_3_6/tmp none rw,bind 0 0
/share/HDA_DATA/.qpkg /mnt/HDA_ROOT/rootfs_2_3_6/share/HDA_DATA/.qpkg none rw,bind 0 0
/share/HDA_DATA/Qdownload /mnt/HDA_ROOT/rootfs_2_3_6/share/Qdownload none rw,bind 0 0
All the mount bind will be replicated by rsync that cannot detect them even with the -x option (stated in rsync(1) man page) Some directories are either dynamic and volatile like /dev /proc /tmp, /bin is a part of the ramdisk, /mnt/HDA_ROOT/.config is yet saved as /.config (except 2 excluded files), /share/* are saved on the share image, the 3-rd partition

When you remove these directories from your replicate the size of both source and image is very close (can differ somewhat as far as the rsync options does not include hardlink resolution)

So the better solution is to exclude these mounted directories from the backup we can achieve that by putting at the end of /etc/default_config/Qraid1_root_exclude.conf:

Code: Select all

/rootfs_2_3_6/bin/**
/rootfs_2_3_6/dev/**
/rootfs_2_3_6/etc/config/**
/rootfs_2_3_6/proc/**
/rootfs_2_3_6/tmp/**
/rootfs_2_3_6/share/HDA_DATA/.qpkg/**
/rootfs_2_3_6/share/Qdownload/**
But his file is overwritten at each boot, so you have to do

Code: Select all

cp /etc/default_config/Qraid1_root_exclude.conf /mnt/HDA_ROOT/.config/
What would you think at Qnap to release qraid1 as GPL? It would help us to solve bugs and develop improvements
1stade
Starting out
Posts: 13
Joined: Wed Sep 05, 2007 6:16 pm

Re: Q-RAID 1 on TS-109 Pro creates two copies of Qmultimedia dir

Post by 1stade »

Can someone help me out here, please.

I seem to have pretty much the same problem as others. I've just used Qraid to swap over hard drive (increase capacity from .5Tb to 1.5Tb), as recommended elsewhere in the forums. The qraid took a couple of days, and the data is all replicated nicely. However, whilst I've got 2 identical drives, I thought it would be a good opportunity to update firmware. There are a couple of additions in the firmware that might be useful.

However the update fails every time, irregardless of how I do it. Doing some investigation I appear to have too much data in my HDA_ROOT directory,
and there is not enough room in this to allow the update to complete.

I appear to have the same issue as just described, with some binds that I guess have duplicated some of the directories.
Here's the output of /etc/mtab

Ä/Ü # cat /etc/mtab
/proc /proc proc rw 0 0
none /dev/pts devpts rw,gid=5,mode=620 0 0
sysfs /sys sysfs rw 0 0
tmpfs /tmp tmpfs rw,size=16M 0 0
usbfs /proc/bus/usb usbfs rw 0 0
/dev/sda4 /mnt/ext ext3 rw 0 0
/dev/sda1 /mnt/HDA_ROOT ext3 rw 0 0
/dev/sda3 /share/HDA_DATA ext3 rw,usrjquota=aquota.user,jqfmt=vfsv0,data=writeback,extents 0 0
/bin /mnt/HDA_ROOT/rootfs_2_3_6/bin none rw,bind 0 0
/dev /mnt/HDA_ROOT/rootfs_2_3_6/dev none rw,bind 0 0
/mnt/HDA_ROOT/.config /mnt/HDA_ROOT/rootfs_2_3_6/etc/config none rw,bind 0 0
/proc /mnt/HDA_ROOT/rootfs_2_3_6/proc none rw,

Now I only have Windows networking, itunes server, NFS (can be turned off), ftp, download, turned on.
I did have slimserver installed once, but should have been uninstalled.

I don't want to go down the road of backing everything up to external drive, wiping hdd in nas, and restoring data back, setting up config etc. again, as the NAS is perfect apart from this ability to update, and thats a weeks worth of work.
I gather or assume from what I've read that others are trying to improve the functionality of Qraid but importantly know how to"fix" their NAS. What do I have to do to get it to Normal without going down the path described above. Are their simple unmount commands I need to run so simply delete those folders that are taking up the space?

Any help greatly appreciated. I have logged call with Qnap support but they haven't come back to me.

Thanks
Adrian
Post Reply

Return to “System & Disk Volume Management”