Another Back-up Hardware Question

[iSign]

I have wanted to write out this question all day... almost started to at least twice, but knew it would take some time & have been putting it off.

I was reading the "why even bother" thread & saw a lot of the tech talent here on that thread & found myself spilling my guts about this huge area of mass confusion as well as of absolute importance to me and my business. (and to others I'm sure)

I decided to post the question/rant in it's own thread instead:

I have a question for you tech guys!

First some background:

I have two 500GB WD drives for backup.

I keep my current data on a server running Windows Server 2003. That server has 2 drives in it.

The first drive in my server is partitioned into C and D. C is my OS and Apps. and D is where all four of my workstations open data files from, and save data to.

The second drive (E) is where nightly backups of the D drive are written.

I consider the E drive as a backup.

My first WD MyBook is an Archive of 11 years worth of data.

The second WD MyBook is a backup of the Archives on the first one. This is stored offsite (at home)

I used to use 2 simpletech Network Drives. They are only 300 GB, so when I outgrew them, I got the WD 500GB USB Drives.

FIRST QUESTION: I recently erased my 500GB Archive backup, because it came formatted as FAT32, and would not work with my laptop at home (Vista) ...since I'm storing it there anyway, I wanted to be able to access the files there, since I work at home a lot. So, after I reformatted it to NTSF, I needed to copy the 500GB of Archive data back onto it. This data is in subdirectories of around 100 GB each.

To get the data all copied, I processed one folder at a time & that took, on average, about 12 hours each... meaning it took an entire week, run in the evenings... and often continuing the next day.

IS THERE A FASTER WAY TO MOVE THIS MUCH DATA BETWEEN WD MyBook DEVICES?

The reason I ask, is that I want BOTH devices to be Vista compatible, as it is likely that I will eventually have to use Vista at work too. I just don't want it to take a week, if I reformat the other FAT32 one.

SECOND QUESTION:
I sometimes run across files, that I guess are corrupted, because in the process of copying 100GB folders to an archive... I will get error messages which state some problem, reading such and such file... when I click "so what" (or OK, or whatever :rolleyes:)... the operation aborts & I could spend an hour figuring out what was copied already & how to select everything not yet copied EXCEPT the offending file... AND THAT'S ON TOP OF THE 4 OR 5 HOURS I ALREADY MAY HAVE LOST, NOT SEEING THE ERROR MESSAGE RIGHT AWAY!!!!

why can't I just tell the computer: COPY THE GOOD FILES... SCREW THE BAD ONES... JUST DON'T STOP UNTILL YOU DO ALL YOU CAN DO!!!

This second issue drives me absolutely nuts, and is exactly why I processed the recent copying of 500GB into several subdirectories at a time. The amazing thing with that process was that this issue NEVER happened once during that procedure.

(hmm...maybe because it's from EXTERNAL to EXTERNAL drive... and the problem I'm ranting about now, is usually when I make my FIRST archive moving data from an INTERNAL drive to an EXTERNAL drive??)

The bummer is that I now want to do another archive of 62GB from my last 3 months work (currently on my D drive on the server)... so I tried to start the copying procedure this morning... but when I checked on it... sure enough, it had been sitting idle for who knows how long... pestering me with some useless error message... which was so frustrating I failed to take note of the offending file before closing the error message & now will not be able to delete it before trying again. :Oops:

PLEASE TELL ME THERE IS A WAY TO AVOID THE PROCESS STOPPING FOR A PROBLEM FILE, INSTEAD OF JUST SKIPPING THE PROBLEM, & GOING ON WITH THE PROCESS WITHOUT ANY INTERVENTION BY ME!

[grafixemporium]

Wow. That's quite a time consuming backup plan you have there. I still don't understand why anyone would want to tote an external hard drive around with them back and forth to work. It's important to back up your data, but you have to live your life too!!

I'm not sure if this will help you or not...

USB 2.0 transfers data at a maximum burst rate of 480 megabits per second. Try to understand, that is not a constant rate... that is the maximum rate it can transfer.

A run of the mill hard drive these days running at 7200rpms will transfer data at a burst rate of around 1 gigabit per second... or more. So the nicer stuff running 10k or 15k is much faster... in excess of 1.6 gigabits per second.

You can figure out the math... but USB 2.0 is quite slow compared to any internal hard drive. Therefore, the method you are using on a regular basis to run backups and then restore them onto your laptop at home... is... well... not ideal.

Look into a backup software that will run your backup on a preset schedule. There are programs out there that can do all kinds of backups... including full, backups, incremental backups, differential backups, mirror backups. Each one has its pros and cons. Differential backups only backup the data that has changed since the last full backup. This is a great way to save tons of time.

Whatever the case, a backup should be just that... a backup. It sounds like you are using your backups as working files once you get them home. Then you run into the problem of having different variations of the same files at home and at work.

If you have to bring your work home with you, burn the file(s) you need on a CD or DVD and bring it home. Give your backup drives a rest.

By the way, if you are encountering that many corrupt files every time you are transferring data, something is wrong. One of your drives could have bad sectors. Run a full scandisk with a surface scan on all your hard drives -- which will take quite a while -- this will tell you if any drives are physically damaged. Data that gets placed in a bad sector will be unreadable. If you find bad sectors, replace that drive ASAP. Bad sectors tend to spread like the common cold.

[Sign-Man Signs]

We just back up online. We can choose what we want.

[iSign]

thanks for the reply Graphixemporium.

I think I may have failed to make my situation clear.

The routine back-ups are nightly automatic, differential backups, done from one internal to another internal hard drive, on my server, by basic backup software included on windows server 2003.

The less ideal part of my current set-up, is about every 4 months or so... I decide to clean out the primary collection of data on the server. This is to "archive it" and free up space for the next 4 months worth of data.
This is when I first move data to an external drive, after which I copy it also to a second external drive.... then, when it exists 2 other places, I delete it off the hard drive on my server. I usually do this selectively, because around 5% of my files are either for jobs in progress, or for reorder jobs that I tend to leave handy for the next reorder.

There really isn't much toting of external drives going on... just 3 times a year to give myself a second (redundant) archive, which is also stored off-site, in case of fire.

I never restore to my laptop. I just figured the archives at home might as well be compatible, in case I have a need for a file some times. I routinely use remote desktop connection to access my office files... so I never "bring work home"... and I never save a variation back to a back-up, or an archive. These drives are for read-only accessing of old files. New versions are saved with all other current data, on the server.

The biggest hassle is that quarterly (or thirdly) archiving of 2 copies of data files off the server, and your posted transfer speeds explain the reason for that very well, but even that is not so bad really... although I'd love to learn a faster way to get data off the internal drives... the real pain was deciding to reformat one of those 500GB drives.

The idea of reformatting the other drive is seeming less and less worthy of my time.

On the matter of corrupt files, I think most of that is the same files giving me grief repeatedly, because of attempts to "move" all my 11 year collection of files & encountering those old corrupt files each time... BUT, there are some new ones, so I will take your advice to heart & run some scans.

Signman, I have 600 gigs of data to keep track of, and the latest 3 months is already over 60 gigs... how long does that take to "just back up onlione"?

[Sign-Man Signs]

Our first back up was about 5 hours at 450 gig. We have four main computers that have about the same, though we don't network each. It's on automatic now and is done at night. I've used it once and it was a breeze to recover a lost file. We back up everything on each computer. I am very happy with thier service. I too used a 220 gig external drive but scrapped it after going to Carbonite.
Here's the link.

http://www.carbonite.com

[iSign]

DAMMIT!!! there it goes again....

I'm down here at twelve-freaking-thirty, trying to get ahead on a few things.... including making a second copy of this quarterly back up.... and BAM the damn procedure stops to gripe about the file bluegreen_truckflames_layers.psd

FINE!!!! skip the freakin' file!!!!!.... but NO the entire process is aborted, the folder had 8.? gigs... the folder I'm copying to now has 4.? gigs in it.... & I have to manually hunt down the offending file, sort through 8 gigs looking for what freakin nonsensical order the damn back-up was following.... and somehow select the crap that ain't dealt with yet, only to hope & pray murphy's law won't bring the roof down on me again ...WHHYYYYYYYYY????????

THERE HAS ABSOLUTELY GOT TO BE A BETTER WAY!!!!! Computers are smarter then this 99% of the things I do with them... WHY HAS THIS GOT TO BE SO FREAKING DISFUNCTIONAL!!!??*&%#@!

[Sign-Man Signs]

Man, I feel your pain. Hang in their brother. Try that Carbonite. It's free for 15 days I think.

[iSign]

may have to do that. Thanks

[signage]

Doug have you ever looked into Acronis imaging software? It creates an image of your drive that can be accessed through the software and you can get individual files or folder off. I also take less space than the original drive/files you are imaging.

www.acronis.com

[choucove]

There are a few causes for why a file would make a backup stop and throw an error like that. First, if it is a Windows component file (and thus in use by Windows) then the backup program would not be able to copy this file. (Try selecting and copying all the files in your My Documents folder and you'll see an error most likely from a few of these Windows components.) Another reason is that the file is in use or open someplace else. This probably isn't the case in your situation. And the third possible reason that I know of would be that the file was corrupted in its original backup.

Like Brian stated above, there are lots of software out there for creating an image of your drive and duplicating it to your other hard drive. The one I'm most familiar with is Norton Ghost, which can be incredibly powerful. It has a built in tool to directly duplicate one drive to the other. I've never had it pop up with an error message of being unable to re-image a file before. It also allows for compression of data to help save a little space.

Norton Ghost version 12 has been kinda hit or miss for me, but I know of several people who use version 10 or 11 without any problems. I have not yet used Acronis, but I have heard it is another good option.

[fozzyber]

Go look at a program called second cop, cheap, works very well,
and stable as a table.
And it can only back up files that have changed, AND moves deleted file to a special folder.
http://www.centered.com/
I have used it for 6+ years it backs up about 750 gig nightly.
(but not for Cypress)

[David Wright]

I have Mybook also and you might want to read this.
http://blogs.computerworld.com/why_im_done_with_portable_hard_drives

[jtrainor56]

I use a program called Smartsync Pro on both my home network and at work if I have a large amount of data to transfer. I never seem to have problems with corrupt files, it copies everything. There is a free 30 day trial... http://www.smartsync.com/

Joe

[GraphixCALC]

Mozy is another online storage solution that seems good and affordable. I've been using their free 2GB edition for a while now, just to see how it goes...so far so good.

[iSign]

about 750 gig nightly.


thanks for all the tips folks. 2GB doesn't seem to do much for me... but 750 nightly!! ...wow! Is this just a differential backup, weeding through 750GB looking for what has changed... or is this actually making copies of 750 GB?

I have Ghost 10, and use it to ghost my production machines (no data) so I can get them up and running again without reloading any programs, or re installing any drivers, or re establishing any personal settings and accounts, such as email, winfax, and ftp account settings. It has worked great for that (the one time I had to restore a ghost went perfect)

I never thought of it as a method for backing up, or archiving data. Maybe I'll try it!

[GK]

thanks for all the tips folks. 2GB doesn't seem to do much for me... but 750 nightly!! ...wow! Is this just a differential backup, weeding through 750GB looking for what has changed... or is this actually making copies of 750 GB?

I have Ghost 10, and use it to ghost my production machines (no data) so I can get them up and running again without reloading any programs, or re installing any drivers, or re establishing any personal settings and accounts, such as email, winfax, and ftp account settings. It has worked great for that (the one time I had to restore a ghost went perfect)

I never thought of it as a method for backing up, or archiving data. Maybe I'll try it!

I would seriously hope it uses some sort of indexing system to let it know what has been modified so you dont need to reupload 750 gigs of unchanged data :cool:

[iSign]

I assume it is just a differential backup, but if not, my guess (since he said it's NOT at his sign shop) ...if it really copies 750GB, that maybe he works in some industry creating, or processing massive amounts of huge files, like a photo processing, or video editing service or something....

[Rhino]

Tell you what...

You fly me down, put me up and feed me for a couple of weeks each quarter, and I'll handle your backups for you..

Deal??:thumb::thumb:

[Gene@mpls]

I second Second Copy- have used it for 6-7 years and it is very flexible. Gene

[iSign]

nice try Rhino! (but that avatar looks expensive to feed)

ok... just dl'd the second copy trial.

Last night, I gave up running the 62 gig quarterly backup to the WD external USB drive... after 3 or 4 errors spread over 3 or 4 hours...

I decided, in order to get a second 03-09 Archive before clearing the original files off the server, I would drag a copy to the other drive in my server (where nightly backups are written to)
woohoo!! this procedure did the same 62 gigs with no hiccups!! (still wonder why though?)

[Columbia Signs]

I went through the same thing. I recently tried Acronis imaging, Memeo, TrueImage 11, NovaBack and they all were less than perfect. Memeo runs in the background and as you change a file it saves it instantly. Slowed my computer down, but kinda neat to have real time back ups. The rest could be set to run at certain times of the day or night.

Then I tried Genie Backup Manager PRO 8.0 and it is PERFECT! It's the only back up software that I've found that really works. You can even set it to back up files on a network! That was a big plus. It also allows you to back up without compression. I have plenty of drive space and really like my back up files to be accessible without the need of the back up software. Another nice feature is you can set it only to back up files that have been changed or new files. Making the back up go fast.

Give it a try.

[iSign]

It also allows you to back up without compression. I have plenty of drive space and really like my back up files to be accessible without the need of the back up software.

BINGO!
That is my gripe with Ghost, or the Windows server backup (and many others). They are great for catastrophic insurance, but for archiving files I may need to hunt through... I like still being able to just root around in windows explorer, or run a search on that drive with windows search feature.

I will look into that. Thanks!

[John M]

I rely on Acronis True Image for my business, both for my machines and for dealing with customers' machines. It creates a single archive file that you can quickly mount as a drive to browse with Explorer.

You wouldn't need any of the expensive versions; Home does just fine.

For the 1-time quick duplication of your MyBooks, the only truly "fast" way would be to remove the drives from their enclosures, mount them directly to the machine, and copy. Otherwise you're at the mercy of USB speed.

You may want to consider switching from a USB backup device to a removable hard drive tray system. You can swap drives as desired but you retain the full SATA speed.

[cfbeagle]

I might say one thing about Second Copy... I use it too with good results, except...

I had it doing backups nightly on one machine to a network drive, it seems the ip address changed on the network drive. Suffice to say when I needed the backup it wasnt there... I watched the log files every now and then, never any errors???

So if you are using it on a network drive, keep an eye on it!

[signage]

Well John I have been preaching about Acronis for a while and no one seems to listen! Good to hear some one with your expertise also recommends it.

[Columbia Signs]

I liked Acronis True Image, however it could not backup network files.

[Techman]

Genie Backup Manager PRO 8.0 and it is PERFECT

Ive used genie for years, it works perfect for me.

[fozzyber]

You may want to consider switching from a USB backup device to a removable hard drive tray system. You can swap drives as desired but you retain the full SATA speed.

ammm??? what about SATA external enclosures?
http://www.newegg.com/Product/Product.aspx?Item=N82E16817153045
and a
http://www.newegg.com/Product/Product.aspx?Item=N82E16812226006

sata,+ external = fast

[gvgraphics]

Doug,

When I use to work IT we used allot of raid hot swap drives in series. What this does is allow you to back up to multiple drives simultaneously and you can install or remove a drive while the system is hot. It cost a little money but it is well worth it for the speed and the ability to hot swap the drives. You would be able to pull one of the drives at night before you leave to use at home and in the morning just slide it back in and let it do it's back-up so it is up to date for the next night.

Here is (http://www.cwol.com/firewire-drives/hot-swap-firewire-kit-1077.htm) a firewire version of them and they can be linked together to create multiple drive back-ups as you wish. They transfer at 2.4 gb per minut and that equals 144 gb per hour data transfer. That is pretty damn fast.

[Rodan68]

I've never used one but I heard the Drobo is a good solution. It has four hard drive bays that can hold a TB each. If a hard drive fails you just pull it out and replace with a new one. All the data will be replaced from the other drives. It's a bit pricey though.

[Columbia Signs]

I think Doug has it right by keeping a backup at a different location. A backup at the same location as the source in a theft situation is useless.

[iSign]

lots of cool ideas here... thanks for the replies, the testimonials for different products... and for those who have also suffered through some of these difficulties, thanks for helping me not feel like the only loser who just can't always seem to keep up with the difficult challenge of figuring out the best & fastest & most secure ways to deal with the data storage requirements in this industry.

John M. I was just remembering the removable hard drives last night, thinking it might be the best of both worlds when my portability demands are almost zero... except that I will still store one drive off site... and the speed and reliability of moving data between internal drives can still exist...

Also, I haven't taken advantage of the firewire technology much yet, but I have firewire ports on all my computers & the MyBook drives... and I have a cable... guess I should try that if it's faster then USB. It is right?

[John M]

Nah, Firewire and USB 2.0 are pretty much a dead heat in raw specs, with Firewire typically being a little faster in practice. Neither are all that attractive when dealing with large amounts of data.

Your best bet is probably a hot-swappable SATA bay with a few extra "caddies" for the various drives you want to use.

The eSATA setup Fozzy posted is also very good; it includes everything you'd need as a complete kit. At that price, buying a couple to have extra drives isn't too expensive at all. That sure beats trying to piece together the various cables and connectors. Don't forget that you'd also need to buy the drives to go in each enclosure!