/r/DataHoarder

Photograph via snooOG

This is a sub that aims at bringing data hoarders together to share their passion with like minded people.

Who are we?

We are digital librarians. Among us are represented the various reasons to keep data -- legal requirements, competitive requirements, uncertainty of permanence of cloud services, distaste for transmitting your data externally (e.g. government or corporate espionage), cultural and familial archivists, internet collapse preppers, and people who do it themselves so they're sure it's done right. Everyone has their reasons for curating the data they have decided to keep (either forever or For A Damn Long Timetm). Along the way we have sought out like-minded individuals to exchange strategies, war stories, and cautionary tales of failures.

We are one. We are legion. And we're trying really hard not to forget.

-- /u/5-4-3-2-1-bang from this thread


Links!!


Rule(s)

  1. Search the Internet, this subreddit and our wiki before posting.
  2. Keep it about datahoarding.
  3. Be excellent to each other.
  4. No memes or 'look at this old storage medium/connection speed/purchase' (except on Free Post Fridays).
  5. Posts must include context/detail.
  6. No unapproved sale threads, advertisement posts, or giveaways. Companies must get prior approval from mod team before posting.
  7. No cryptocurrency posts.
  8. We are not your personal archival army.
  9. r/techsupport exists.
  10. No requests, use r/DHExchange

Free Post Friday
On Fridays we'll allow posts that don't normally fit in the usual data-hoarding theme, including posts that would usually be removed by rule 4: “No memes or 'look at this [thing]'”
Just make sure to tag the post with the flair [Free-Post Friday!] and give a little background info/context.


Related Subreddits
Data Hoarding/Curation:

Servers and Homelabs:

Tech Support:

Sales & Marketplace:

/r/DataHoarder

784,699 Subscribers

2

Non-Helium HDDs

Looking to upgrade to a non-helium HDD this holiday season. Does anyone have any good recommendations for 18TB-24TB? Thank you in advance!

Edit: Thank you for the replies. I should have added more context.

I recently had an accident with a helium drive. After contacting two separate data recovery companies, they advised that the data is unrecoverable due to the drive being sealed with helium. Now I’m looking for an HDD that has large capacity, but that isn’t sealed with helium.

10 Comments
2024/11/06
21:05 UTC

2

Seeking advice for ~200TB S3 compatible office storage pool

We have been using 2 JBODS to sustain our storage requirements for imagery data at our office. each JBOD is connected to a single server and both have identical copies of the data we need to store.

Each JBOD is USB-C connected and has 8 drives, 4x 12TB and 4x 16TB WD Reds, and we have some extra 8TB drives as well. No raid, storage pool or nothing is set up for either of them.

I have been looking into getting a better setup for our storage, as we would like occasional access from other machines on the network, instead of being limited to working on whatever machine has physical access to one of the JBODs.

We have the following hard requirements for such a server:

  1. It has to be S3 compatible (I believe TrueNAS Core supports this?). Since we are working with satelliteimagery, the individual files may be quite large, but we may only need to read parts of the images. We do not want to transfer the whole file, just to access part of it.
  2. It would be preferable to have a single server managing the storage, it will not be doing anything else. Currently we have a compute node with 128GB ECC ram in a GIGABYTE B550 gaming x MB, with a 5950x, and we could make use of the hardware in this if it makes sense.
  3. The available storage capacity should be able to expand to ~200 TB after parity has been taken into consideration. It would be nice if it is easy to expand the capacity, but worst case we just buy a duplicate server.
  4. The durability is not extremely important, so redundancy/backups are not to be considered currently. the data we have stored is available online, but since we need most of the data available before we can work with it, any downtime due to a subset of the data missing has to be minimized. I read about zfs DRaid to speed up array rebuilds, would that make the most sense for us? Anyway, in the case all the data goes away due to a fire, we can recover.
  5. It should be able to sustain reads from 2-4 concurrent clients, at 100-200 MB/s each (so ~2.5G links seems appropriate?). Writes are not an issue, as the data is static and content is added slowly over time.
  6. ???

Any input would be greatly appreciated!

We are located in Denmark / EU, so we prefer not focusing on US-only vendors.

4 Comments
2024/11/06
20:24 UTC

1

What's the best way to backup Android phone?

I need to backup my phone and transfer all files, call logs, text messages, apps to new phone

3 Comments
2024/11/06
19:40 UTC

16

Climate mirror in 2024?

After Trump‘s victory a group of scientists set out to archive climate data that was in danger of deletion by the Trump administration. Now that he is back, is there any comparable effort to save whatever data might need saving? Climate mirror page seems to still be at 2016.

As for being unpolitical: It might not happen at all, we can’t say. But I am sure I am not the only person who is concerned.

7 Comments
2024/11/06
18:49 UTC

35

Netflix is removing their interactive specials. How would I go about ripping these?

26 Comments
2024/11/06
18:02 UTC

0

HELP - Need to Move iPhone Media with Dates to Hard Drive

Hi guys, recently l've been collecting all of my media from Snapchat and Instagram and whatnot onto my phone so i can finally have all of my media in one place, and I have over 6,000 items. The next step was to take that media and back it up on a hard drive. I also aim to label each item by YMD-HMS so i can see absolutely everything in the exact order I have it saved on my phone as. Unfortunately, l've been thwarted at every stop so far. I plugged my iPhone into my Windows computer and uploaded everything to a folder. I then tried to use Bulk Rename to rename them all to their Date Taken, but the metadata just stopped existing on all of my Snapchat media, around 2,000 things. For some reason, that data also went out the door with around 3,000 other items just from my camera roll. I also have allowed Bulk Rename the permissions for EXIF data, so it's not that. I tried uploading all of my photos to Shutterfly and then downloading them, but the same issue- no data on the date and time, even though that is literally how it is sorted in the site. Useless. I really don't know what to do and am at a loss. This should not be so impossible nor frustrating. I just want to take my perfectly crafted roll and plop it on a hard drive with no differences in the chronological order. If anyone can give me any advice or direction, I would be immensely grateful.

1 Comment
2024/11/06
17:11 UTC

1

External HDD enclosure - safe setup?

I've been reading though this sub, in hopes of figuring out the best solution, but I only got more and more lost. To clarify, I know there are better (and more expensive) 'pro' solutions suggested here, but it's really an overkill for my use case: I simply need a way to connect a few 1-2TB CMR HDDs externally - from time to time- and copy the data from my PC.

That's why I was thinking of 3.5 HDD enclosure. Unfortunately, they have terrible rep, all seem cheap (no more expensive, quality versions?), and filled with terrible reviews... Also, I've read that there are power fluctuations and poor wall wart power supplies with these cases, and I must say I'm actually green in these power supply matters. Makes me wonder - is there a safe way, like a better power supply/cable I can use with these enclosures, to eliminate the risks?

PS: At the very least my SSDs enclosure setup is ordered and hopefully it doesn't have these problems.. since it's not HDD to deal with. For my main external SSD workflow I ordered ANYOYO NVMe Enclosure and for some additional storage and reading files like music I got this cheaper one from Sabrent. Also, bought my own USB cables for them just to be sure. I hope it will be safe... Wonder if anyone here used these enclosures?

5 Comments
2024/11/06
16:34 UTC

2

Suggestions for backup workflow with my RAID

Hi all, long time listener first time caller.

I'm a video editor but also am obviously a hoarder when it comes to saving and backing up old projects, either for clients or personal projects.

I recently consolidated my production drives into a large RAID. It's directly connected and I'm preferring to avoid networked drives. I'm trying to sort out the way involving the least amount of clutter to maintain my backups.

My plan is to use a hard drive dock and use bare hard drives to sort and backup the main production RAID on the regular. Then just store them securely, and disconnected.

Just curious if you have any suggestions or alternatives to this workflow. I probably have 8 TB of archival data, and 12TB and growing of an active project I'll maintain indefinitely.

I know this doesn't cover an offsite backup, just assume I'm doing it for this question. And this is really more about convenience and clutter. I won't have large data dumps of footage too regularly, so manually backing that up is easy and will be part of footage ingestion.

I may have a secondary external drive automatically backed up for the less chunky data with daily turnover (project files, support graphics, etc.). I have a pile of external HDs in enclosures I am tired of existing and needing their own power cables and real estate on a desk.

5 Comments
2024/11/06
15:59 UTC

3

Looking for advice with LTO drive.

Hi all, I'm looking for some advice on setting up a backup solution with an LTO drive (either LTO7 or LTO8). I've found an LTO7 drive from a recycler for around $500, which seems like a good deal. I’m experienced with servers and Linux, but my main machines are Mac-based. I've looked into options like SAS to Thunderbolt or using a PCI-E SAS card with a Thunderbolt adapter, but those are pretty costly.

I need to back up 150TB of data from my QNAP and want to have two sets: one on-site and one off-site. That’s a total of 300TB. The off-site backup would be overseas, so I’m hesitant to use hard drives due to the risk during shipping. That’s why I’m leaning towards LTO tapes.

Does anyone have recommendations for the best setup? Would building a small miniATX machine for the drive make sense, or should I consider a Mac setup with something like YoYotta? I’m tech-savvy, but given the cost involved, I’d rather get input from others who've done this before making a purchase.

Thanks in advance!

3 Comments
2024/11/06
15:02 UTC

0

What do I need to know about using WD My Cloud EX2 in 2024+?

I want to preface that I am adware of the wd hacks and will not be using any of the cloud features and I'm assuming I can disable all the cloud stuff.

I found a mycloud ex2 (no storage) for 15$ and bought it on a whim. I have a spare windows pc as my nas right now so I'm not looking to replace anything. This will be just something to test and play around in with some old hard drives.

I'm trying to find info on using this and results are flooded with posts saying to stop using mycloud. I'm trying to find info on running custom firmwares and not having luck finding substantial guides.

It seems like you can run debian on this though, and maybe some other cfws, however, there's very little info on this. Usually, you'd be able to find guides and youtube tutorials/showcases on stuff like this, but not finding anything.

6 Comments
2024/11/06
14:40 UTC

476

Deleted 15TB worth of stuff and it felt amazing

Like most of us here, I just accumulate stuff because downloading and curating is fun and very addictive.

I have four internal drives and seven externals. About 100TB in total. It got to the point where every single drive was almost full. File explorer in Windows a sea of red. I'd juggled stuff about from one drive to another as much as I possibly could, but there was nowhere left to go. I needed another drive.

Somehow I just couldn't stomach the thought of buying yet another drive. Wasting hundreds of pounds just to add more stuff I don't even use.

I have all these TV shows that I've never watched and almost certainly never will, but it's nice to have the choice, right? I've also often thought why do I have all nine seasons of this extremely common and easy to obtain show that I've never watched a minute of? Same with films. I've got 1,300 of them. I don't watch films, at all. I've watched one film in 2024. But hey, I might one day.

I always thought it would make sense to just keep season 1 of shows and delete the rest, and download them if/when I need them. I have fast internet, usenet, public trackers, private trackers, real debrid. It's so easy to get stuff. But I could never bring myself to do it. I just couldn't. You know how it is.

But one show was taking up 0.7TB on it's own and I've never watched it. I had to do something, so I deleted season 2 onwards. And seeing the difference it made triggered something inside me. I'd broken through the mental barrier and then I couldn't stop. Spent a whole afternoon deleting seasons 2 onwards of almost every easily obtainable show I had. It felt amazing seeing the free space numbers go up and up.

When I was done I had roughly 17TB of free space. File explorer now a sea of blue. One of my drives had almost 6TB free, wtf? It felt amazing, like I'd freed myself from something. Two weeks on I don't regret it one bit and I haven't missed any of the stuff I deleted in any way.

Not sure if this is an advice post or a confessional at this point 😂 This post will probably go down like a lead balloon in here, but seriously - deleting stuff felt so incredibly freeing and now I have tons of space for things that are actually useful and that I might actually want and use!

116 Comments
2024/11/06
13:18 UTC

1

LTO-5 Streamer issues with LTO-4 Tapes

Hi, i recently bought a HP AQ282A LTO-5 SAS streamer with some LTO-4 and 5 tapes. I successfully filled the first LTO-4 tape with about 500GB of data.

But after that, i was unable to write to another LTO-4 tape. I always get input/output errors after about 60-90GB written. I tested about 4-5 different LTO-4 tapes, with the exact same command as i wrote the first tape.

Then i tested it with an LTO-5 tape and it worked without problems.

The command was (within a script):
tar -cv -b 512 -f - $SOURCES | mbuffer -m 6G -L -P 90 | dd of=$TARGET bs=256k iflag=fullblock

DMESG output:

[ 8624.847731] st 0:3:0:0: [st0] Sense Key : Aborted Command [current] 
[ 8624.847744] st 0:3:0:0: [st0] Add. Sense: Data offset error

OS: Linux Mint

I can't find anything about the "Data offset error" online.

Does anyone have an idea why it doesn't work with LTO-4 tapes anymore or what i can do about the error?

1 Comment
2024/11/06
12:46 UTC

2

Which SSD to choose to backup my projects for years ( Reliable & durable brand ) ?

I want to keep my music projects safe for years so i could re use them in future as well , so i am looking for better and durable , reliable ssd 2tb Samsung t7 2tb Sandisk extreme portable 2tb Crucial x9 or x10 2tb

19 Comments
2024/11/06
12:01 UTC

0

Where to watch the Arabic dub for Steven Universe?

Hello,

So I have a friend (aka: someone i met yesterday) who is looking for the Arabic dub for Steven Universe.

According to sources, when Netflix had the rights to stream the first season, the Arabic dub was available (without censorship, but that’s not important for her)

I looked up on Internet Archive and on pirate websites, but they have watermarks (mostly Cartoon Network overlays)

Anyone can help?

1 Comment
2024/11/06
07:01 UTC

249

Accidentally wiped 500GB of mostly deleted YouTube videos.

Well it finally happened to me. I was manually cleaning up an odd temporary file I found in one of my YouTube channel directories that I archive, and I carelessly used the “rm” command without thinking about what it would do.

The command ran for a couple seconds, and there I sat thinking “Oh no, why is it taking so long.”. Then after a quick ls command, the panic set in and I realized I deleted now mostly missing and unretrievable content.

I froze for a minute and thought about the best way to recover, test disk, raid, whatever, and then quickly remembered my mirror drives that my main data drives rsync to once a week. There was about 5 minutes where I carefully double checked all the commands I was entering, and then BAM! Rsync got to work on restoring my deleted directory from my backup drives.

Those backups run once a week via rsync, and have been doing so for about a year now, never once being used or even thought about, until today when I finally REALLY needed them.

All this to say, always have a backup of any data you care about. I know most of us here don’t need another lesson on their importance, but I hope my short story can serve as a lesson on how critical they really are for us in this hobby. I know having a mirror backup saved my data this time, and hopefully it’ll continue to do so in the future.

EDIT: I did recover my data thanks to the backup drives. I was able to use Rsync to copy the deleted channel from a backup that the main drives back up to once a week.

38 Comments
2024/11/06
06:59 UTC

0

Website prevents me from seeing source, copying from the page, and website downloaders aren't working

I wanted to mirror a useful website on my laptop. this appears to not be working. The website prevents me from seeing source, copying from the page, and website downloaders aren't working, mainly winhttrack and cyoteck web copy. Those are the two I've tried so far. I was wondering any way to by-pass this so it could be archived?

7 Comments
2024/11/06
06:58 UTC

0

Broken cheap USB HDD

What is the best course of action to recover a maximum amount of data for those cheap HDD you bought 10 years ago, when I try to manually copy the files in Windows it get stuck pretty badly for a long time, any truely free software that can avoid that pain?

5 Comments
2024/11/06
03:11 UTC

0

is there a resource for detailed DVD information based on ISBN or other identifier ? some animation DVDs i have are interlaced and some progressive... curious if there's a way to know what options are available for archive purposes.

got the question in the title...

wondering if there's any source other than digging into fandom pages, to learn if there's certain DVDs that feature one or the other encoding. i'm interested in upscaling some old cartoons that don't have high-def releases, and starting with progressive sources makes life a lot easier.

thanks :)

8 Comments
2024/11/06
01:28 UTC

0

Can't have too many scrapers.

1 Comment
2024/11/06
00:06 UTC

0

Program for local management of photos?

I don't want cloud services, I don't want to have to import to some library to be able to do the basic meta tagging and organising I want to do. File Explorer is just too basic and crashes if you do too much. Are there any programs like that out there?

2 Comments
2024/11/05
23:43 UTC

0

Which would you choose?

Which of these 2 drives would you add to your Plex server if there were no other options and why?

  1. WD140EDGZ-11B1PA0 14GB sata white label with 11k hours (no bad blocks).

  2. A "new" Manufacturer Recertified WD Ultrastar DC HC530 sata 14GB from serverpartdeals

2 Comments
2024/11/05
23:02 UTC

26

Too many external drives, is DAS the way to go.

My external storage needs are probably around 6-8 TB. I initially thought of a NAS and liked the idea of running a plex server on that. However my home is not network friendly. I tried those powerline ethernet adapters and transfer rates were pathetic.

I'm leaning towards a DAS and I like the offerings from OWC, mainly because the enclosures have extra USB-A ports and a SD card reader, things Apple keeps eliminating, and these enclosures are about the same price as the ones from Terramaster and QNAP. I'm thinking of getting recertified EXOS drives from serverpartdeals for a little over $10/TB.

My use case is photos/videos, time machine backup, and movies to stream via Plex to an AppleTV. I'm currently using a 5TB external drive, USB3.0 and I can edit 4k videos off that. Data transfer rates are around 100 MB/s. With a 7200 drive and thunderbolt 3 connection I'm thinking I would get around 200 MB/s.

And if you come this far, a total newbie question. I haven't decided to just start with JBOD or go RAID1. Everyone says RAID is not a backup. Is that because of the chance the enclosure itself fails? I can't see both drives failing at the same time and I understand for a true backup you need offsite storage. But life comes with risk and I'm willing to risk a fire or some rando robs my house and takes my hard drives. I will periodically backup the real important stuff with my loose external drives.

32 Comments
2024/11/05
22:58 UTC

0

LTO-4 tape drive for playing around with

I already know that this drive is before LTFS which means it's not as easy to use these tapes, I remember my school admin getting very angry over an LTO-3 autoloader and their upgrade to LTO-6 made things a lot easier.

The tape drive is only meant to be as a thing to play around with as I already do have some LTO-3 tapes to use it with and I know that this drive will write to a generation before it, I may put this tape drive to use backing up some photos and videos if I can get it working reliably enough, if not, it will be backing up my retro PC instead.

I also heard that you can only write to a tape once which means you fill it up fully or you waste space, kind of like a single session disc which has to be formatted to be reused and that if you want to make changes or add data, you would have to copy the tape to a hard drive, make changes and put the data back on the tape, is this true? (I can understand the frustration of the school IT admin when he forgets a student directory meaning he has to reformat the tape and redo the backup if it's true, I felt happy for him when he got the new drive)

I also know it needs SAS to interface with the computer and I am working on getting provisions for that to put into my computer.

The tape drive is completely free, they had it in the recycle pile and I called dibs on it first, I don't have it yet because they have to get the drive out of the system that it's in and then update the entry of the server on their database which means I'll get it next Tuesday.

Edit: my cat clicked the AMA button, it's not an AMA, it's meant to be a normal post, sorry

15 Comments
2024/11/05
22:24 UTC

0

best way to backup Synology Nas with large, 6TB Shared Folder?

5 Comments
2024/11/05
22:13 UTC

0

Question regarding gallery-dl and filename format

Hi there,

I am using gallery-dl to fetch a reddit post. With `-K` I see that the tag `author` is available. But if I use the option `-f '{author}_{title[:220]}.{extension}'` and the post contains a link to a third party, e.g. imgur, then gallery-dl is not using the metadata available for the reddit post and I get something like `None_None.xyz`.

Is it possible to tell gallery-dl to use the metadata from the reddit post and not the third party site?

Thank you for any hints!

0 Comments
2024/11/04
23:04 UTC

0

Short, Non-Reviewer, UnifyDrive UT2 Review

I am by no means a tech reviewer and this is actually my very first review of a product. Hope it comes across as very unbiased and helpful to some.

First disclaimer, I received the UnifyDrive UT2 free of charge, with no expectations in return, it is currently in a Kickstarter campaign with a few days left. I promised u/UDPT (support for Unifydrive) that I would use the unit and give my honest feedback on it. I took the UT2 for a real-world spin to see how it would fit into my workflow. First impressions? This device packs a ton into a small, rugged package, about the size of a thick smartphone with a rubber cover that gives it a durable feel. It’s loaded with connectivity options — USB-C, Ethernet, HDMI, and SD card slots — so it seemed ready for just about any backup scenario​​​​.

The SD card backup feature quickly became my favorite. After a photo shoot, I just tapped the PlugBackup button twice, and it smoothly copied my files over. When connected directly, the transfer speeds were impressive, making it easy to handle high-res RAW files without a hitch. But the wireless side of things didn’t measure up. The Wi-Fi connectivity was hit or miss. Had an easier time connecting to it's hotspot feature rather than my wireless network which was frustrating given the UT2’s promise of portability and wireless functionality​​​​.

The mobile app while technically functional is clunky and often feels more like a beta version. The app felt essential to get the most out of the UT2, yet not reliable enough for smooth day-to-day use. Overall, the UT2 still shines as a storage solution, especially when I need to back up images on the fly. If UnifyDrive polishes the software, this could become a powerful go-to for creators who need storage versatility​​​​.

0 Comments
2024/11/05
00:29 UTC

0

My Great Big Archive

0 Comments
2024/11/05
01:33 UTC

Back To Top