/r/DataHoarder

Photograph via snooOG

This is a sub that aims at bringing data hoarders together to share their passion with like minded people.

Who are we?

We are digital librarians. Among us are represented the various reasons to keep data -- legal requirements, competitive requirements, uncertainty of permanence of cloud services, distaste for transmitting your data externally (e.g. government or corporate espionage), cultural and familial archivists, internet collapse preppers, and people who do it themselves so they're sure it's done right. Everyone has their reasons for curating the data they have decided to keep (either forever or For A Damn Long Timetm). Along the way we have sought out like-minded individuals to exchange strategies, war stories, and cautionary tales of failures.

We are one. We are legion. And we're trying really hard not to forget.

-- /u/5-4-3-2-1-bang from this thread


Links!!


Rule(s)

  1. Search the Internet, this subreddit and our wiki before posting.
  2. Keep it about datahoarding.
  3. Be excellent to each other.
  4. No memes or 'look at this old storage medium/connection speed/purchase' (except on Free Post Fridays).
  5. Posts must include context/detail.
  6. No unapproved sale threads, advertisement posts, or giveaways. Companies must get prior approval from mod team before posting.
  7. No cryptocurrency posts.
  8. We are not your personal archival army.
  9. r/techsupport exists.
  10. No requests, use r/DHExchange

Free Post Friday
On Fridays we'll allow posts that don't normally fit in the usual data-hoarding theme, including posts that would usually be removed by rule 4: “No memes or 'look at this [thing]'”
Just make sure to tag the post with the flair [Free-Post Friday!] and give a little background info/context.


Related Subreddits
Data Hoarding/Curation:

Servers and Homelabs:

Tech Support:

Sales & Marketplace:

/r/DataHoarder

783,360 Subscribers

2

y'all think 15,168,433,411 write errors on a cache drive is problematic or nah ?

2 Comments
2024/11/01
01:40 UTC

2

Planning to use a dvd duplicator case to hold drives for ripping but can't figure up how to connect them to pc.

I've been trying to find a card that will fit in the back of the case to connect all of the sata ports to 1 usb 3.0 that way it can move data. Any advice?

3 Comments
2024/11/01
01:08 UTC

1

WinX DVD Ripper?

I got WinX DVD ripper for professional use of content we’d already purchased licensing rights to.

The licensing agency could only offer DVD or streaming options, but what I really would prefer for what I’m working with is a .mov file. I started using WinX and after several hours on the only computer I have with a disc drive (old and slow) it seems like it’s stuck downloading half the film and then repeating scenes. So the file is the correct length, but the second half is just repeated footage on loop until the time hits.

Has anyone encountered this? Is there a way to correct it? I’ve downloaded 18 titles from this DVD, 16 of which are just clips off the menu and bonus content. I’m sooo close to having the whole film converted, but I can’t seem to get past this last hump of a second half. Any advice super appreciated!

3 Comments
2024/11/01
00:17 UTC

0

Flippingbook download methods not working

Trying to download the contents of a flippingbook (https://online.flippingbook.com/view/591563722/) and methods mentioned on reddit as well as the GitHub tool aren't working. Tried chromes override content and no change in the webpage. Downthemall couldn't see the content.

1 Comment
2024/10/31
23:42 UTC

19

[Friday Funny] Abomination which should not be used to store any data

10 Comments
2024/10/31
23:32 UTC

1

Help! I don't know what I need.

Hey everyone,
long time lurker, first time poster around here.

Recently a friend of mine lost about 1.5TB of data on his only external HDD due to a Windows function (I don't know the name as I use Windows sometimes but mostly MacOS). I currently have no physical storage but I've been gathering a lot of documents, scientific papers and reports. It's mostly .pdfs but I got some videos/audio. I have all this media on iCloud but cloud services are expensive and are subject to the goodwill of the provider.

With all this in mind, of course my first thought was to get a cool looking NAS with a shitload of HDD but unfortunately money is limited and so is physical space. I've been looking into pros and cons of NAS and budget options.
Ultimately, as per my research, I don't know if I NEED a NAS as much as I want a NAS. I have no need to keep it network attached as I mostly use one Macbook for my work and occasionally an iPad and a Windows 11 desktop, all three with USB-C/Thunderbolt ports.
Maybe an external 2 or 4 TB NVMe ssd with a USB-C port would do the trick? Maybe I should buy two and backup the main one every month or so?

I'm sure you're all way more experienced than I am and will give me some ideas on how to work from here. My idea is to have an external drive (SSD, HDD, NVMe) to dump all my media and in the future one of the same capacity to backup the main one.
As per my research solid state is less prone to problems while more expensive per TB, but as my "hoard" is small (for now) maybe this solution will work.

Anyway thank you a lot in advance and any advice is welcome.

4 Comments
2024/10/31
22:45 UTC

3

PCI x16 4xNVME card

Has anyone used one? How reliable are they?

I want to get one to add 4 1TB NVME drives in a RAID-5 to my desktop. I have verified that my motherboard supports bifurcation. I am just weary, they seem like a "too good to be true" tech item.

14 Comments
2024/10/31
21:52 UTC

3

How to re-encode images that are up to 100,000 pixels in height?

I have an app which screenshots websites and saves them as .png, then I use ffmpeg to re-encode it into AV1 using libaom-av1 so they maintain more or less the same quality (for my purposes at least) while taking up only 10-20% as much storage.

However there's some images which are up to 100k pixels in height, and the max I seem to be able to process is around 50-60k pixel height. All images have a width of 1920px.

I've been looking for a solution to this but can't seem to find one, any ideas?

4 Comments
2024/10/31
20:31 UTC

1

Cleaning up handful of .tar archives

After many years I finally implemented a reasonable backup strategy. Before it just was based on bundling the most important stuff + some completely unnecessary system related files in a tar, and dragging it along my data journey.

I already cleaned up two of the largest tar archives manually (both ~200GB und now combined to 60GB). There was a lot overhead and stuff I already backed up anyway, with only small differences. Also there are often nested .tar files because of the dragging it along behind me.

What would you do in order to cleanup everything? I still have 6 archives remaining with a total nearly 500GB. I have space und uncompress everything, but manually handling it feels just like a task you could automate.

7 Comments
2024/10/31
20:20 UTC

0

MD1200 Problems

I've used a MD1200 DAS shelf attached to a self built server for many years. Yesterday its started acting up.

5am i got woken by one of the kids, saying there was a "ghost noise" coming from their wardrobe... Turned out the noise was the MD1200 ramping its fans to 100% and back down again at random. Its so loud you could hear it thru the floor from the basement...

I gave it a bit of a poke, pulled each PSU in turn at which point the fans ramped down but to a middling speed, rather than the usual 20% setting that i apply over serial. At least not 100%! I fiddled with a few other bits but it was not responding to speed changes on the console at all. It usually runs with only one PSU powered up, so i switched power to the other which didnt seem to help. I left it be. A few hours later it ramped the fans up again, and more hotswapping the PSUs again fixed it, but still not responding to fan speed.

After all that messing about, i decided this morning to power it fully down and swap controllers with my spare. Did that got it back up and running on one PSU, fan control was working again. Kinda hoped it was fixed.

This evening i noticed my CCTV server was broken and eventually realised all the VM's were down. syslogs were full of crazy io errors, some head scratching later i realised the disk shelf was completely dead. PSU lights all green but the actual unit was just off. Switched the PSU's off and back on and it came back to life.

However as yesterday, i again have no fan control, and if it was going to just sit at the default speed then fine, but instead its back ramping the fans up to 100% at random...

So back to square one!

Anyone seen anything like this before? Clearly theres a hardware fault of some sort, but i'm not sure if i should try to swap out the power supplies, or maybe just call it quits on the whole thing and try to find something else.

1 Comment
2024/10/31
20:09 UTC

0

New to hoarding… need advice

Hello! I’m new to this space and would like to start setting up a backup system for my gaming PC.

I’ve been reading plenty of stuff online, but the more I read the more I seem to be confused. So im just gonna give a brief description of what I need, and hope that someone may provide advice on what I should get and be doing:

I have a gaming PC that contains all my gameplay videos, family photos, music, etc. basically everything entertainment is on this thing. What I need is to be able to backup this system and these files in the case any SSDs or the system just breaks or fails. I will need to access them quite frequently on at least a weekly basis. I was hoping to do something like setup a NAS(?) and just have at least two identical copies of this backup.

I hope I made my needs clear. I thank anyone who provides insight into making this work. Your help will be immensely appreciated!

5 Comments
2024/10/31
20:00 UTC

0

Digikam or something else?

Had a little mishap where my iPhone got a complete wipe and reset a few days ago. I didn't lose anything, but it did make me realize how much I relied on my iPhone for sorting and searching my photos. Now that they're only on my computer, well... I'm looking for a good way to actually look at them lol. I'm thinking Digikam will do all the things I need it to, but it's kinda a massive program and I'm a bit overwhelmed. So I wanted to see if it was actually my best option before I dive it too deep.

What I need:

  • Ability to handle over 40k photos
  • View photos on map
  • Edit/update geotagging data and save to photo
  • Tags (name, event, subject) that save to photo
  • Update incorrect or missing timestamps and save to photo

I think one of the most important aspects is that I want the tags and updates to save to the photo so those things can be seen/used by other programs. I want to be able to pull up every photo taken at a certain location. I want to be able to tag photos of my pets with tags like "My Pets" "Dogs" "Cats" "Otto" "Calvin" "Rosa", and be able to pull up photos of all of my pets, all of my cats but not the dogs, or just pictures of my cat Calvin.

If it matters, my photos are currently sorted into folders by date, So "2024/2024-04/2024-04-23 11.05.14" is the path for a photo taken at 11:05 on April 23rd of this year. Also, most of my photos are saved as HEIC format.

My biggest concern so far with Digikam is how long it seems to take to index and update the large number of photos I have.

1 Comment
2024/10/31
19:49 UTC

0

Looking for efficient data storage (power and cost) doesn't need to be "super" reliable

I know how it goes you can't have it all unless you spend the coin! I have a small homelab setup - basically a NUC running proxmox on a built in NVME. I have all my media and content on two old external usb hard drives 1 TB and 5 TB, the ol western digital my books or whatever. These are old - and haven't been treated the best. I want to upgrade to something A) larger and B) reliable enough. Nothing on the drives will be irreplaceable but definitely irritating to replace. I was looking at DAS type units as an improvement in the total storage - (Yottamaster type units) however have read various things about the unreliability - performance wise I dont need raid - mostly just lots of storage. Does anyone here have any ideas on what might be a good route - and/or to what degree the reliability of DAS type storage is (like monthly issues, yearly issues, maybe once a decade a big issue). Or a different cost effective option. I think optimally I would want ~16 TB storage (so maybe 2 x 8TB drives, but would like the opportunity to double it in the next few years.

tldr: looking at a 4 or 5 bay DAS - want access to BIG cheap storage on my Proxmux NUC

3 Comments
2024/10/31
19:48 UTC

0

What sofware to copy specific files?

I'm new to this backup data. Is there free software to clone specific files from one HDD to another? I'm currently using Windows 10.

8 Comments
2024/10/31
19:44 UTC

0

Looking to put home movies into 3 conversions on my NAS. What 3 would you choose?

So I have various home movies made from vhs, CD/DVD's (some I aquired from my two dead parents), and many from SD card that I have saved on my various harddrives and such. I have my NAS set up with 4 bays. Thinking of using 2 of them and the other two as back up. Advise?

Main question because we never know what the future will hold in replaying our old movies and pictures. I have been saving in .mov, .avi, .mp4

I have the option to pick others like MKV, M4V, HEVC MP4 or HEVC MKV

I use macbook pro laptop and using wondershare conversion.

6 Comments
2024/10/31
19:36 UTC

0

LSI SAS 9207-8e I/O error

Hello, I recently got a quantum superloader 3 tape system for extremely cheap too experiment with and picked up a 9207-8e card and cable too interface with it. In windows I cant seem too get the card too work properly after installing the driver from Broadcom, it just gives me error 10 I/O error. Does anyone have any recommendations?

EDIT: The card also doesnt seem too show up during the boot process and from what I have read it should be.

EDIT2: I think I may have been sold a dead/dieing card. contacting seller, thanks for the help guys.

5 Comments
2024/10/31
18:09 UTC

1

Downloading Age Restricted Youtube Vids

I like to keep a backup of all my favorite YouTube series and documentaries

but I've notice that a few of them are age restricted and none of my normal download methods work

ByClick, Seal, Firefox Extensions, every website i can find

some claim the videos are private, which they aren't, some can't find the video, and others just get hung up looking

is there any known programs or sites or other browser extensions that actually work?

1 Comment
2024/10/30
17:49 UTC

1

SyncBackPro: Is it possible to password-protect Ransomeware option

Hello all,

I have a question about the Ransomeware Detection option in SyncBackPro. On v11 it's located at
Burger Menu -> Global Settings -> Randomeware Detection
This feature allows a specific file (User or SBP-created) to be used to detect ransomeware changes. If the hash of this file is modified, then SBC assumes that a ransomeware attack has occured.

Afaik, this feature isn't password-protected, meaning that a hacker who gains access to the system and observes that SBP is installed, can simply remove the RD before encrypting everything on the hard disk. I know that individual profiles can be password-protection, bu this is a separate issue.

TL;DR. I've just noticed that each profile also has a RD option available. Is this the way to go? What's the point of the global RD option in this case?

0 Comments
2024/10/30
19:12 UTC

0

20TB easystore

Hello, these are on sale (https://www.bestbuy.com/site/wd-easystore-20tb-external-usb-3-0-hard-drive-black/6500985.p?skuId=6500985) currently. Looking to plan a trip to Paris soon. Are they more expensive there? Tyring to offset cost of trip by taking 1 or 2.

1 Comment
2024/10/30
21:59 UTC

1

New 8TB Seagate Exos knocking the whole case. Why? How to fix?

I let it warm up after unpackaging for an hour as recommended. I noticed the knocking now as I copy the files from the to-be replaced 1TB WD-Purple to it.

EDIT: It was running on 50C according to HDSentinel

0 Comments
2024/10/31
00:16 UTC

0

Which 12TB to 24TB External Drives Contain 7200RPM Drives Capable of 250mb/s?

I have some 8TB WD_Black D10 external drives that I use for media, backups and data transfers. I am told these contain HC320 drives, which are 7200RPM. I am getting sustained transfer speeds of 250 mb/s with these drives, which saves me a lot of time versus my 5400RPM drives, but I need higher storage capacity.

Hoping someday to go SSD for blazing speeds, but until the prices come down for higher capacity SSD, that’s not an option, so 250 mb/s will have to do for now.

 So I looked for higher capacity 12TB to 24TB external drives, and can’t find any that are 7200RPM. I searched the threads here and some folks say the 12TB to 24TB Easystore drives contain EDGZ drives (which I’m told are same as HC560 etc) and spin at 7200RPM. In theory, if they both contain the same type drives, I figured the Easystore should match the data transfer speeds of my D10 drives.

 So I ordered a 20TB Easystore, BUT it maxed out at @ 150 mb/s, whereas my D10’s are at 250 mb/s.

 Do I need to change settings or do something to get it to perform at 250 mb/s like my D10s?

 Shouldn’t it equal the transfer speeds of my D10s?

 If it doesn’t, is there another high capacity 7200RPM external drive that would equal the speed of the D10s?

 Thanks for your help!

2 Comments
2024/10/31
01:30 UTC

1

How loud really are exos drives?

I'm looking at picking up two 18tb recertified exos drives from serverpartdeals and running them in raid 1 for my NAS, but I've heard a lot of complaints about the noise. How loud are they really? If I put them in my NAS case in my networking rack that's next to my bed/desk would I be able to hear them? Or are they not that loud

1 Comment
2024/10/31
02:10 UTC

0

How to back up cloud server with the OS version and applications in one go

Hi everyone, I'm curious how to back up cloud server with the OS version and applications in one so that when something happens we can restore everything quickly.

0 Comments
2024/10/31
04:54 UTC

4

Jonsbo N5 details for those curious

Jonsbo N5 details for those curious

Had a tough time finding the specs I wanted, so I said fuck it and built it.

For anyone curious, while tight, a 280mm AIO does fit in the case but it does NOT allow for 3 120mm fans on each side, at least that I’ve figured out.

EATX board fit well, though it is tight.

In terms of airflow, zero concerns with the hard drives, mine are cooking on Unraid at 25-30C just like they were before.

I am in fact a crazy bastard that threw a 14900k in this thing, the 280MM is handling it well so far, cinebench pushed 80-85C max at peak.

Going to be messing with the fan positioning a bit…whether that is changing the top fans to intake as well and doing exhaust on the top, or tweaking PWN settings, but overall this a solid, solid case.

0 Comments
2024/10/31
05:56 UTC

0

Google Maps Photo Scraping?

Is this possible? Say I want to download a lot of photos of one place, uploaded to Google Maps. Would there be any way to automate this? Apparently it violates some Terms of Use of Google Maps though...

Would still love to know.

0 Comments
2024/10/31
07:05 UTC

0

MG09ACA18TE Hard Drive Structure

Hello everybody,

any chance any one in here can get me the drives geometrie data like cylinders, heads, sectors and sectors per byte. Would really apreciate it and big thanks in advance!

Greetz from Germany

0 Comments
2024/10/31
11:56 UTC

455

I think i am done....for now.

After using several commercial NAS I decided to build my own server with Truenas Scale. The mini itx N100 boards and the freedom of the operating system couldn't leave me happier. 5 X 16 Tb each. Really stable storage. And low consumption.

116 Comments
2024/10/31
13:21 UTC

0

I need advice on multiple video compression

Hi guys I'm fairly new to data compression and I have a collection of old videos I'd like to compress down to a manageable size (163 files, 81GB in total) I've tried zipping it but it doesn't make much of a difference and I've tried searching for solutions online which tells me to download software for compressing video but I can't really tell the difference from good ones and the scam sites....

Can you please recommend a good program that can compress multiple videos at once.

13 Comments
2024/10/31
15:26 UTC

1

Is there a limit to how many hard drives Snapraid can handle?

Is there a limit to how many hard drives Snapraid can handle? My company is considering getting 4 DAS daisychained to host a lot of archive data. It would be hundreds of hard drives.

0 Comments
2024/10/30
22:37 UTC

0

How can I find videos for a channel I used to watch?

Basically this channel would upload archived videos of some political commentary things, and it's been a while since I visited the channel only to be surprised that no videos exist on the channel for some reason. Does anyone know how to rediscover these videos using any kind of archive helper? I couldn't find the URL

Here is the channel name hopefully there can be some help.

Thanks folks.

9 Comments
2024/10/31
14:34 UTC

Back To Top