Please use this thread to discuss and ask questions about the curation of your digital data.

​

This thread is sorted to "new" so as to see the newest posts.

​

For a subreddit devoted to storage of data, backups, accessing your data over a network etc, please check out /r/DataHoarder.

Hi everyone,

Having searched the sub and read a lot of posts here and in other related subs, I see that there are many ways to approach the mess cleaning process. What I also noticed (I may be wrong, and please correct me) is that there are two main ways to go: folders with files and files with tags (and, of course, a multitude of mixes thereof).

Currently I'm contemplating the Great Cleaning: I've got 15 different HDDs/SSDs with over 20TB data on them, all mixed and messy as you can imagine – folders with subfolders and sub-subfolders, backups of backups and another backup-just-in-case, and full drive dumps before a major OS re-installation, and partial dumps and backups of those, etc., etc. Types of files are also plenty: media (audio, video, photos), docs in many formats (TXT, DOC, Pages), spreadsheets in many formats too, PDFs, etc.

As part of my goal is to sort out photos (most precious part of my entire digital mess), which in itself is another great endeavor, I was thinking of first separating photos from the rest of the pile, and then work with those two large chunks separately. Here I come to understanding that not only photos, but videos too should be in that "photos" pile (I'm not talking about movies (downloaded or ripped), I'm talking about videos I made with my phone or camera to be either a part of home photos/videos library or to be used for a project (like amateur filmmaking).

The other large chunk of data is all the rest – all other files.

So my idea was to employ this workflow:

1. Separate photos and videos from the rest of the mess. Basically, create two large piles – Photos (where photos and videos go) and Docs (for the simplicity to name it this way, where all the rest goes).

2. Dedupe the Docs pile with good deduplicating software (I have Gemini 2 and some other tools – I'm on the Mac).

3. Deal with the Photos pile (not actually a part of this post, so just a step with other steps following).

4. Deal with the Docs pile.

The this #4 is what I'm struggling with. My current "organization" of this kind of data is project-based if I can call it so. For example, I have a folder named "Work_Current" where I keep projects on which I'm currently working. They are also in folders named by project ("Project A", "Project B", etc.). In those folders there are mixed kinds of files – a project may involve documents as word-processing files (DOC, Pages, TXT) or PDFs, spreadsheets (Excel or Numbers) and even Adobe Photoshop or Adobe Illustrator files (PSD or AI), and sometimes even Adobe Premiere or Adobe Aftereffects projects with their respective subfolders (like "Source", "Output", not to mention the self-created Adobe subfolders which sometimes happens).

At first I liked the idea of using tags while having all the files in one big folder. This will involve two steps as I see it: 1) rename files using some naming convention into something like That_Important_Meeting_Notes_[file_metadata (if any can be used)]_date (yyyymmdd).ext); and 2) tagging those files using several tags – for example, a project tag + some other tag. This seems to serve the purpose of easy data retrieval (use a project name or a part of it to get files related to this particular project).

On the other hand, the Decimal system also appeals to me because it seems to be very hierarchically and neatly organized. But again I will have a folder/file structure (though much more organized and slimmed down).

What bothers me in both approaches is that whichever I choose I may end up with not enough tags or folder categories, and this may again bring me to the point when some newer or previously uncategorized files remain in a messy pile, and I will need to re-do all this over again.

The hierarchical folder structure, from another perspective, may (not necessarily, but) save me the hassle of renaming and tagging all the multitude of files (while I don't diminish the usefulness of tags per se even in this scenario), and move the deduplicated Doc pile into corresponding Decimal-based structure. Here, again, as I see it, I will need to very thoughtfully plan the hierarchy very well beforehand.

So, what would you advise as the more appropriate approach in this situation? What I'm actually looking for is to a) clean this mess most effectively and efficiently with view to b) be able to retrieve data easily.

Thank you all for your thoughts, much appreciated in advance.

This is long. Go to the conclusion for the main point if you wish.

Somehow over a decade I ended up with +30,000 images. I always wanted to sort and tag the most significant of them. More scary than that number is the landscape for file tagging applications.

I tried the new darling TagStudio, but to my horror it creates folders in your folders with .json junk instead of tucking away a proprietary database in a undisclosed Windows location (aka AppData/Roaming). No solution is good.

Ignoring those solutions I started using the awkward image sorting tools like Photosift. Those programs suck. They often assign a directory to a keyboard letter so if you have more categories than keyboard buttons you are out of luck and you have to memorize the key-folder combination.

I decided to write my own clumsy sorting tool just to get away from this. It just lists the folders inside a directory, adds to a list and I type the first letter of that list that is the destiny of the current pic. Unlimited categories, no memorization, etc.

Those programs either move or copy the original file. By copying you can have a same item that has multiple meanings in multiple folders, so the folders somewhat act as tags. This is still not perfect. You have multiple copies of the same file wasting disk space and one file is independent of the other copies.

Unless you use hard links! So I modified my sorting tool to do hard link operations. Now this approach somewhat works. But what are hard links?

Hard links are multiple points of entry to the same data on your disk. Unlike shortcuts they 'behave' like the 'original' file instead of the dreadful .ink files. Deduplication tools offer hard linking or synlinking options to save space in your disk without modifying file structures. That's the main advantage of the same file existing in more than one place at the same time.

The result of this mad tagging is 30,000 images sorted into the 5,000 best ones which were then sorted into 150 categories. In this journey most images are 'duplicated' 3 to 5 times across multiple folders without wasting any disk space. The same can be done with folders as symbolic links so I plan to create folder categories, which are in a sense nested tags.

Advantages:

No sidecar files, intrusive folders, hidden databases or junk json files. The folder structure itself act as tags and containers for tags. Any program can interact and modify the structure. No extra disk space is needed.

Disadvantages:

A basic file browser can't do complex operations like searching duplicates across multiple folders. So checking how many tags does a file have (where its copies are) or delete the same image from multiple folders is an inconvenience. The excellent Everything program can help on that but that's still cumbersome to extract the filename and analyze paths. My file sorting program can view the tags for an image but not the images available for a given group of tags. Also every base file must have a distinct name across the whole folder structure. If you backup this without proper caution you are essentially creating a zip bomb.

Conclusion:

By abusing hard links and symlinks it's possible to create a 'clean' tag system just using folders and duplicates but there is no application available to handle this unorthodox approach as a viable solution. The all-in-one solution should be able to create, observe and modify the folder structure without leaving garbage data as legacy but the folder structure itself.

If you want to try to do this yourself I recommend the following programs and using them in that order:

Link Shell Extension (LSE) - to visualize and creation of hard links and symlinks

Advanced Renamer - To give unique names to groups of files

Photosift - for sorting images across subfolders as copies

Alldup - for deduplication of files as hardlinks

Everything - for faster access to individual files

Hi all

Looking for Windows software that will keep two drives synced, basically I use my portable drive when traveling for work when I come home I plug it into my desktop and move stuff over manually.. I want something that when I plug in the portable drive into my desktop it will sync everything to the desktop drive and keep up with any changes on the portable drive I basically want them to be ongoing mirror images of each other.

Hi all -

I have a friend who has come to me for help. She has photos - zillions of them - as well as screenshots, various non-photo image files, documents stored as images (she's a lawyer and has all sorts of discovery received as .jpeg or .tiff). Some photos are in Google "takeouts", some are in Mac Photo Libraries, some are just files in various folders spread throughout the file system, some are email attachments, well, you get the idea. Many of the Mac Photo Libraries have duplicates from other libraries. Long and short, it's basically image vomit.

My task is to organize all this stuff and remove duplicates. She'd like a photo library of her actual photos (i.e. non-document/screenshot/etc) and some sort of means of storing all the other stuff. I'm not really clear on how Photos deals with the actual files so I don't know if something like Gemini can deal with those or not and I'm not sure how to separate the actual photos from the documents stored as images without opening them to review.

Any and all thoughts, ideas, tool suggestions and the like would be greatly appreciated!!

Hello! I'm currently stuck on a Mac, which means programs like MediaMonkey and foobar2000 tend to run into problems due to general lack of support. Parallels is very clunky to use, and so I tried to curate my fairly large music library with Clementine.

However, everytime I try to open Clementine, it just crashes, only giving me a brief glimpse at the program running, before crashing. I send in a report, try to reopen it, and the cycle repeats. I tried installing it through Terminal, but I'm sadly not as experienced in that as I would like, so trying to figure out the proper procedure, versus just opening a DMG file, is quite frustrating.

Hopefully, you guys can offer some help. I've heard a lot of good things about Clementine, and since it's free, it will hopefully be a better option than Audivrana (which has wonky tagging), or Swinsian (which doesn't allow a truly comprehensive way to seperate artists)

I occasionally have tastings for various foods or hot sauces and would like to automate the data collection of the paper survey given to my guests into an csv or similar format that I can then evaluate and improve upon. Since this is a hobby/just for fun initiative, ideally looking for something open source or free that can handle scantron style OCR data collection.

Is anyone familiar with a solution like this? Usually there are ten or so guests, but there can be 50 or so data points depending on the number of sauces or food items being evaluated.

I’m curious about how you all go about organizing your file systems. I’ve been experimenting with different ways to keep my files organized, and I’m eager to hear what works best for you all!

Do you use any scripts or software to sort files automatically, or do you prefer a more manual approach? What tips, tricks, or personal philosophies have you found helpful for keeping everything in order?

Thanks in advance for sharing your methods!

I'm looking to rework my file management system on Mac OS and I have a few questions for people on this sub. I want a hierarchical directory structure, something like roboyoshi's filetree:

1. Where in the MacOS directory structure do I put all of this? In roboyoshi's and others' structures it starts in a directory called "root." At least on my computer, I can't modify folders at the actual root. Do I then start in /Users or /Users/[user]?
2. What about when categories seem to overlap? For example, if I'm doing a personal multimedia project involving music, video, etc.

Thanks!

Has anyone got a spreadsheet of say the top X UDC classifications? I'm starting a PKMS and want to build an classification/tag system for business, technology and science domains. I've got the UDC PDFs from the usual places, but it's less useful than one would imagine because:

1. It's not a text-friendly PDF, i.e. would need further OCR processing to be searchable
2. There's a lot of areas I won't be using, e.g. literature, etc. and I'll probably spend as much time searching for the right location as I will in adding knowledge.

Does anyone know of some already converted electronic version of these classifications? If there are none for UDC, perhaps Dewey?

Please use this thread to discuss and ask questions about the curation of your digital data.

​

This thread is sorted to "new" so as to see the newest posts.

​

For a subreddit devoted to storage of data, backups, accessing your data over a network etc, please check out /r/DataHoarder.

Is there a good "tool" to use to extract some of my favorite thread, favorite writings of my friends there? It's a senior site, having a lot of trouble, and I fear some threads will be gone forever??

I heard of a "scraping tool" but couldn't find one, and if possible, I'd like to have Opensource tool/software. Thank you for any help at all ;)

(Translation is used　　I am not computer savvy)
I am addicted to scanning and collecting various manuals
For many years images were saved as png
Then the folder was a zip folder
Many a little makes a mickle
We had to review it because the capacity had increased too much
I made it into a pdf and it took up a lot less space.
compared to the identical one.
It doesn't look or feel any worse for wear.
Why is it lighter?
When converting to pdf, there is a function in the software called “optimization
This is to reduce the image quality and make it lighter.
However, I am not using this function and the image is still in png format.
strange!
I'm thinking of changing everything to pdf if it makes no difference and only reduces the capacity.
Is there a reason why most of the world uses formats like zip or rar instead of basic pdf?

Hi, I've a problem to be solved and I'm here to ask you for suggestion.

I've a huge quantity of files (~50_000 PDFs) that need to be tagged and I've a fixed structure where store these tags.

something like:

_system VARCHAR(50) NOT NULL,
version VARCHAR(10) NOT NULL,
name VARCHAR(50) NOT NULL,
y YEAR NOT NULL,
language ENUM (...) NOT NULL,
type ENUM (...) NOT NULL,

so some tags are limited (enums) but other are not (strings, numbers).

My question is: "is possible to automate or speed up someway the process ?" because manually process these files will consume hundreds of working hours.

Greetings! I am a data hoarder/curator in my spare time and a compliance engineer by trade. After our last audit I'm starting to dig into the task of curating all of our previous audit responses to help looking up answer for future audits.

To that end I'm looking for a tool or combination of tools that process all 30,000 files (Word, Excel, PDF, TXT and image files) and curate them. Auto-tag them, pull everything into one big searchable database to search on key words & phrases, etc.

As this audit data this would have to stay on prem but in my early searches I've found if I want something that leverages AI for auto-tagging, it isn't on-prem.

Any suggestions are appreciated. Really just trying to wrap my arms around it at this point.

Hey everyone,

I've recently noticed an increase in bot accounts on TikTok posting inappropriate content that promotes OF accounts. However, these accounts don’t seem to get banned, despite violating TikTok’s ToS. After digging into this, I downloaded one of these videos and found something interesting.

When I download the video through TikTok, the frames appear as abstract patterns (like lines over gradient backgrounds). However, when I download the same video externally, it shows the inappropriate content that users are seeing. This leads me to believe that these bots are using a technique where they layer video content, sending one version of the video to TikTok's moderation tools and another version to actual users.

Here’s what I think is happening: The video likely uses layered video encoding, where it has two "layers" or streams—one with harmless frames and another with the actual inappropriate content. It could be manipulating metadata, specifically keyframes and predictive frames, so that TikTok’s AI moderation only detects the innocuous content, while human viewers see the real video. This allows the bots to bypass moderation since TikTok’s AI may be scanning the abstract frames, approving the video, while different frames are shown to users.

• Has anyone seen or experienced something similar with layered video encoding?
• How do these bots achieve this separation between frames seen by TikTok’s moderation system and frames seen by users?
• What tools (FFmpeg, HandBrake, etc.) and techniques might be used to encode videos like this?

Looking forward to your insights on this!

Hey everyone,

I've recently noticed an increase in bot accounts on TikTok posting inappropriate content that promotes OF accounts. However, these accounts don’t seem to get banned, despite violating TikTok’s ToS. After digging into this, I downloaded one of these videos and found something interesting.

When I download the video through TikTok, the frames appear as abstract patterns (like lines over gradient backgrounds). However, when I download the same video externally, it shows the inappropriate content that users are seeing. This leads me to believe that these bots are using a technique where they layer video content, sending one version of the video to TikTok's moderation tools and another version to actual users.

Here’s what I think is happening: The video likely uses layered video encoding, where it has two "layers" or streams—one with harmless frames and another with the actual inappropriate content. It could be manipulating metadata, specifically keyframes and predictive frames, so that TikTok’s AI moderation only detects the innocuous content, while human viewers see the real video. This allows the bots to bypass moderation since TikTok’s AI may be scanning the abstract frames, approving the video, while different frames are shown to users.

• Has anyone seen or experienced something similar with layered video encoding?
• How do these bots achieve this separation between frames seen by TikTok’s moderation system and frames seen by users?
• What tools (FFmpeg, HandBrake, etc.) and techniques might be used to encode videos like this?

Looking forward to your insights on this!

hello, i’m not sure if this is the correct subreddit for this but i’m currently completing a task for school that requires me to create crosswalk transfers from dublin core to mods. i need to convert a sizable amount of dc elements (alongside their respective metadata) but i can’t seem to locate a program that can do this for me—it seems as though i have to manually map each element individually using this guide: https://www.loc.gov/standards/mods/dcsimple-mods.html.

so, i’m pondering these two things: 

1. am i stupid and is there actually an encoding program that does exist but i just can’t find it. i’ve used this program (https://nsteffel.github.io/dublin\_core\_generator/generator\_nq.html) for a past assignment to generate xml from dc elements so surely there should be one for dc2mods?
2. if no such programs exist, does this mean that in professional settings massive collections are all encoded by hand? that seems a bit unreasonable and a bad use of time

for example if i have the dc element “bluebird” as the title, can’t i just input it somewhere so it gives me the mods version "<titleInfo><title>bluebird</titleInfo></title>" without having to manually type it all out?

i apologize if this sounds really asinine, pls be kind. i’m incredibly new to the field of metadata and am still a student

Hello, I have thousands of image files that all follow the same format, and I'd like to extract the data from about 20 fields in the images. I currently have 500 images but anticipate gathering many more. Do you know of any free image OCRs with high accuracy and that allow customization of which fields of pixels on the image to pull from? I'll be compiling all of the data into a CSV and there's too much data to split it myself, which is why it's important I find an OCR where I can specify which pixels on the image to look at for each data point. Thank you in advance!

Hello Guys,

I have a problem that I need to solve. I have a huge file (1.5GB) that is 3000 pages. i need to make it editable/searchable with great precision. What is the best approach for this and how to make it? even if an by using online or cloud tool? The file mostly contains drawings, and reports. Any help is appreciated

Hi,
I recently launched pdfnamer.xyz
A tool that helps you rename your PDF Files according to their content.
I started this project for myself because I hated it to search through PDF Invoices when I was doing my vat tax.
If you download or scan PDFs they have all kinds of naming (invoice.pdf, 2134343223.pdf, etc.), but none was matching my template YYMMDD_Supplier_Topics.pdf (I am a Monk in this regard).
So I created this tool for myself and after a lot of friends and colleagues told me to make it public, I invested some time and created a SaaS around it.
And here we are :)

If you are interested, please check it out. Your feedback is highly welcome!

Regards Christian

Rename your PDFs now: pdfnamer.xyz

I do digital art as a hobby so I have a big folder of projects that need organizing. My workflow is that I create a folder for every drawing that I do. Inside is the PSD file and all the reference images I need plus intermediate output files etc. But the end result of any project is just one image. I would love it if that final image were the thumbnail for my folder so that I can get a birds eye view of my portfolio and search more easily through it. I don't like relying on names because as we know file names are a complicated topic. Plus if I want to show it to someone they can easily get a quick glance.

Ideally I'd like to be able to say "For each folder look for a file called PREVIEW.jpeg and use it as a thumbnail". Just like a README.

Edit: Also if there is a way to set the preview of a each PSD file that would be useful.

I have 500gb+ worth of family photos that my parents keep, they never really sorted anything properly so it's a complete mess, I wanna make it easier to navigate, it's gonna be hard but possible.

So I wanted to ask if there are any good tools or something that can help me/do exactly that? It might be ready hard as many of the extremely old photos are from a digital camera and old 2008 phone.

If I'm gonna do it myself, I seriously have no damn clue how I'll do it.

Please use this thread to discuss and ask questions about the curation of your digital data.

​

This thread is sorted to "new" so as to see the newest posts.

​

For a subreddit devoted to storage of data, backups, accessing your data over a network etc, please check out /r/DataHoarder.

OCR automation software for Windows that can help you batch OCR an entire folder of scanned PDFs. Simply configure any folder in your computer as a magic folder. OCRvision automatically adds an invisible text layer to the scanned PDF document, making it easy to retrieve important information. Try OCRvision today and see how it can streamline your workflow!

https://www.ocrvision.com/

I'm trying to find an image viewing/categorisation program that I used a few years ago. I cannot remember its name. It had an unusual way of presenting the images in the collection (directory): they were all shown as items arranged on an infinite canvas. They were not necessarily arranged in a highly ordered way, but might be shown scattered or in clumps. You could zoom in and out very far. You could sort the photos, which would "clump" them together based on the sorting criteria. You could also manually arrange them. You could drag to select a group of photos, then tag them, move them, etc.

It was obviously not the most efficient way to browse or organise photos but it had its appeal. To emphasise, the "canvas" was the program's metaphor for viewing the photos, not any kind of document that was being created.

Does this ring any bells for anyone?

I am currently in the process of cleaning up all of my different folder systems and consolidating them in a PARAs frame work. I have run into the issue multiple times where I have a folder with files in it (e.g. Planned Projects.md) and I find a file that is named exactly the same but shall be put in the same folder. Now because of bulk moving folders, I don't want to rename each file (even with PowerToys a pain) but just want to be able to drop it in the folder and it gets "renamed" automatically like Windows those when creating copies. I currently run Windows 10 and Windows 11. I am very grateful for any tips, tricks and software recommendations.