LinuxQuestions.org
Help answer threads with 0 replies.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices


Reply
  Search this Thread
Old 03-17-2025, 04:54 AM   #1
Julius_Niezufus
Member
 
Registered: Feb 2025
Distribution: Manjaro
Posts: 47

Rep: Reputation: 0
Need info on ext4 for data rescue


[Log in to get rid of this advertisement]
I accidentally installed a Linux distribution on my encrypted 8 TB HDD.

I basically have a backup of that drive, but in an attempt to save the files/changes to files I added after my last backup, i chose not to just resort to that backup, but to actually overwrite the accidentally overwritten block with the corresponding blocks from my backup, as i presume, the changes I made are to be found at the end of the used disk space and didn't get overwritten. Now the question is: How much data do i have to write back to my disk, aka. how much disk space got corrupted?

The files of the accidentally installed distro had a total size a bit more than 500 MB + 100 MB EFI partition. The file system is ext4.

So here are my questions:
How much overhead is used by ext4 for things like GDT, block bitmaps, inode tables, etc?
Anything else, which needs some overhead space?
Does ext4 write anything into the last blocks of the disk?
Anything else, i need to consider?

Additional question added:
Is my assumption right, that when installing a new distro, all content is written from the start of the disk on, and all blocks after that remain untouched?
I hope the files aren't randomly spread all over the place.

Any info is appreciated

Last edited by Julius_Niezufus; 03-17-2025 at 07:32 AM.
 
Old 03-17-2025, 07:18 AM   #2
jefro
Moderator
 
Registered: Mar 2008
Posts: 22,361

Rep: Reputation: 3692Reputation: 3692Reputation: 3692Reputation: 3692Reputation: 3692Reputation: 3692Reputation: 3692Reputation: 3692Reputation: 3692Reputation: 3692Reputation: 3692
Use something like testdisk photorec. Any attempt to play with live disk can further damage data

A dd clone first might help,
 
Old 03-17-2025, 07:23 AM   #3
Julius_Niezufus
Member
 
Registered: Feb 2025
Distribution: Manjaro
Posts: 47

Original Poster
Rep: Reputation: 0
I'm aware of this, thank you, but that doesnt answer my questions.

Last edited by Julius_Niezufus; 03-17-2025 at 07:25 AM.
 
Old 03-17-2025, 08:03 AM   #4
rknichols
Senior Member
 
Registered: Aug 2009
Distribution: Rocky Linux
Posts: 4,820

Rep: Reputation: 2240Reputation: 2240Reputation: 2240Reputation: 2240Reputation: 2240Reputation: 2240Reputation: 2240Reputation: 2240Reputation: 2240Reputation: 2240Reputation: 2240
Quote:
Originally Posted by Julius_Niezufus View Post
I accidentally installed a Linux distribution on my encrypted 8 TB HDD.

I basically have a backup of that drive, but in an attempt to save the files/changes to files I added after my last backup
...
Additional question added:
Is my assumption right, that when installing a new distro, all content is written from the start of the disk on, and all blocks after that remain untouched?
I hope the files aren't randomly spread all over the place.
They are.

Furthermore, the LUKS header for your encrypted drive will almost certainly have been overwritten. That makes recovery of the encrypted content impossible unless you have a backup of that LUKS header. If your backup is a bit-for-bit image of the drive (i.e., not a copy of the files), then you could use the LUKS header from that backup to allow decryption of the overwritten drive. I might be able to help with the procedure for doing that. Otherwise, recovery of that encrypted information is impossible.
 
Old 03-17-2025, 08:29 AM   #5
Julius_Niezufus
Member
 
Registered: Feb 2025
Distribution: Manjaro
Posts: 47

Original Poster
Rep: Reputation: 0
Quote:
They are.
Does that refer to the question whether files are placed randomly on a disk instead of consecutively?
Just for clarification, we're not talking about an SSD but an HDD.

Quote:
Furthermore, the LUKS header for your encrypted drive will almost certainly have been overwritten. That makes recovery of the encrypted content impossible unless you have a backup of that LUKS header.
Yes, I do have a backup of the VeraCrypt header.
Quote:
If your backup is a bit-for-bit image of the drive
Yes, it is, except for the changes and new files I've added since its creation. But I assume these changes to reside at the end of the disk, so I expect these changes to be unaffected. (Am I really wrong about this?)
Quote:
I might be able to help with the procedure for doing that.
YES, PLEASE!!!!!

Oh, here's another thing that's bugging me:

I've taken a look at the corrupted drive in a hex-editor and what I've seen heavily discomforts me:
I'd have expected to see either a large junk of gibberish (where my data is supposed to be stored) or large parts of zeros (where no data resides)
What I see instead is several MBs of gibberish followed by several MBs of zeroes. Now this pattern seems to repeat over the whole disk. I havent yet found either several TBs of gibberish or zeroes. Is the reason for that within the behavior of VeraCrypt or the underlying encrypted NTFS file system? I'm pretty sure it cannot originate from the accidental overwrite, because this would have taken hours or even days.

Last edited by Julius_Niezufus; 03-17-2025 at 08:37 AM. Reason: Additional questions
 
Old 03-17-2025, 11:00 AM   #6
rknichols
Senior Member
 
Registered: Aug 2009
Distribution: Rocky Linux
Posts: 4,820

Rep: Reputation: 2240Reputation: 2240Reputation: 2240Reputation: 2240Reputation: 2240Reputation: 2240Reputation: 2240Reputation: 2240Reputation: 2240Reputation: 2240Reputation: 2240
Placement of files is different in NTFS vs. ext4. NTFS does initially fill space from the beginning of the disk, but new or changed content tends to go into the first available free space, which could be anywhere in the disk that space has been made available due to files having been deleted. In ext4, the entire space is divided into block groups, and files are scattered among the block groups, with an attempt to keep related files (same directory) together. Your changed files could be anywhere within the maximum space that the NTFS filesystem used.

Since you initially mentioned only ext4, I assumed that your original encryption was using LUKS. Now you mention "VeraCrypt or the underlying encrypted NTFS file system." So, I am now guessing that you originally had an NTFS filesystem encrypted by VeraCrypt and overwrote it with an ext4 filesystem encrypted with LUKS. Is that correct? What I would expect to see now is many megabytes of gibberish where the original NTFS data resided (now partially overwritten by ext4 data) followed by a pattern of blocks of gibberish and blocks of zeros where the NTFS filesystem had free space that had never been used, and that region is now partially overwritten by data at the start of each ext4 block group.

I have no experience with VeraCrypt, unfortunately.

Last edited by rknichols; 03-17-2025 at 11:03 AM. Reason: no experience comment
 
Old 03-17-2025, 11:34 AM   #7
Julius_Niezufus
Member
 
Registered: Feb 2025
Distribution: Manjaro
Posts: 47

Original Poster
Rep: Reputation: 0
Quote:
So, I am now guessing that you originally had an NTFS filesystem encrypted by VeraCrypt and overwrote it with an ext4 filesystem encrypted with LUKS. Is that correct?
Correct.
Quote:
Your changed files could be anywhere within the maximum space that the NTFS filesystem used.
I rarely delete files from that disk, therefore I expect those changes to be found at the end of the used space with very few exceptions.
Quote:
In ext4, the entire space is divided into block groups, and files are scattered among the block groups,
Are these block groups at least created only when used?
Nightmares coming true...
 
Old 03-17-2025, 12:21 PM   #8
jailbait
LQ Guru
 
Registered: Feb 2003
Location: Virginia, USA
Distribution: Debian 12
Posts: 8,387

Rep: Reputation: 581Reputation: 581Reputation: 581Reputation: 581Reputation: 581Reputation: 581
The following web page has a verbal description of ext4 layout starting at section "Layout". The web page has a graphic layout of an exp file system starting at section "Graphical view of Disk Layout". The layout is created when you format an ext4 partition.


https://blogs.oracle.com/linux/post/...-layout-part-1

Last edited by jailbait; 03-17-2025 at 12:23 PM.
 
1 members found this post helpful.
Old 03-17-2025, 01:07 PM   #9
Julius_Niezufus
Member
 
Registered: Feb 2025
Distribution: Manjaro
Posts: 47

Original Poster
Rep: Reputation: 0
Quote:
The layout is created when you format an ext4 partition.
So, the installer has in fact already written stuff all accross the whole 8 TB???
Wouldn't that take hours on such a large drive?
 
Old 03-17-2025, 01:17 PM   #10
michaelk
Moderator
 
Registered: Aug 2002
Posts: 26,800

Rep: Reputation: 6333Reputation: 6333Reputation: 6333Reputation: 6333Reputation: 6333Reputation: 6333Reputation: 6333Reputation: 6333Reputation: 6333Reputation: 6333Reputation: 6333
To make matters worse metadata backup superblocks are written throughout the disk as well as the backup GPT at the end of the disk which may have changed. I don't think you can guarantee the disk is written consecutively from the start during the install. ext4 primary metadata is on the order of ~2% of the total size of the partition. Did the installer create partitions(s) to fill the entire disk? Do you have a clone of the disk?
 
Old 03-17-2025, 01:27 PM   #11
Julius_Niezufus
Member
 
Registered: Feb 2025
Distribution: Manjaro
Posts: 47

Original Poster
Rep: Reputation: 0
Quote:
metadata backup superblocks are written throughout the disk
how large are they?
Quote:
backup GPT at the end of the disk
Yes, I found that one, but I dont expect that to be a big deal, I was far away from using the full disk space, not even the hidden volume (which is usually stored at the end of the disk) was full.
Quote:
Did the installer create partitions(s) to fill the entire disk?
None except the EFI partition.
Quote:
Do you have a clone of the disk?
Yes, but that clone was made 3 weeks ago. That means some new stuff won't be on that clone. Otherwise I wouldn't bother.
 
Old 03-17-2025, 02:05 PM   #12
rknichols
Senior Member
 
Registered: Aug 2009
Distribution: Rocky Linux
Posts: 4,820

Rep: Reputation: 2240Reputation: 2240Reputation: 2240Reputation: 2240Reputation: 2240Reputation: 2240Reputation: 2240Reputation: 2240Reputation: 2240Reputation: 2240Reputation: 2240
Quote:
Originally Posted by michaelk View Post
I don't think you can guarantee the disk is written consecutively from the start during the install.
In fact, you can guarantee that it is not. ext4 tries to keep usage of the block groups balanced.

Just to confirm, I created a 100GB ext4 filesystem, mounted it on /mnt/tmp, and ran the following:
Code:
for N in {101..125}; do
    mkdir /mnt/tmp/Dir$N
    cp /etc/cups/ppd/* /mnt/tmp/Dir$N
done
That creates 125 directories on the new filesystem and populates each directory with copies of the 7 files that my system has in /etc/cups/ppd. That uses less than 1% of the space on the new filesystem. Running "ls -iR /mnt/tmp" reveals (by the inode numbers) that the directories are spread around in the ext4 block groups. In my case, the lowest and highest inode nummbers used were 131073 and 6422536, which range from the beginning to the end of the filesystem (6553600 total inodes, 8192 inodes per block group). As expected, the files within each directory were kept together (consecutive inode numbers).
Code:
/mnt/tmp/D101:
6291458 Brother.ppd
6291459 Brother.ppd.O
6291460 HP_OfficeJet_Pro_8710.ppd
6291461 HP.ppd
6291462 HP.ppd.O
6291463 Panasonic.ppd
6291464 Panasonic.ppd.O

/mnt/tmp/D102:
131074 Brother.ppd
131075 Brother.ppd.O
131076 HP_OfficeJet_Pro_8710.ppd
131077 HP.ppd
131078 HP.ppd.O
131079 Panasonic.ppd
131080 Panasonic.ppd.O
...
/mnt/tmp/D124:
1835010 Brother.ppd
1835011 Brother.ppd.O
1835012 HP_OfficeJet_Pro_8710.ppd
1835013 HP.ppd
1835014 HP.ppd.O
1835015 Panasonic.ppd
1835016 Panasonic.ppd.O

/mnt/tmp/D125:
6160386 Brother.ppd
6160387 Brother.ppd.O
6160388 HP_OfficeJet_Pro_8710.ppd
6160389 HP.ppd
6160390 HP.ppd.O
6160391 Panasonic.ppd
6160392 Panasonic.ppd.O
 
Old 03-17-2025, 02:08 PM   #13
Julius_Niezufus
Member
 
Registered: Feb 2025
Distribution: Manjaro
Posts: 47

Original Poster
Rep: Reputation: 0
Ok, that means, my plan to restore the disk has basically gone to hell.
 
Old 03-17-2025, 02:20 PM   #14
rknichols
Senior Member
 
Registered: Aug 2009
Distribution: Rocky Linux
Posts: 4,820

Rep: Reputation: 2240Reputation: 2240Reputation: 2240Reputation: 2240Reputation: 2240Reputation: 2240Reputation: 2240Reputation: 2240Reputation: 2240Reputation: 2240Reputation: 2240
Quote:
Originally Posted by Julius_Niezufus View Post
I rarely delete files from that disk, therefore I expect those changes to be found at the end of the used space with very few exceptions.
In the Windows system partition, every time Windows does an update many files are replaced, leaving a large number of small to medium holes in the filesystem where new files might be allocated.

Last edited by rknichols; 03-17-2025 at 02:21 PM.
 
Old 03-17-2025, 02:43 PM   #15
Julius_Niezufus
Member
 
Registered: Feb 2025
Distribution: Manjaro
Posts: 47

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by rknichols View Post
In the Windows system partition, every time Windows does an update many files are replaced, leaving a large number of small to medium holes in the filesystem where new files might be allocated.
There was no Windows installed on that drive. It was just documents, pictures, movies, etc.
 
  


Reply



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
NAS ext4 drives initialized with Win10 Computer manager :( guid parttable and ext4 filesystem loss uyjjhak Linux - Server 4 04-20-2021 06:32 PM
How to change file system from ext4 journeying to ext4 writeback ? 5883 Linux - Newbie 6 03-10-2014 08:04 AM
modprobe: Module ext4 not found. no dependencies for kernel module 'ext4' found Aquarius_Girl Linux - Newbie 6 01-25-2012 05:07 AM
[SOLVED] Is my data safe after running "sync" if I use data=writeback,barrier=0 (ext4)? *Dark Dragon* Linux - General 4 01-11-2012 02:25 PM
Is it safe to format USB flash to ext4 or ext4? joham34 Linux - Newbie 2 01-08-2011 11:58 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Software

All times are GMT -5. The time now is 07:56 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration