Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum. |
| Notices |
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
Are you new to LinuxQuestions.org? Visit the following links:
Site Howto |
Site FAQ |
Sitemap |
Register Now
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
|
 |
|
03-17-2025, 04:54 AM
|
#1
|
|
Member
Registered: Feb 2025
Distribution: Manjaro
Posts: 47
Rep:
|
Need info on ext4 for data rescue
[ Log in to get rid of this advertisement]
I accidentally installed a Linux distribution on my encrypted 8 TB HDD.
I basically have a backup of that drive, but in an attempt to save the files/changes to files I added after my last backup, i chose not to just resort to that backup, but to actually overwrite the accidentally overwritten block with the corresponding blocks from my backup, as i presume, the changes I made are to be found at the end of the used disk space and didn't get overwritten. Now the question is: How much data do i have to write back to my disk, aka. how much disk space got corrupted?
The files of the accidentally installed distro had a total size a bit more than 500 MB + 100 MB EFI partition. The file system is ext4.
So here are my questions:
How much overhead is used by ext4 for things like GDT, block bitmaps, inode tables, etc?
Anything else, which needs some overhead space?
Does ext4 write anything into the last blocks of the disk?
Anything else, i need to consider?
Additional question added:
Is my assumption right, that when installing a new distro, all content is written from the start of the disk on, and all blocks after that remain untouched?
I hope the files aren't randomly spread all over the place.
Any info is appreciated
Last edited by Julius_Niezufus; 03-17-2025 at 07:32 AM.
|
|
|
|
03-17-2025, 07:18 AM
|
#2
|
|
Moderator
Registered: Mar 2008
Posts: 22,361
|
Use something like testdisk photorec. Any attempt to play with live disk can further damage data
A dd clone first might help,
|
|
|
|
03-17-2025, 07:23 AM
|
#3
|
|
Member
Registered: Feb 2025
Distribution: Manjaro
Posts: 47
Original Poster
Rep:
|
I'm aware of this, thank you, but that doesnt answer my questions.
Last edited by Julius_Niezufus; 03-17-2025 at 07:25 AM.
|
|
|
|
03-17-2025, 08:03 AM
|
#4
|
|
Senior Member
Registered: Aug 2009
Distribution: Rocky Linux
Posts: 4,820
|
Quote:
Originally Posted by Julius_Niezufus
I accidentally installed a Linux distribution on my encrypted 8 TB HDD.
I basically have a backup of that drive, but in an attempt to save the files/changes to files I added after my last backup
...
Additional question added:
Is my assumption right, that when installing a new distro, all content is written from the start of the disk on, and all blocks after that remain untouched?
I hope the files aren't randomly spread all over the place.
|
They are.
Furthermore, the LUKS header for your encrypted drive will almost certainly have been overwritten. That makes recovery of the encrypted content impossible unless you have a backup of that LUKS header. If your backup is a bit-for-bit image of the drive (i.e., not a copy of the files), then you could use the LUKS header from that backup to allow decryption of the overwritten drive. I might be able to help with the procedure for doing that. Otherwise, recovery of that encrypted information is impossible.
|
|
|
|
03-17-2025, 08:29 AM
|
#5
|
|
Member
Registered: Feb 2025
Distribution: Manjaro
Posts: 47
Original Poster
Rep:
|
Does that refer to the question whether files are placed randomly on a disk instead of consecutively?
Just for clarification, we're not talking about an SSD but an HDD.
Quote:
|
Furthermore, the LUKS header for your encrypted drive will almost certainly have been overwritten. That makes recovery of the encrypted content impossible unless you have a backup of that LUKS header.
|
Yes, I do have a backup of the VeraCrypt header.
Quote:
|
If your backup is a bit-for-bit image of the drive
|
Yes, it is, except for the changes and new files I've added since its creation. But I assume these changes to reside at the end of the disk, so I expect these changes to be unaffected. (Am I really wrong about this?)
Quote:
|
I might be able to help with the procedure for doing that.
|
YES, PLEASE!!!!!
Oh, here's another thing that's bugging me:
I've taken a look at the corrupted drive in a hex-editor and what I've seen heavily discomforts me:
I'd have expected to see either a large junk of gibberish (where my data is supposed to be stored) or large parts of zeros (where no data resides)
What I see instead is several MBs of gibberish followed by several MBs of zeroes. Now this pattern seems to repeat over the whole disk. I havent yet found either several TBs of gibberish or zeroes. Is the reason for that within the behavior of VeraCrypt or the underlying encrypted NTFS file system? I'm pretty sure it cannot originate from the accidental overwrite, because this would have taken hours or even days.
Last edited by Julius_Niezufus; 03-17-2025 at 08:37 AM.
Reason: Additional questions
|
|
|
|
03-17-2025, 11:00 AM
|
#6
|
|
Senior Member
Registered: Aug 2009
Distribution: Rocky Linux
Posts: 4,820
|
Placement of files is different in NTFS vs. ext4. NTFS does initially fill space from the beginning of the disk, but new or changed content tends to go into the first available free space, which could be anywhere in the disk that space has been made available due to files having been deleted. In ext4, the entire space is divided into block groups, and files are scattered among the block groups, with an attempt to keep related files (same directory) together. Your changed files could be anywhere within the maximum space that the NTFS filesystem used.
Since you initially mentioned only ext4, I assumed that your original encryption was using LUKS. Now you mention "VeraCrypt or the underlying encrypted NTFS file system." So, I am now guessing that you originally had an NTFS filesystem encrypted by VeraCrypt and overwrote it with an ext4 filesystem encrypted with LUKS. Is that correct? What I would expect to see now is many megabytes of gibberish where the original NTFS data resided (now partially overwritten by ext4 data) followed by a pattern of blocks of gibberish and blocks of zeros where the NTFS filesystem had free space that had never been used, and that region is now partially overwritten by data at the start of each ext4 block group.
I have no experience with VeraCrypt, unfortunately.
Last edited by rknichols; 03-17-2025 at 11:03 AM.
Reason: no experience comment
|
|
|
|
03-17-2025, 11:34 AM
|
#7
|
|
Member
Registered: Feb 2025
Distribution: Manjaro
Posts: 47
Original Poster
Rep:
|
Quote:
|
So, I am now guessing that you originally had an NTFS filesystem encrypted by VeraCrypt and overwrote it with an ext4 filesystem encrypted with LUKS. Is that correct?
|
Correct.
Quote:
|
Your changed files could be anywhere within the maximum space that the NTFS filesystem used.
|
I rarely delete files from that disk, therefore I expect those changes to be found at the end of the used space with very few exceptions.
Quote:
|
In ext4, the entire space is divided into block groups, and files are scattered among the block groups,
|
Are these block groups at least created only when used?
Nightmares coming true...
|
|
|
|
03-17-2025, 12:21 PM
|
#8
|
|
LQ Guru
Registered: Feb 2003
Location: Virginia, USA
Distribution: Debian 12
Posts: 8,387
|
The following web page has a verbal description of ext4 layout starting at section "Layout". The web page has a graphic layout of an exp file system starting at section "Graphical view of Disk Layout". The layout is created when you format an ext4 partition.
https://blogs.oracle.com/linux/post/...-layout-part-1
Last edited by jailbait; 03-17-2025 at 12:23 PM.
|
|
|
1 members found this post helpful.
|
03-17-2025, 01:07 PM
|
#9
|
|
Member
Registered: Feb 2025
Distribution: Manjaro
Posts: 47
Original Poster
Rep:
|
Quote:
|
The layout is created when you format an ext4 partition.
|
So, the installer has in fact already written stuff all accross the whole 8 TB???
Wouldn't that take hours on such a large drive?
|
|
|
|
03-17-2025, 01:17 PM
|
#10
|
|
Moderator
Registered: Aug 2002
Posts: 26,800
|
To make matters worse metadata backup superblocks are written throughout the disk as well as the backup GPT at the end of the disk which may have changed. I don't think you can guarantee the disk is written consecutively from the start during the install. ext4 primary metadata is on the order of ~2% of the total size of the partition. Did the installer create partitions(s) to fill the entire disk? Do you have a clone of the disk?
|
|
|
|
03-17-2025, 01:27 PM
|
#11
|
|
Member
Registered: Feb 2025
Distribution: Manjaro
Posts: 47
Original Poster
Rep:
|
Quote:
|
metadata backup superblocks are written throughout the disk
|
how large are they?
Quote:
|
backup GPT at the end of the disk
|
Yes, I found that one, but I dont expect that to be a big deal, I was far away from using the full disk space, not even the hidden volume (which is usually stored at the end of the disk) was full.
Quote:
|
Did the installer create partitions(s) to fill the entire disk?
|
None except the EFI partition.
Quote:
|
Do you have a clone of the disk?
|
Yes, but that clone was made 3 weeks ago. That means some new stuff won't be on that clone. Otherwise I wouldn't bother.
|
|
|
|
03-17-2025, 02:05 PM
|
#12
|
|
Senior Member
Registered: Aug 2009
Distribution: Rocky Linux
Posts: 4,820
|
Quote:
Originally Posted by michaelk
I don't think you can guarantee the disk is written consecutively from the start during the install.
|
In fact, you can guarantee that it is not. ext4 tries to keep usage of the block groups balanced.
Just to confirm, I created a 100GB ext4 filesystem, mounted it on /mnt/tmp, and ran the following:
Code:
for N in {101..125}; do
mkdir /mnt/tmp/Dir$N
cp /etc/cups/ppd/* /mnt/tmp/Dir$N
done
That creates 125 directories on the new filesystem and populates each directory with copies of the 7 files that my system has in /etc/cups/ppd. That uses less than 1% of the space on the new filesystem. Running "ls -iR /mnt/tmp" reveals (by the inode numbers) that the directories are spread around in the ext4 block groups. In my case, the lowest and highest inode nummbers used were 131073 and 6422536, which range from the beginning to the end of the filesystem (6553600 total inodes, 8192 inodes per block group). As expected, the files within each directory were kept together (consecutive inode numbers).
Code:
/mnt/tmp/D101:
6291458 Brother.ppd
6291459 Brother.ppd.O
6291460 HP_OfficeJet_Pro_8710.ppd
6291461 HP.ppd
6291462 HP.ppd.O
6291463 Panasonic.ppd
6291464 Panasonic.ppd.O
/mnt/tmp/D102:
131074 Brother.ppd
131075 Brother.ppd.O
131076 HP_OfficeJet_Pro_8710.ppd
131077 HP.ppd
131078 HP.ppd.O
131079 Panasonic.ppd
131080 Panasonic.ppd.O
...
/mnt/tmp/D124:
1835010 Brother.ppd
1835011 Brother.ppd.O
1835012 HP_OfficeJet_Pro_8710.ppd
1835013 HP.ppd
1835014 HP.ppd.O
1835015 Panasonic.ppd
1835016 Panasonic.ppd.O
/mnt/tmp/D125:
6160386 Brother.ppd
6160387 Brother.ppd.O
6160388 HP_OfficeJet_Pro_8710.ppd
6160389 HP.ppd
6160390 HP.ppd.O
6160391 Panasonic.ppd
6160392 Panasonic.ppd.O
|
|
|
|
03-17-2025, 02:08 PM
|
#13
|
|
Member
Registered: Feb 2025
Distribution: Manjaro
Posts: 47
Original Poster
Rep:
|
Ok, that means, my plan to restore the disk has basically gone to hell.
|
|
|
|
03-17-2025, 02:20 PM
|
#14
|
|
Senior Member
Registered: Aug 2009
Distribution: Rocky Linux
Posts: 4,820
|
Quote:
Originally Posted by Julius_Niezufus
I rarely delete files from that disk, therefore I expect those changes to be found at the end of the used space with very few exceptions.
|
In the Windows system partition, every time Windows does an update many files are replaced, leaving a large number of small to medium holes in the filesystem where new files might be allocated.
Last edited by rknichols; 03-17-2025 at 02:21 PM.
|
|
|
|
03-17-2025, 02:43 PM
|
#15
|
|
Member
Registered: Feb 2025
Distribution: Manjaro
Posts: 47
Original Poster
Rep:
|
Quote:
Originally Posted by rknichols
In the Windows system partition, every time Windows does an update many files are replaced, leaving a large number of small to medium holes in the filesystem where new files might be allocated.
|
There was no Windows installed on that drive. It was just documents, pictures, movies, etc.
|
|
|
|
All times are GMT -5. The time now is 07:56 AM.
|
|
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.
|
Latest Threads
LQ News
|
|