DownloadManager's cleanOrphans method erroneously removes files that have been moved from the Downloads folder [148573846] - Visible to Public

Assigned

Bug

Description

ap...@sudomemo.net

created issue #1

Jan 30, 2020 03:57PM

Device: Moto Z3 Play

Build #: PPWS29.183-29-1-9

OS: Android 9

Android's DownloadManager has a function called cleanOrphans; files that have passed through the Downloads folder and then get moved to a different folder get seen as orphaned files by DownloadManager, which deletes them when the device enters a doze mode idle maintenance window.

Even though the files have been moved out of the folder, it's still able to find them to delete them, perhaps it's storing file inode # and looking it up by them.

I set up a service using termux and inotifywait to monitor the folder that my stuff was being removed from, and it logged a series of DELETE calls to the files, which helped me track it down in logcat. Interestingly, the files were not opened and they didn't have their attributes checked, it simply started with the DELETEs.

Snippet:

```
Watcher: DELETE image0-23.jpg @ Thu Jan 30 04:28:04 EST 2020
Watcher: DELETE image-1.jpg @ Thu Jan 30 04:28:06 EST 2020
Watcher: DELETE image-2.jpg @ Thu Jan 30 04:28:09 EST 2020
Watcher: DELETE image-1.png @ Thu Jan 30 04:28:11 EST 2020
Watcher: DELETE image-3.jpg @ Thu Jan 30 04:28:13 EST 2020
Watcher: DELETE image-4.jpg @ Thu Jan 30 04:28:16 EST 2020
Watcher: DELETE image-5.jpg @ Thu Jan 30 04:28:20 EST 2020
```

Logcat of when it happened:

```
01-30 04:27:37.837 1043 9368 D vold : idle maintenance started
01-30 04:27:37.880 1043 9368 D vold : Starting trim of /data
01-30 04:27:38.394 1043 9368 I vold : Trimmed 0 bytes on /data in 513ms
01-30 04:27:38.394 1043 9368 D vold : Starting trim of none
01-30 04:27:38.394 1043 9368 W vold : Failed to open none: No such file or directory
01-30 04:27:38.394 1043 9368 D vold : Starting trim of /persist
01-30 04:27:38.660 1043 9368 I vold : Trimmed 19324928 bytes on /persist in 265ms
01-30 04:27:38.660 1043 9368 D vold : Starting trim of /mnt/vendor/persist
01-30 04:27:38.660 1043 9368 W vold : Failed to open /mnt/vendor/persist: Permission denied
01-30 04:27:38.662 1043 9368 D vold : idle maintenance completed
01-30 04:27:38.794 2151 2182 I chatty : uid=1000(system) Binder:2151_1 expire 12 lines
01-30 04:27:40.302 1366 1550 E storaged: getDiskStats failed with result NOT_SUPPORTED and size 0
01-30 04:27:41.095 17722 17477 D DownloadManager: Removed 0 stale downloads
01-30 04:27:41.135 2151 3741 I chatty : uid=1000(system) Binder:2151_10 expire 10 lines
01-30 04:27:41.189 2151 2505 I chatty : uid=1000(system) NetworkWatchlis expire 2 lines
01-30 04:27:41.219 2151 2213 I chatty : uid=1000(system) ActivityManager expire 11 lines
01-30 04:27:42.045 17722 17477 D DownloadManager: Missing /storage/emulated/0/Download/animation0-4.gif, deleting 4117
01-30 04:27:42.997 17722 17477 D DownloadManager: Missing /storage/emulated/0/Download/139748760132.jpg, deleting 4118
01-30 04:27:43.119 17722 17477 D DownloadManager: Missing /storage/emulated/0/Download/animation0-5.gif, deleting 4119
01-30 04:27:43.345 17722 17477 D DownloadManager: Missing /storage/emulated/0/Download/139105418232.jpg, deleting 4120
01-30 04:27:43.480 17722 17477 D DownloadManager: Missing /storage/emulated/0/Download/136605285441.jpg, deleting 4121
01-30 04:27:43.578 17722 17477 D DownloadManager: Missing /storage/emulated/0/Download/8ed6f01f-9ce7-4972-a058-c5bae4f92aed.jpg, deleting 4122
01-30 04:27:43.666 17722 17477 D DownloadManager: Missing /storage/emulated/0/Download/1503164927.png, deleting 4123
01-30 04:27:43.872 17722 17477 D DownloadManager: Missing /storage/emulated/0/Download/image0-23.jpg, deleting 4124
01-30 04:27:43.964 17722 17477 D DownloadManager: Missing /storage/emulated/0/Download/image-1.jpg, deleting 4125
01-30 04:27:44.105 17722 17477 D DownloadManager: Missing /storage/emulated/0/Download/image-2.jpg, deleting 4126
01-30 04:27:44.182 17722 17477 D DownloadManager: Missing /storage/emulated/0/Download/image-1.png, deleting 4127
01-30 04:27:44.278 17722 17477 D DownloadManager: Missing /storage/emulated/0/Download/image-3.jpg, deleting 4128
01-30 04:27:44.356 17722 17477 D DownloadManager: Missing /storage/emulated/0/Download/image-4.jpg, deleting 4129
01-30 04:27:44.450 17722 17477 D DownloadManager: Missing /storage/emulated/0/Download/image-5.jpg, deleting 4130
01-30 04:27:44.613 17722 17477 D DownloadManager: Missing /storage/emulated/0/Download/8rQ9X50.jpg, deleting 4131
01-30 04:27:44.742 17722 17477 D DownloadManager: Missing /storage/emulated/0/Download/ol5me5qagea99.png, deleting 4132
01-30 04:27:44.845 17722 17477 D DownloadManager: Missing /storage/emulated/0/Download/1uidureiv1a87.jpg, deleting 4133
01-30 04:27:44.933 17722 17477 D DownloadManager: Missing /storage/emulated/0/Download/nrqv5hE.jpg, deleting 4134
01-30 04:27:45.060 17722 17477 D DownloadManager: Found 879 files in database
01-30 04:27:45.060 17722 17477 D DownloadManager: Found 5 files on disk
01-30 04:27:59.572 2151 3738 I chatty : uid=1000(system) Binder:2151_E expire 15 lines
01-30 04:27:59.572 2151 2620 I chatty : uid=1000(system) Binder:2151_3 expire 10 lines
01-30 04:27:59.573 2151 4367 I chatty : uid=1000(system) Binder:2151_1B expire 6 lines
01-30 04:27:59.573 2151 3379 I chatty : uid=1000(system) Binder:2151_A expire 10 lines
01-30 04:27:59.576 2151 4358 I chatty : uid=1000(system) Binder:2151_13 expire 12 lines
```

In my case, the images I was downloading were things like brand assets, family pictures from Facebook, memes, and so on. I then sorted them into folders via the Google Photos app.

There's many, many cases of this happening to people:

https://support.google.com/photos/thread/200457?hl=en "Pictures got deleted on their own", 909 replies, varied issues here but many people describing the same situation

https://support.google.com/photos/thread/106201?hl=en "Photos disappeared after moving to another device folder", 55 replies, same situation described

https://support.google.com/photos/thread/560768?hl=en This user has Google photos backup turned on so they don't actually lose the images, but they get repeatedly deleted from the device.
Note this sentence: "..., and it ends up downloading them to a "Downloads" album/folder in the Google Photos app, and I have to RE-organize into the albums/folders they were in" indicating that they pass through the Downloads folder first, thus through DownloadManager.

I found one mention of cleanOrphans here: https://issuetracker.google.com/issues/78078399

A patch was submitted, but never merged.

https://android-review.googlesource.com/c/platform/packages/providers/DownloadProvider/+/674009

Eventually, it was closed automatically.

I would appreciate more attention to this issue. I've lost a lot of great memes...as well as brand assets and such that I painstakingly sorted into their own folders, only for them to be summarily removed. But beyond the images/videos being lost, it could be more important things, such as PDFs with receipts, etc.

Steps to reproduce:

- Download some images or videos
- Move them to a different folder (I used the Google Photos app to do the actual move, but I don't think that matters)
- Wait overnight for the system to do its doze mode maintenance and for DownloadManager's cleanOrphans to run
- Your files are now gone

Mentioned issues (1)

All the content is deleted if Rename Downlaod Folder.

“

https://issuetracker.google.com/78078399 ”

ap...@

Links (10)

Comments

pa...@gmail.com <pa...@gmail.com> #2Jan 30, 2020 07:18PM

Same thing happened to me last night. Thankfully it was just memes but I spent hours organizing them into different folders.

za...@gmail.com <za...@gmail.com> #3Jan 31, 2020 12:16AM

It doesn't seem to actually be DownloadIdleService (What you're seeing as DownloadManager in the logs).

You can force it to run immediately with the following ADB command:

adb shell cmd jobscheduler run -f com.android.providers.downloads -100

Testing on a Galaxy Note9 running Android Pie and an OPPO Reno2, also running Android Pie, I can't reproduce this by forcing DownloadIdleService to run. While it does create the log entries you're seeing, it doesn't actually delete the files that have been moved. I've tried moving them using Google Photos, OPPO's built-in Photos app, Samsung's My Files, app, and MiXPlorer. I've tried downloading images from both Chrome and Discord.

There's also a comment above what generates the logs:

// File appears to live on internal storage, or a
// currently mounted device, so remove it from database.
// This logic preserves files on external storage while
// media is removed.

This (and the corresponding code) tells me it only deletes the database entry, and doesn't actually touch the filesystem.

Unfortunately, I think it's something else in the maintenance cycle that isn't directly related to DownloadManager.

te...@google.com <te...@google.com> Jan 31, 2020 05:40AM

Assigned to te...@google.com.

te...@google.com <te...@google.com> #4Jan 31, 2020 09:48AM

@apburk, thank you for reporting this issue. For us to further investigate this issue, please provide the following additional information:

Android bug report (to be captured after reproducing the issue)
For steps to capture a bug report, please refer: https://developer.android.com/studio/debug/bug-report#bugreportdevice

Alternate method
Navigate to “Developer options”, ensure “USB debugging” is enabled, then enable “Bug report shortcut”. Capture bug report by holding the power button and selecting the “Take bug report” option.

Note: Please upload the files to google drive and share the folder to android-bugreport@google.com, then share the link here.

ap...@sudomemo.net <ap...@sudomemo.net> #5Feb 1, 2020 05:24AM

Hey, thanks for the instructions. I have some questions about the actual timing of the bug report, however. I think I can have Termux do the same thing as before but also trigger a bug report when it detects that files have started being deleted (I believe there is an intent I can signal via am to do this, but can't recall off the top of my head) Should this be done before the deletion starts, during, or after? Doing it before might be a bit of a challenge as I can't access system-wide logcat from within Termux, though I suppose I could hook it up to my laptop and watch logcat to trigger it.

Cheers,

Austin.

te...@google.com <te...@google.com> #6Feb 4, 2020 11:24AM

Please reproduce the issue again and take the bugreport immidiately after the files get deleted.

Note: Please upload the files to google drive and share the folder to android-bugreport@google.com, then share the link here.

td...@gmail.com <td...@gmail.com> #7Feb 7, 2020 06:37AM

I have observed the same behaviour on my pixel 4. I'll try and reproduce it and get the bugreport filed too.

te...@google.com <te...@google.com> #8Feb 11, 2020 09:25AM

Please share relevant information mentioned in

comment #6

, if you are still facing the issue.

ap...@sudomemo.net <ap...@sudomemo.net> #9Feb 12, 2020 09:30PM

I should have the bugreport by tomorrow or the day after - it's taken a bit to come up with a way to take the report, but I now have something that will play loud music right when it happens, waking me up so I can take the bug report.

ap...@gmail.com <ap...@gmail.com> #10Feb 16, 2020 05:11AM

I've done so, though there doesn't seem to be a way to generate a folder link from the mobile Drive app, so here's a link to the zip: https://drive.google.com/a/sudomemo.net/file/d/1zjF37F9l22DnIuf1p7AkqCYwCgvNbNTI/view?usp=drivesdk

It's been shared with that email address.

vc...@gmail.com <vc...@gmail.com> #11Feb 16, 2020 10:17PM

Same thing happens to my phone. I can't even find photos from last month but the photos on my cloud from long ago appears in my camera folder.

te...@google.com <te...@google.com> #12Feb 19, 2020 06:44AM

Thank you for reporting this issue. We’ve shared this with our product and engineering teams and will continue to provide updates as more information becomes available.

as...@gmail.com <as...@gmail.com> #13Mar 29, 2020 04:19PM

I have the same problem on Moto g6. How do I recover these files? What is the best way to do this? They were just on my phone a few days ago

sa...@outlook.com <sa...@outlook.com> #14Apr 28, 2020 10:03AM

Any updates? My images were just deleted again today.

wi...@gmail.com <wi...@gmail.com> #15Apr 28, 2020 04:07PM

I'm tired of downloading and re-downloading again and it kept deleting and it came into my conclusion that it really is a bug. Can those photos be retrieved once this was fixed?

li...@gmail.com <li...@gmail.com> #16May 3, 2020 03:57PM

I'm having the same issue

ga...@gmail.com <ga...@gmail.com> #17May 16, 2020 03:18PM

Same thing happened to me

kh...@gmail.com <kh...@gmail.com> #18May 19, 2020 02:12AM

I have the same problem on OnePlus 5t

ma...@gmail.com <ma...@gmail.com> #19May 22, 2020 12:47AM

What now?

sa...@outlook.com <sa...@outlook.com> #20May 23, 2020 10:01AM

I know I've already commented but are there any updates on this issue? Once again all my files were deleted off my device and Im getting quite sick of having to redownload them.

ta...@gmail.com <ta...@gmail.com> #21Jun 4, 2020 02:42PM

Any updates ? My phone has just deleted all 900 of my photos yet again???

ta...@gmail.com <ta...@gmail.com> #22Jun 4, 2020 02:43PM

This keeps happening. I've got photos of my son I'm losing. This is ridiculous. Nothing I do stops it.

t....@e3n.de <t....@e3n.de> #23Jun 5, 2020 07:21AM

regarding

comment #6

- it's difficult to produce a bug report because you don't know when it's going to happen. I've not been able to observe a regular routine so that I can wait and catch it happening. If I could, I'd have more than one bug report for you. I can verify, that, connected by cable/usb-drive and copying the exact same images to the the exact same folders does not have the this problem. It only happens, if images are downloaded to the phone.

sa...@outlook.com <sa...@outlook.com> #24Jun 30, 2020 10:13AM

All my images and videos! Gone yet again! This time it didnt even take a month from the last time this happend. Just, woke up and poof, gone. Im losing stuff every time I need to redownload everything. Im beyond pissed. Is this actually being looked into??

ra...@gmail.com <ra...@gmail.com> #25Jul 29, 2020 03:14AM

I've also have this issue on my Motorola One Action running Andriod 10. This is extremely bothersome

c9...@gmail.com <c9...@gmail.com> #26Aug 3, 2020 02:45AM

Hi, I've been experiencing this problem for months now as well. I am using a Oppo R11S and I have had over a hundred photos in my downloads folder that have been deleted every time I switch off my phone at night. Is there any way to rectify this issue? It is seriously affecting my experience of using Android.

rd...@gmail.com <rd...@gmail.com> #27Aug 6, 2020 02:23PM

This is happening to me as well on a Pixel 3. Last night it deleted a bunch of video files that were in use in a video editing app. HOURS of work gone... this needs a patch.

lo...@gmail.com <lo...@gmail.com> #28Aug 9, 2020 08:00PM

Occurring on an updated Google Fi Moto x4, did not appear to occur before. Lost many pictures out of folders in the Pictures directory, unrecoverable but the folders they were in are still there, empty.

sa...@gmail.com <sa...@gmail.com> #29Sep 4, 2020 11:31PM

I'm not phone or computer savvy so I apologize for this stupid question but, is there a solution yet? Took photos of my son on vacation hours ago and they all just deleted 😭. Phone was dragging, restarted, system update, and then I had to helplessly watch each photo get deleted right in front of me. Nothing I could do and I'm devastated. Why is Google not addressing this?

sa...@google.com <sa...@google.com> Sep 11, 2020 11:51AM

Reassigned to ra...@google.com.

br...@gmail.com <br...@gmail.com> #30Nov 4, 2020 12:48AM

Mine is a little different. My albums keep deleting themselves but the photos stay in Google Photos. I’m thankful that I don’t lose any images but I really would like for my albums to not delete themselves without my permission. I also happen to have an iPhone so I’m really confused on why my albums delete on their own.

xa...@gmail.com <xa...@gmail.com> #31Nov 5, 2020 11:20AM

I'm having this issue on a Note 20 Ultra, however my Note 10 5G is fine. Same exact settings as far as I can tell but here's what gets deleted about 1 week after I reloaded phone (wishing that'd fix this). Photos not taken by camera, all downloaded ringtones, music, notifications, files not opened in last 24 hours. I download all of said items from OneDrive. I've verified OneDrive, Google Photos, Samsung's backup cloud, etc are all set to NOT delete any items. Only OneDrive is uploading the camera. I do not have logs but the phone was formatted and re-setup on Oct 28 and deleted all those files last night/yesterday evening. I checked my Note 10, no issue but I had downloaded a bunch of new items on the 28th to it as well.

he...@gmail.com <he...@gmail.com> #32Nov 23, 2020 09:07AM

Yup, I'm getting it too. At least I now know I'm not insane.

Is there no fix?? No workaround for now? I need things separated into albums.

da...@gmail.com <da...@gmail.com> #33Dec 3, 2020 01:19AM

i've been talking to Google support for 2+ months about this and no one has a damn clue. Please get this fixed, so stupid Google. going back to iOS

sa...@gmail.com <sa...@gmail.com> #34Dec 27, 2020 05:24AM

Same deal here. I downloaded and sorted some pictures into folders so they don't clutter up "downloads" before going to sleep After waking up they have vanished.
Nice to see Google has not moved in almost a whole year. That's great service.

ww...@gmail.com <ww...@gmail.com> #35Dec 28, 2020 02:35AM

It just happemed to me last night!!

So...@hotmail.com <So...@hotmail.com> #36Jan 20, 2021 07:21PM

People have been having this issue for over 2 years with no fix released. Are google developers really this incompetent? This will be the last Google product I own

wm...@gmail.com <wm...@gmail.com> #37Feb 4, 2021 02:18PM

I can reproduce this issue on my Mi 10 Ultra (rooted, MIUI 12.3 by xiaomi.eu 20.12.10 Beta).
If I rename (instead of just moving) the downloaded file, then it won't get deleted.

android-download-manager-bug.jpg

1.6 MB

View

Download

ju...@gmail.com <ju...@gmail.com> #38Feb 16, 2021 01:34AM

Having the same issue on my fully up to date P30 Lite - wtf is going with this serious bug?! I have lost family photos and baby pictures sent to me by my friends as well. This is absolutely pathetic.

ho...@gmail.com <ho...@gmail.com> #39Mar 16, 2021 02:15AM

This is sad.

ah...@gmail.com <ah...@gmail.com> #40Apr 14, 2021 12:54PM

so is it possible to even restore these photos ?
coz this is realy frustrating !

li...@gmail.com <li...@gmail.com> #41Jul 12, 2021 03:13AM

any updates on this issue? I've been having the same 5 gifs deleted from my device's folder. I've even tried redownloading them from my iPad, syncing to Google Photos, and downloading to my Android phone. STILL gets deleted. i may have other photos deleted from my device that i just haven't noticed yet. It's still in Google Photos in the cloud, but i need these particular gifs on my device. Redownloading is time-consuming and really frustrating. This seems like an issue that's been happening to other people for at least 2 years. Google, step on it! We need a solution!

su...@gmail.com <su...@gmail.com> #42Jul 30, 2022 12:42PM

Hi, my device also having the same issue, I've lost my family photos which is I couldn't get them back, I'm attaching the bug report in Google drive and the link is here. The folder has been shared to the email given.

https://drive.google.com/drive/folders/1RrxjrnRdh0k3Kz_GBBYHUbUT8zW1-uTp

el...@gmail.com <el...@gmail.com> #43May 27, 2023 05:08PM

I have this issue too. Just lost lot of videos from my device. Modern software is nothing but a pile of incompetence.