Ceph » rbd

Hi Sean,

We are seeing an issue with rbd-nbd when trying to upgrade our operating system from bionic to focal. This behavior occurs intermittently when we create map, format, and unmap images simultaneously (around 20-30). After running a test with this amount of traffic we end up with the host accumulating a large number of rbd-nbd processes stuck in uninterruptible sleep.

Have you attempted to grab the stack traces for one of these stuck processes, e.g. /proc/<pid>/stack for each thread?

Around when the bad processes are created there’s also a number of errors about bad RIP values. We were exploring this as a kernel issue but the kernel version between our bionic and focal instances is almost identical (5.4.0-1037-aws vs 5.4.0-1038-aws).

Do you have a example? Attaching a full dmesg from the node exhibiting the issue would be even better.

$ rbd-nbd --version
ceph version 14.2.21 (5ef401921d7a88aea18ec7558f7f9374ebd8f5a6) nautilus (stable)

$ ceph --version
ceph version 14.2.22 (ca74598065096e6fcbd8433c8779a2be0c889351) nautilus (stable)

Why is rbd-nbd version (slightly) different from ceph?

$ rbd-nbd --version
ceph version 15.2.13 (c44bc49e7a57a87d84dfff2a077a2058aa2172e2) octopus (stable)

$ ceph --version
ceph version 14.2.22 (ca74598065096e6fcbd8433c8779a2be0c889351) nautilus (stable)

And here, why has (only?) rbd-nbd been upgraded to the octopus release?

Hi Sean,

All but one rbd-nbd processes are stuck trying to grab nbd->config_lock mutex:

[1816218.597378] INFO: task rbd-nbd:3607847 blocked for more than 120 seconds.
[1816218.604607]       Tainted: G        W         5.8.0-1038-aws #40~20.04.1-Ubuntu
[1816218.619054] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[1816218.631875] rbd-nbd         D    0 3607847 3607037 0x00000000
[1816218.631878] Call Trace:
[1816218.631888]  __schedule+0x394/0xa60
[1816218.631893]  ? check_preempt_curr+0x7a/0x90
[1816218.631895]  ? ttwu_do_wakeup+0x1e/0x160
[1816218.631896]  schedule+0x55/0xc0
[1816218.631898]  schedule_preempt_disabled+0xe/0x10
[1816218.631900]  __mutex_lock.isra.0+0x17d/0x4e0
[1816218.631905]  ? security_capable+0x3d/0x60
[1816218.631907]  __mutex_lock_slowpath+0x13/0x20
[1816218.631909]  mutex_lock+0x32/0x40
[1816218.631914]  nbd_ioctl+0x6d/0x410 [nbd]
[1816218.631916]  ? __schedule+0x39c/0xa60
[1816218.631920]  blkdev_ioctl+0x102/0x250
[1816218.631923]  block_ioctl+0x3d/0x50
[1816218.631926]  ksys_ioctl+0x9d/0xd0
[1816218.631928]  __x64_sys_ioctl+0x1a/0x20
[1816218.631931]  do_syscall_64+0x49/0xc0
[1816218.631934]  entry_SYSCALL_64_after_hwframe+0x44/0xa9

But nbd->config_lock mutex is held by another rbd-nbd process which is trying to flush the I/O queue and it is probably /dev/nbd11:

[1816218.631950] INFO: task rbd-nbd:3607956 blocked for more than 120 seconds.
[1816218.643479]       Tainted: G        W         5.8.0-1038-aws #40~20.04.1-Ubuntu
[1816218.653994] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[1816218.661232] rbd-nbd         D    0 3607956 3607271 0x00004000
[1816218.661236] Call Trace:
[1816218.661252]  __schedule+0x394/0xa60
[1816218.661257]  ? __kfifo_to_user_r+0xa0/0xa0
[1816218.661259]  schedule+0x55/0xc0
[1816218.661265]  blk_mq_freeze_queue_wait+0x4b/0xb0
[1816218.661269]  ? wait_woken+0x80/0x80
[1816218.661270]  blk_mq_freeze_queue+0x1b/0x20
[1816218.661275]  nbd_add_socket+0x5e/0x1d0 [nbd]
[1816218.661278]  nbd_ioctl+0x2f7/0x410 [nbd]
[1816218.661282]  blkdev_ioctl+0x102/0x250
[1816218.661285]  block_ioctl+0x3d/0x50
[1816218.661288]  ksys_ioctl+0x9d/0xd0
[1816218.661290]  __x64_sys_ioctl+0x1a/0x20
[1816218.661296]  do_syscall_64+0x49/0xc0
[1816218.661300]  entry_SYSCALL_64_after_hwframe+0x44/0xa9

[1816087.525808] block nbd11: Possible stuck request 00000000e7205288: control (read@1024,1024B). Runtime 30 seconds
[1816118.241672] block nbd11: Possible stuck request 00000000e7205288: control (read@1024,1024B). Runtime 60 seconds
[1816148.961506] block nbd11: Possible stuck request 00000000e7205288: control (read@1024,1024B). Runtime 90 seconds
[1816179.681332] block nbd11: Possible stuck request 00000000e7205288: control (read@1024,1024B). Runtime 120 seconds
[1816210.401195] block nbd11: Possible stuck request 00000000e7205288: control (read@1024,1024B). Runtime 150 seconds

The question is why /dev/nbd11 is unable to perform I/O. You said that "eventually the host becomes unresponsive to ceph commands" -- do you mean that other, non rbd(-nbd) related ceph commands don't work either? Can the host talk to the cluster at that point at all?

The version difference is because we are trying to upgrade the instance that is using rbd-nbd from bionic to focal - and it doesn’t look like there are nautilus rbd-nbd binaries for focal. Is there an expectation that ceph and rbd-nbd versions match? Or that one is higher than the other?

Yes, the expectation is that "ceph --version" and "rbd-nbd --version" match. Where are you grabbing the packages from?

I'm also not clear on the kernel versions. Previously you mentioned 5.4.0-1037-aws and 5.4.0-1038-aws but the attached dmesg shows 5.8.0-1038-aws. I didn't download any of these packages so I'm not 100% sure but from a quick look at the changelog, neither 5.4.0-1037-aws nor 5.4.0-1038-aws have the "nbd: freeze the queue while we're adding connections" patch that added that blk_mq_freeze_queue() call to nbd_add_socket(). And 5.8.0-1038-aws does.

Are you sure you reproduced the same issue? Unless I'm misreading the changelog, these stack traces are impossible to generate on 5.4.0-1037-aws or 5.4.0-1038-aws. "nbd: freeze the queue while we're adding connections" showed up in 5.4 series only in 5.4.0-1042.

It would good if you could reproduce varying just one thing at a time. For example, stay on bionic ceph packages so that all versions match and try to upgrade just the kernel.

Hello!

So I found nbd11 in the process list. It’s a mount command.

$ ps aux | grep rbd
root     10053  0.1  0.1 1429864 24416 pts/0   Sl   15:52   0:00 rbd-nbd -c /etc/ceph/ceph.abb431cf-6265-445c-8f0d-548f4b1b4b01.conf map lucid_staging_rbd_0/sean-test-image-582
root     19798  0.0  0.1 1429860 24172 pts/0   Sl   15:47   0:00 rbd-nbd -c /etc/ceph/ceph.abb431cf-6265-445c-8f0d-548f4b1b4b01.conf map lucid_staging_rbd_0/sean-test-image-291

This has the following two stack traces I don’t know if that tells you more,

$ sudo cat /proc/3612529/stack
[<0>] poll_schedule_timeout.constprop.0+0x46/0x70
[<0>] do_sys_poll+0x3ef/0x540
[<0>] __x64_sys_ppoll+0xa8/0xe0
[<0>] do_syscall_64+0x49/0xc0
[<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9

$ sudo cat /proc/3612535/stack
[<0>] __bread_gfp+0xb7/0xf0
[<0>] ext4_fill_super+0x1d7/0x2ca0
[<0>] mount_bdev+0x17f/0x1b0
[<0>] ext4_mount+0x15/0x20
[<0>] legacy_get_tree+0x2b/0x50
[<0>] vfs_get_tree+0x2a/0xc0
[<0>] do_mount+0x7a8/0xa20
[<0>] __x64_sys_mount+0x8b/0xe0
[<0>] do_syscall_64+0x49/0xc0
[<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9

You said that "eventually the host becomes unresponsive to ceph commands" -- do you mean that other, non rbd(-nbd) related ceph commands don't work either?

So this behavior seems slightly different when I run my test script - compared to when we see this in the wild. When I’m running the script I’m doing it on the node itself and after triggering this I’m still able to run ceph and rbd commands. When this happens in the wild the container on the node stops being responsive - future calls timeout - and we can’t even get shell onto the container. The container is also unkillable at this point and we have to bounce the node in order to clean up the container and the stuck processes.

Yes, the expectation is that "ceph --version" and "rbd-nbd --version" match. Where are you grabbing the packages from?

We are getting packages from here: https://download.ceph.com/debian-nautilus/dists/focal/main/binary-amd64/

Which doesn’t look like it includes nautilus binaries for rbd-nbd. If there’s an alternative location to get focal binaries we would definitely try that.

I'm also not clear on the kernel versions. Previously you mentioned 5.4.0-1037-aws and 5.4.0-1038-aws but the attached dmesg shows 5.8.0-1038-aws. I didn't download any of these packages so I'm not 100% sure but from a quick look at the changelog, neither 5.4.0-1037-aws nor 5.4.0-1038-aws have the "nbd: freeze the queue while we're adding connections" patch that added that blk_mq_freeze_queue() call to nbd_add_socket(). And 5.8.0-1038-aws does.

Sorry - I didn’t realize there was a relevant bug fix between those two versions. We have tried a couple configurations at this point the dmesg I sent was from 5.8.0-1038-aws. Naively the error looked similar to what we were seeing on 5.4.0-1038-aws - so I thought they were the same. At this point we’ve tried the following setups,

Nautilus (14.2.22) - Bionic (5.4.0-1037) - Test Passed - Stable when deployed..
Nautilus (14.2.22) - Bionic (5.4.0.1054.37) - Test Fails
Nautilus (15.2.13) - Focal (5.4.0-1038) - Test Fails - Unstable when deployed.
Nautilus (15.2.13) - Focal (5.8.0-1038) - Test Fails - Unstable when deployed.
Nautilus (15.2.13) - Focal (5.11.0-1014 edge) - Test Passed - Unstable when deployed. (issue looks slightly different).

If there are nautilus binaries on focal for rbd-nbd available somewhere - we would be pretty interested in trying them out.

Nautilus (14.2.22) - Bionic (5.4.0-1037) - Test Passed - Stable when deployed..
Nautilus (14.2.22) - Bionic (5.4.0.1054.37) - Test Fails

Right, you are picking up the backport of "nbd: freeze the queue while we're adding connections" patch in 5.4.0-1042.

Nautilus (15.2.13) - Focal (5.4.0-1038) - Test Fails - Unstable when deployed.
Nautilus (15.2.13) - Focal (5.8.0-1038) - Test Fails - Unstable when deployed.

Right, you are picking up the backport of "nbd: freeze the queue while we're adding connections" patch in 5.8.0-1038. 5.4.0-1038 does not have it.

Nautilus (15.2.13) - Focal (5.11.0-1014 edge) - Test Passed - Unstable when deployed. (issue looks slightly different).

Right, although you say that the issue looked slightly different, 5.11.0-1014 includes the original of that patch (it landed in 5.11-rc6).

15.y.z is Octopus, not Nautilus.

We are getting packages from here: https://download.ceph.com/debian-nautilus/dists/focal/main/binary-amd64/

Which doesn’t look like it includes nautilus binaries for rbd-nbd. If there’s an alternative location to get focal binaries we would definitely try that.

I don't think we ever built Nautilus for Focal. If you look at https://download.ceph.com/debian-nautilus/pool/main/c/ceph/ it has always been Xenial and Bionic (whether ceph-*, librbd or rbd-nbd).

Thanks for the detailed reply!

So this definitely sounds like a kernel issue and related to this patch. Is there a bug in that patch that we are hitting - or is this working as intended and something we should be working around for our particular use case.

The patch is probably working as intended. The problem is that it added a synchronization point to the "rbd-nbd map" sequence -- it tries to flush the I/O queue while holding the lock (nbd->config_lock mutex). Given "Possible stuck request" messages in the log it is clear that the queue wasn't getting flushed. Repeated attempts to execute "rbd-nbd map" or any other command on that image then hung trying to get the lock.

As for why the queue wasn't getting flushed, based on the attached dmesg these might be related:

[1816055.259936] blk_update_request: I/O error, dev nbd11, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[1816055.259939] Buffer I/O error on dev nbd11, logical block 0, async page read
[1816055.262453] blk_update_request: I/O error, dev nbd11, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[1816055.262455] Buffer I/O error on dev nbd11, logical block 0, async page read
[1816055.262465] Buffer I/O error on dev nbd11, logical block 0, async page read
[1816055.596192] block nbd11: Attempted send on invalid socket

It is hard to tell because other devices also reported similar errors. I would concentrate on these I/O errors and stuck requests: try to investigate where they are coming from and see if you can get rid of them. Check if the cluster has reported any slow requests, try a newer kernel (you said that 5.11 passes your smoke test but fails in production in a different fashion, perhaps try 5.14 from ppa?), etc.

The other question is how does it even get into a situation where "rbd-nbd map" encounters a non-empty I/O queue? Could it be that your script sometimes attempts to map the same image twice?

Hello!

So we decided that part of our woes might have been related to our old versions of ceph and rbd-nbd. So we prioritized updating our service to use a more up to date version of ceph our new configuration is this,

kernel
5.4.0-1080-aws

ubuntu
20.04.5 LTS (Focal Fossa)

ceph client
$ ceph --version
ceph version 16.2.10 (45fa1a083152e41a408d15505f594ec5f1b4fe17) pacific (stable)

ceph server
$ ceph tell osd.* version
osd.0: {
    "version": "16.2.7",
    "release": "pacific",
    "release_type": "stable" 
}
osd.1: {
    "version": "16.2.7",
    "release": "pacific",
    "release_type": "stable" 
}
osd.2: {
    "version": "16.2.7",
    "release": "pacific",
    "release_type": "stable" 
}

In general this has proven more stable than our original configuration using the same kernel. But we are still seeing a similar error consistently on one of our services. We have reproduced this in a couple ways the most consistent way is by doing the following series of commands twice for the same image name,

• rbd -c ${CONF_FILE} create --size 1G --pool ${CEPH_POOL} ${VOLUME}
• rbd-nbd -c ${CONF_FILE} map ${CEPH_POOL}/${VOLUME}
• mkfs -t ext4 -E nodiscard,lazy_itable_init=1,lazy_journal_init=1 $device
• mkdir -p /tmp/${VOLUME}
• mount $device /tmp/${VOLUME}
• umount $device
• rbd-nbd -c ${CONF_FILE} unmap ${CEPH_POOL}/${VOLUME}
• rbd -c ${CONF_FILE} remove --pool ${CEPH_POOL} ${VOLUME}

When this is done in parallel (n=5). This pretty consistently results in a stuck process.

$ ps aux | grep nbd
lucid     7351  0.0  0.0   6308   732 pts/1    S+   13:27   0:00 grep --color=auto nbd
root     20055  0.0  0.0      0     0 ?        I<   13:21   0:00 [kworker/u9:2-knbd0-recv]
root     22229  0.0  0.0      0     0 ?        I<   13:21   0:00 [kworker/u9:4-knbd3-recv]
root     29246  0.0  0.0      0     0 ?        I<   13:22   0:00 [knbd1-recv]
root     29627  0.0  0.1 1060384 19320 pts/0   Dl   13:22   0:00 rbd-nbd -c /etc/ceph/ceph.1f7cd679-2fb0-4fd1-97af-49bb17d5ea76.conf map rbd0/ceph-test-image-4-2
root     29672  0.0  0.1 1060304 20796 pts/0   Dl   13:22   0:00 rbd-nbd -c /etc/ceph/ceph.1f7cd679-2fb0-4fd1-97af-49bb17d5ea76.conf map rbd0/ceph-test-image-5-2
root     29709  0.0  0.1 1060400 20340 pts/0   Dl   13:22   0:00 rbd-nbd -c /etc/ceph/ceph.1f7cd679-2fb0-4fd1-97af-49bb17d5ea76.conf map rbd0/ceph-test-image-3-2

This is quite similar to when we catch this in the wild. There’s usually a single knbdXXX-recv process which is implicated in the dmesg output and a number of rbd-nbd processes related to that particular nbd. When it gets into this state it seems to block all other rbd-nbd commands - and the host will build up quite a few stuck messages trying to operate on other images.

Stack for each rbd-nbd process:

ceph-test-image-3-2
$ sudo cat /proc/29709/stack
[<0>] __blkdev_get+0x79/0x570
[<0>] blkdev_get+0xdd/0x130
[<0>] blkdev_open+0x87/0xa0
[<0>] do_dentry_open+0x1df/0x3a0
[<0>] vfs_open+0x2f/0x40
[<0>] path_openat+0x2f9/0x16a0
[<0>] do_filp_open+0x9b/0x110
[<0>] do_sys_open+0x1ba/0x2e0
[<0>] __x64_sys_openat+0x20/0x30
[<0>] do_syscall_64+0x57/0x190
[<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9

ceph-test-image-4-2
$ sudo cat /proc/29627/stack
[<0>] blk_mq_freeze_queue_wait+0x4b/0xb0
[<0>] blk_mq_freeze_queue+0x1a/0x20
[<0>] nbd_add_socket+0x5e/0x1f0 [nbd]
[<0>] nbd_ioctl+0x287/0x3e0 [nbd]
[<0>] blkdev_ioctl+0x91f/0x9f0
[<0>] block_ioctl+0x3d/0x50
[<0>] do_vfs_ioctl+0xa9/0x640
[<0>] ksys_ioctl+0x75/0x80
[<0>] __x64_sys_ioctl+0x1a/0x20
[<0>] do_syscall_64+0x57/0x190
[<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9

ceph-test-image-5-2
$ sudo cat /proc/29672/stack
[<0>] blk_queue_enter+0xfc/0x200
[<0>] generic_make_request+0xea/0x300
[<0>] submit_bio+0x46/0x1c0
[<0>] submit_bh_wbc+0x182/0x1b0
[<0>] block_read_full_page+0x234/0x390
[<0>] blkdev_readpage+0x18/0x20
[<0>] do_read_cache_page+0x384/0x7c0
[<0>] read_cache_page+0x12/0x20
[<0>] read_dev_sector+0x27/0xd0
[<0>] read_lba+0x130/0x220
[<0>] efi_partition+0x131/0x770
[<0>] check_partition+0x13f/0x250
[<0>] rescan_partitions+0xaf/0x360
[<0>] bdev_disk_changed+0x5a/0x60
[<0>] __blkdev_get+0x4bd/0x570
[<0>] blkdev_get+0xdd/0x130
[<0>] blkdev_open+0x87/0xa0
[<0>] do_dentry_open+0x1df/0x3a0
[<0>] vfs_open+0x2f/0x40
[<0>] path_openat+0x2f9/0x16a0
[<0>] do_filp_open+0x9b/0x110
[<0>] do_sys_open+0x1ba/0x2e0
[<0>] __x64_sys_openat+0x20/0x30
[<0>] do_syscall_64+0x57/0x190
[<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9

Stack for each recv process:

kworker/u9:2-knbd0-recv
$ sudo cat /proc/20055/stack
[<0>] worker_thread+0xbc/0x410
[<0>] kthread+0x121/0x140
[<0>] ret_from_fork+0x35/0x40

kworker/u9:4-knbd3-recv
$ sudo cat /proc/22229/stack
[<0>] worker_thread+0xbc/0x410
[<0>] kthread+0x121/0x140
[<0>] ret_from_fork+0x35/0x40

knbd1-recv
$ sudo cat /proc/29246/stack
[<0>] rescuer_thread+0x2ef/0x370
[<0>] kthread+0x121/0x140
[<0>] ret_from_fork+0x35/0x40

dmesg output

[  740.148985] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null)
[  740.201622] block nbd0: NBD_DISCONNECT
[  740.201737] block nbd0: Disconnected due to user request.
[  742.105915] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null)
[  742.163490] block nbd0: NBD_DISCONNECT
[  742.163648] block nbd0: Disconnected due to user request.
[  745.246400] block nbd0: Device being setup by another task
[  745.371400] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null)
[  745.399184] EXT4-fs (nbd1): mounted filesystem with ordered data mode. Opts: (null)
[  745.415717] block nbd0: NBD_DISCONNECT
[  745.415826] block nbd0: Disconnected due to user request.
[  745.457019] block nbd1: NBD_DISCONNECT
[  745.457190] block nbd1: Disconnected due to user request.
[  749.114388] block nbd0: Device being setup by another task
[  749.148754] block nbd0: NBD_DISCONNECT
[  749.150304] block nbd0: Disconnected due to user request.
[  749.150436] blk_update_request: I/O error, dev nbd0, sector 2096128 op 0x0:(READ) flags 0x80700 phys_seg 4 prio class 0
[  749.156094] blk_update_request: I/O error, dev nbd0, sector 2096168 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
[  749.161525] blk_update_request: I/O error, dev nbd0, sector 2096184 op 0x0:(READ) flags 0x80700 phys_seg 2 prio class 0
[  749.167198] blk_update_request: I/O error, dev nbd0, sector 2096208 op 0x0:(READ) flags 0x80700 phys_seg 4 prio class 0
[  749.172206] blk_update_request: I/O error, dev nbd0, sector 2096248 op 0x0:(READ) flags 0x80700 phys_seg 21 prio class 0
[  749.177500] blk_update_request: I/O error, dev nbd0, sector 2096424 op 0x0:(READ) flags 0x80700 phys_seg 6 prio class 0
[  749.183019] blk_update_request: I/O error, dev nbd0, sector 2096480 op 0x0:(READ) flags 0x80700 phys_seg 10 prio class 0
[  749.187814] blk_update_request: I/O error, dev nbd0, sector 2096568 op 0x0:(READ) flags 0x80700 phys_seg 9 prio class 0
[  749.198548] blk_update_request: I/O error, dev nbd0, sector 2096128 op 0x0:(READ) flags 0x0 phys_seg 8 prio class 0
[  749.215741] Buffer I/O error on dev nbd0, logical block 2096128, async page read
[  749.226969] Buffer I/O error on dev nbd0, logical block 2096129, async page read
[  749.235786] Buffer I/O error on dev nbd0, logical block 2096130, async page read
[  749.242839] Buffer I/O error on dev nbd0, logical block 2096131, async page read
[  749.250662] Buffer I/O error on dev nbd0, logical block 2096132, async page read
[  749.258753] Buffer I/O error on dev nbd0, logical block 2096133, async page read
[  749.266871] Buffer I/O error on dev nbd0, logical block 2096134, async page read
[  749.273469] Buffer I/O error on dev nbd0, logical block 2096135, async page read
[  749.282450] blk_update_request: I/O error, dev nbd0, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[  749.298522] Buffer I/O error on dev nbd0, logical block 0, async page read
[  749.305006] Buffer I/O error on dev nbd0, logical block 1, async page read
[  749.311101] ldm_validate_partition_table(): Disk read failed.
[  749.311212] Dev nbd0: unable to read RDB block 0
[  749.315767]  nbd0: unable to read partition table
[  749.342840] EXT4-fs (nbd1): mounted filesystem with ordered data mode. Opts: (null)
[  749.412485] block nbd1: NBD_DISCONNECT
[  749.412734] block nbd1: Disconnected due to user request.
[  749.811610] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null)
[  749.857304] block nbd0: NBD_DISCONNECT
[  749.857418] block nbd0: Disconnected due to user request.
[  750.043276] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null)
[  750.101844] block nbd0: NBD_DISCONNECT
[  750.101941] block nbd0: Disconnected due to user request.
[  751.378387] block nbd0: Device being setup by another task
[  751.530774] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null)
[  751.544975] EXT4-fs (nbd1): mounted filesystem with ordered data mode. Opts: (null)
[  751.584228] block nbd0: NBD_DISCONNECT
[  751.584349] block nbd0: Disconnected due to user request.
[  751.598118] block nbd1: NBD_DISCONNECT
[  751.598211] block nbd1: Disconnected due to user request.
[  752.070393] block nbd0: Device being setup by another task
[  752.200260] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null)
[  752.222175] EXT4-fs (nbd1): mounted filesystem with ordered data mode. Opts: (null)
[  752.245984] block nbd0: NBD_DISCONNECT
[  752.246141] block nbd0: Disconnected due to user request.
[  752.263661] block nbd1: NBD_DISCONNECT
[  752.263780] block nbd1: Disconnected due to user request.
[  753.562399] block nbd0: Device being setup by another task
[  753.696131] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null)
[  753.734899] EXT4-fs (nbd1): mounted filesystem with ordered data mode. Opts: (null)
[  753.747409] block nbd0: NBD_DISCONNECT
[  753.747496] block nbd0: Disconnected due to user request.
[  753.778737] block nbd1: NBD_DISCONNECT
[  753.778851] block nbd1: Disconnected due to user request.
[  754.306386] block nbd0: Device being setup by another task
[  754.430231] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null)
[  754.458573] EXT4-fs (nbd1): mounted filesystem with ordered data mode. Opts: (null)
[  754.477765] block nbd0: NBD_DISCONNECT
[  754.477939] block nbd0: Disconnected due to user request.
[  754.509250] block nbd1: NBD_DISCONNECT
[  754.509373] block nbd1: Disconnected due to user request.
[  755.530392] block nbd0: Device being setup by another task
[  755.675447] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null)
[  755.720513] block nbd0: NBD_DISCONNECT
[  755.720625] block nbd0: Disconnected due to user request.
[  755.723619] EXT4-fs (nbd1): mounted filesystem with ordered data mode. Opts: (null)
[  755.783065] block nbd1: NBD_DISCONNECT
[  755.783175] block nbd1: Disconnected due to user request.
[  756.298444] block nbd0: Device being setup by another task
[  756.335855] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null)
[  756.382226] block nbd0: NBD_DISCONNECT
[  756.382313] block nbd0: Disconnected due to user request.
[  756.437128] EXT4-fs (nbd1): mounted filesystem with ordered data mode. Opts: (null)
[  756.481980] block nbd1: NBD_DISCONNECT
[  756.482160] block nbd1: Disconnected due to user request.
[  758.874451] block nbd0: Device being setup by another task
[  758.906416] block nbd0: Device being setup by another task
[  758.930445] block nbd1: Device being setup by another task
[  758.938497] block nbd0: Device being setup by another task
[  758.962417] block nbd0: Device being setup by another task
[  758.964249] block nbd1: Device being setup by another task
[  758.990516] block nbd1: Device being setup by another task
[  758.990518] block nbd2: Device being setup by another task
[  759.022468] block nbd2: Device being setup by another task
[  759.050473] block nbd3: Device being setup by another task
[  759.122582] EXT4-fs (nbd1): mounted filesystem with ordered data mode. Opts: (null)
[  759.126359] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null)
[  759.134863] EXT4-fs (nbd2): mounted filesystem with ordered data mode. Opts: (null)
[  759.198876] block nbd2: NBD_DISCONNECT
[  759.198972] block nbd2: Disconnected due to user request.
[  759.202601] block nbd1: NBD_DISCONNECT
[  759.203092] block nbd0: NBD_DISCONNECT
[  759.203307] block nbd1: Disconnected due to user request.
[  759.203623] block nbd0: Disconnected due to user request.
[  759.204388] EXT4-fs (nbd3): mounted filesystem with ordered data mode. Opts: (null)
[  759.267929] EXT4-fs (nbd4): mounted filesystem with ordered data mode. Opts: (null)
[  759.287033] block nbd3: NBD_DISCONNECT
[  759.287384] block nbd3: Disconnected due to user request.
[  759.342434] block nbd4: NBD_DISCONNECT
[  759.342578] block nbd4: Disconnected due to user request.
[  759.810436] block nbd0: Device being setup by another task
[  759.898413] block nbd0: Device being setup by another task
[  759.918413] block nbd0: Device being setup by another task
[  759.927327] block nbd1: Device being setup by another task
[  759.942407] block nbd0: Device being setup by another task
[  759.954419] block nbd1: Device being setup by another task
[  759.978456] block nbd1: Device being setup by another task
[  759.995305] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null)
[  759.999721] EXT4-fs (nbd1): mounted filesystem with ordered data mode. Opts: (null)
[  760.002448] block nbd2: Device being setup by another task
[  760.030453] block nbd2: Device being setup by another task
[  760.047161] block nbd0: NBD_DISCONNECT
[  760.047314] block nbd0: Disconnected due to user request.
[  760.054436] block nbd3: Device being setup by another task
[  760.058643] block nbd1: NBD_DISCONNECT
[  760.058910] block nbd1: Disconnected due to user request.
[  760.162645] EXT4-fs (nbd2): mounted filesystem with ordered data mode. Opts: (null)
[  760.210342] EXT4-fs (nbd3): mounted filesystem with ordered data mode. Opts: (null)
[  760.215101] block nbd2: NBD_DISCONNECT
[  760.215214] block nbd2: Disconnected due to user request.
[  760.255747] EXT4-fs (nbd4): mounted filesystem with ordered data mode. Opts: (null)
[  760.279125] block nbd3: NBD_DISCONNECT
[  760.279261] block nbd3: Disconnected due to user request.
[  760.313357] block nbd4: NBD_DISCONNECT
[  760.313452] block nbd4: Disconnected due to user request.
[  761.870447] block nbd0: Device being setup by another task
[  761.898409] block nbd0: Device being setup by another task
[  761.922441] block nbd0: Device being setup by another task
[  761.922453] block nbd1: Device being setup by another task
[  761.946412] block nbd0: Device being setup by another task
[  761.978517] block nbd1: Device being setup by another task
[  762.002565] block nbd1: Device being setup by another task
[  762.003127] block nbd2: Device being setup by another task
[  762.006844] block nbd1: NBD_DISCONNECT
[  762.011450] block nbd1: Disconnected due to user request.
[  762.011943] print_req_error: 99 callbacks suppressed
[  762.011944] blk_update_request: I/O error, dev nbd1, sector 512 op 0x0:(READ) flags 0x84700 phys_seg 9 prio class 0
[  762.021583] blk_update_request: I/O error, dev nbd1, sector 768 op 0x0:(READ) flags 0x80700 phys_seg 10 prio class 0
[  762.030476] blk_update_request: I/O error, dev nbd1, sector 512 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[  762.038792] buffer_io_error: 102 callbacks suppressed
[  762.038794] Buffer I/O error on dev nbd1, logical block 512, async page read
[  762.042467] block nbd2: Device being setup by another task
[  762.044478] blk_update_request: I/O error, dev nbd1, sector 513 op 0x0:(READ) flags 0x0 phys_seg 7 prio class 0
[  762.059237] Buffer I/O error on dev nbd1, logical block 513, async page read
[  762.064992] Buffer I/O error on dev nbd1, logical block 514, async page read
[  762.073372] Buffer I/O error on dev nbd1, logical block 515, async page read
[  762.087345] Buffer I/O error on dev nbd1, logical block 516, async page read
[  762.093039] Buffer I/O error on dev nbd1, logical block 517, async page read
[  762.097908] Buffer I/O error on dev nbd1, logical block 518, async page read
[  762.098218] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null)
[  762.102712] Buffer I/O error on dev nbd1, logical block 519, async page read
[  762.110035] blk_update_request: I/O error, dev nbd1, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[  762.118447] Buffer I/O error on dev nbd1, logical block 0, async page read
[  762.123420] blk_update_request: I/O error, dev nbd1, sector 1 op 0x0:(READ) flags 0x0 phys_seg 7 prio class 0
[  762.131631] Buffer I/O error on dev nbd1, logical block 1, async page read
[  762.137565] blk_update_request: I/O error, dev nbd1, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[  762.145622] blk_update_request: I/O error, dev nbd1, sector 1 op 0x0:(READ) flags 0x0 phys_seg 7 prio class 0
[  762.151869] EXT4-fs (nbd2): mounted filesystem with ordered data mode. Opts: (null)
[  762.153437] blk_update_request: I/O error, dev nbd1, sector 0 op 0x0:(READ) flags 0x0 phys_seg 8 prio class 0
[  762.158460] block nbd3: Device being setup by another task
[  762.162338] ldm_validate_partition_table(): Disk read failed.
[  762.166349] blk_update_request: I/O error, dev nbd1, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[  762.174966] Dev nbd1: unable to read RDB block 0
[  762.177901] block nbd0: NBD_DISCONNECT
[  762.179253] block nbd0: Disconnected due to user request.
[  762.179945]  nbd1: unable to read partition table
[  762.228396] block nbd2: NBD_DISCONNECT
[  762.228526] block nbd2: Disconnected due to user request.
[  762.305853] EXT4-fs (nbd3): mounted filesystem with ordered data mode. Opts: (null)
[  762.327258] EXT4-fs (nbd4): mounted filesystem with ordered data mode. Opts: (null)
[  762.368113] block nbd3: NBD_DISCONNECT
[  762.368280] block nbd3: Disconnected due to user request.
[  762.398405] block nbd4: NBD_DISCONNECT
[  762.398516] block nbd4: Disconnected due to user request.
[  762.714413] block nbd0: Device being setup by another task
[  762.775462] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null)
[  762.790521] block nbd0: Device being setup by another task
[  762.822432] block nbd1: Device being setup by another task
[  762.837068] block nbd0: NBD_DISCONNECT
[  762.837185] block nbd0: Disconnected due to user request.
[  762.948183] EXT4-fs (nbd1): mounted filesystem with ordered data mode. Opts: (null)
[  762.980475] EXT4-fs (nbd2): mounted filesystem with ordered data mode. Opts: (null)
[  762.982464] block nbd0: Device being setup by another task
[  763.006402] block nbd1: Device being setup by another task
[  763.010530] block nbd1: NBD_DISCONNECT
[  763.010636] block nbd1: Disconnected due to user request.
[  763.034439] block nbd2: Device being setup by another task
[  763.038823] block nbd2: NBD_DISCONNECT
[  763.038970] block nbd2: Disconnected due to user request.
[  763.120833] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null)
[  763.173651] block nbd0: NBD_DISCONNECT
[  763.173763] block nbd0: Disconnected due to user request.
[  763.208737] EXT4-fs (nbd3): mounted filesystem with ordered data mode. Opts: (null)
[  763.256202] block nbd3: NBD_DISCONNECT
[  763.256307] block nbd3: Disconnected due to user request.
[  764.698416] block nbd0: Device being setup by another task
[  764.722417] block nbd0: Device being setup by another task
[  764.754428] block nbd0: Device being setup by another task
[  764.754964] block nbd1: Device being setup by another task
[  764.782404] block nbd0: Device being setup by another task
[  764.790433] block nbd1: Device being setup by another task
[  764.814455] block nbd1: Device being setup by another task
[  764.814588] block nbd2: Device being setup by another task
[  764.854445] block nbd2: Device being setup by another task
[  764.874438] block nbd3: Device being setup by another task
[  764.921050] EXT4-fs (nbd1): mounted filesystem with ordered data mode. Opts: (null)
[  764.989313] EXT4-fs (nbd2): mounted filesystem with ordered data mode. Opts: (null)
[  765.002930] block nbd1: NBD_DISCONNECT
[  765.003038] block nbd1: Disconnected due to user request.
[  765.033183] EXT4-fs (nbd3): mounted filesystem with ordered data mode. Opts: (null)
[  765.050455] block nbd0: Device being setup by another task
[  765.059746] EXT4-fs (nbd4): mounted filesystem with ordered data mode. Opts: (null)
[  765.063008] block nbd2: NBD_DISCONNECT
[  765.063151] block nbd2: Disconnected due to user request.
[  765.129291] block nbd3: NBD_DISCONNECT
[  765.129529] block nbd3: Disconnected due to user request.
[  765.160820] block nbd4: NBD_DISCONNECT
[  765.160949] block nbd4: Disconnected due to user request.
[  765.208713] EXT4-fs (nbd1): mounted filesystem with ordered data mode. Opts: (null)
[  765.275602] block nbd1: NBD_DISCONNECT
[  765.275707] block nbd1: Disconnected due to user request.
[  765.598470] block nbd0: Device being setup by another task
[  765.646418] block nbd0: Device being setup by another task
[  765.670407] block nbd0: Device being setup by another task
[  765.674417] block nbd1: Device being setup by another task
[  765.706451] block nbd1: Device being setup by another task
[  765.722411] block nbd0: Device being setup by another task
[  765.730401] block nbd2: Device being setup by another task
[  765.746436] block nbd1: Device being setup by another task
[  765.774405] block nbd2: Device being setup by another task
[  765.806617] block nbd3: Device being setup by another task
[  765.866077] EXT4-fs (nbd1): mounted filesystem with ordered data mode. Opts: (null)
[  765.892045] EXT4-fs (nbd2): mounted filesystem with ordered data mode. Opts: (null)
[  765.915198] EXT4-fs (nbd3): mounted filesystem with ordered data mode. Opts: (null)
[  765.932968] block nbd1: NBD_DISCONNECT
[  765.933089] block nbd1: Disconnected due to user request.
[  765.956114] block nbd2: NBD_DISCONNECT
[  765.956229] block nbd2: Disconnected due to user request.
[  765.976869] block nbd3: NBD_DISCONNECT
[  765.977005] block nbd3: Disconnected due to user request.
[  765.997852] EXT4-fs (nbd4): mounted filesystem with ordered data mode. Opts: (null)
[  766.064636] block nbd4: NBD_DISCONNECT
[  766.064719] block nbd4: Disconnected due to user request.
[  767.398406] block nbd0: Device being setup by another task
[  767.422415] block nbd0: Device being setup by another task
[  767.450405] block nbd0: Device being setup by another task
[  767.450421] block nbd1: Device being setup by another task
[  767.470404] block nbd0: Device being setup by another task
[  767.478412] block nbd1: Device being setup by another task
[  767.486418] block nbd0: Device being setup by another task
[  767.498429] block nbd1: Device being setup by another task
[  767.506421] block nbd2: Device being setup by another task
[  767.522417] block nbd1: Device being setup by another task
[  767.530412] block nbd2: Device being setup by another task
[  767.558420] block nbd2: Device being setup by another task
[  767.558478] block nbd3: Device being setup by another task
[  767.582425] block nbd3: Device being setup by another task
[  767.618453] block nbd4: Device being setup by another task
[  767.691718] EXT4-fs (nbd1): mounted filesystem with ordered data mode. Opts: (null)
[  767.761011] block nbd1: NBD_DISCONNECT
[  767.761146] block nbd1: Disconnected due to user request.
[  767.769457] EXT4-fs (nbd2): mounted filesystem with ordered data mode. Opts: (null)
[  767.782605] EXT4-fs (nbd3): mounted filesystem with ordered data mode. Opts: (null)
[  767.841085] EXT4-fs (nbd4): mounted filesystem with ordered data mode. Opts: (null)
[  767.844926] block nbd2: NBD_DISCONNECT
[  767.845044] block nbd2: Disconnected due to user request.
[  767.858627] EXT4-fs (nbd5): mounted filesystem with ordered data mode. Opts: (null)
[  767.862693] block nbd3: NBD_DISCONNECT
[  767.862774] block nbd3: Disconnected due to user request.
[  767.926292] block nbd4: NBD_DISCONNECT
[  767.926529] block nbd4: Disconnected due to user request.
[  767.946180] block nbd5: NBD_DISCONNECT
[  767.946352] block nbd5: Disconnected due to user request.
[  768.487932] block nbd0: Device being setup by another task
[  768.554539] block nbd0: Device being setup by another task
[  768.578451] block nbd1: Device being setup by another task
[  768.586409] block nbd0: Device being setup by another task
[  768.610438] block nbd1: Device being setup by another task
[  768.634421] block nbd2: Device being setup by another task
[  768.642422] block nbd0: Device being setup by another task
[  768.666420] block nbd1: Device being setup by another task
[  768.690423] block nbd2: Device being setup by another task
[  768.694742] EXT4-fs (nbd1): mounted filesystem with ordered data mode. Opts: (null)
[  768.712813] block nbd3: Device being setup by another task
[  768.758839] block nbd1: NBD_DISCONNECT
[  768.759077] block nbd1: Disconnected due to user request.
[  768.792254] EXT4-fs (nbd2): mounted filesystem with ordered data mode. Opts: (null)
[  768.799272] EXT4-fs (nbd3): mounted filesystem with ordered data mode. Opts: (null)
[  768.854031] block nbd2: NBD_DISCONNECT
[  768.854121] block nbd2: Disconnected due to user request.
[  768.862499] EXT4-fs (nbd4): mounted filesystem with ordered data mode. Opts: (null)
[  768.869003] block nbd3: NBD_DISCONNECT
[  768.869163] block nbd3: Disconnected due to user request.
[  768.916643] block nbd4: NBD_DISCONNECT
[  768.916810] block nbd4: Disconnected due to user request.
[  770.322395] block nbd0: Device being setup by another task
[  770.346408] block nbd0: Device being setup by another task
[  770.370411] block nbd0: Device being setup by another task
[  770.370432] block nbd1: Device being setup by another task
[  770.398420] block nbd0: Device being setup by another task
[  770.398454] block nbd1: Device being setup by another task
[  770.426424] block nbd0: Device being setup by another task
[  770.430496] block nbd1: Device being setup by another task
[  770.438450] block nbd2: Device being setup by another task
[  770.454408] block nbd1: Device being setup by another task
[  770.462439] block nbd2: Device being setup by another task
[  770.502433] block nbd2: Device being setup by another task
[  770.502467] block nbd3: Device being setup by another task
[  770.546426] block nbd3: Device being setup by another task
[  770.581786] block nbd4: Device being setup by another task
[  770.613751] EXT4-fs (nbd1): mounted filesystem with ordered data mode. Opts: (null)
[  770.692012] block nbd1: NBD_DISCONNECT
[  770.692154] block nbd1: Disconnected due to user request.
[  770.736578] EXT4-fs (nbd2): mounted filesystem with ordered data mode. Opts: (null)
[  770.780398] EXT4-fs (nbd4): mounted filesystem with ordered data mode. Opts: (null)
[  770.791583] EXT4-fs (nbd3): mounted filesystem with ordered data mode. Opts: (null)
[  770.835819] block nbd2: NBD_DISCONNECT
[  770.835935] block nbd2: Disconnected due to user request.
[  770.868918] EXT4-fs (nbd5): mounted filesystem with ordered data mode. Opts: (null)
[  770.876607] block nbd4: NBD_DISCONNECT
[  770.876772] block nbd4: Disconnected due to user request.
[  770.878482] block nbd3: NBD_DISCONNECT
[  770.878576] block nbd3: Disconnected due to user request.
[  770.959014] block nbd5: NBD_DISCONNECT
[  770.959115] block nbd5: Disconnected due to user request.
[  771.350408] block nbd0: Device being setup by another task
[  771.554405] block nbd0: Device being setup by another task
[  771.582407] block nbd1: Device being setup by another task
[  771.582407] block nbd0: Device being setup by another task
[  771.598396] block nbd0: Device being setup by another task
[  771.606481] block nbd1: Device being setup by another task
[  771.638438] block nbd1: Device being setup by another task
[  771.642472] block nbd2: Device being setup by another task
[  771.659399] EXT4-fs (nbd1): mounted filesystem with ordered data mode. Opts: (null)
[  771.670404] block nbd2: Device being setup by another task
[  771.702425] block nbd3: Device being setup by another task
[  771.716507] block nbd1: NBD_DISCONNECT
[  771.716716] block nbd1: Disconnected due to user request.
[  771.810726] EXT4-fs (nbd2): mounted filesystem with ordered data mode. Opts: (null)
[  771.841280] EXT4-fs (nbd3): mounted filesystem with ordered data mode. Opts: (null)
[  771.855920] EXT4-fs (nbd4): mounted filesystem with ordered data mode. Opts: (null)
[  771.863032] block nbd2: NBD_DISCONNECT
[  771.863147] block nbd2: Disconnected due to user request.
[  771.897169] block nbd3: NBD_DISCONNECT
[  771.897270] block nbd3: Disconnected due to user request.
[  771.916182] block nbd4: NBD_DISCONNECT
[  771.916282] block nbd4: Disconnected due to user request.
[  773.346418] block nbd0: Device being setup by another task
[  773.370395] block nbd0: Device being setup by another task
[  773.398406] block nbd0: Device being setup by another task
[  773.406417] block nbd1: Device being setup by another task
[  773.414430] block nbd0: Device being setup by another task
[  773.430433] block nbd1: Device being setup by another task
[  773.438502] block nbd0: Device being setup by another task
[  773.454429] block nbd1: Device being setup by another task
[  773.458403] block nbd2: Device being setup by another task
[  773.482438] block nbd1: Device being setup by another task
[  773.482502] block nbd2: Device being setup by another task
[  773.510413] block nbd3: Device being setup by another task
[  773.518412] block nbd2: Device being setup by another task
[  773.542407] block nbd3: Device being setup by another task
[  773.688806] block nbd4: Device being setup by another task
[  773.724999] EXT4-fs (nbd1): mounted filesystem with ordered data mode. Opts: (null)
[  773.726332] EXT4-fs (nbd2): mounted filesystem with ordered data mode. Opts: (null)
[  773.752553] EXT4-fs (nbd3): mounted filesystem with ordered data mode. Opts: (null)
[  773.757299] EXT4-fs (nbd4): mounted filesystem with ordered data mode. Opts: (null)
[  773.812146] block nbd2: NBD_DISCONNECT
[  773.812256] block nbd2: Disconnected due to user request.
[  773.819981] block nbd1: NBD_DISCONNECT
[  773.820697] block nbd1: Disconnected due to user request.
[  773.832688] block nbd3: NBD_DISCONNECT
[  773.832805] block nbd3: Disconnected due to user request.
[  773.843061] block nbd4: NBD_DISCONNECT
[  773.843183] block nbd4: Disconnected due to user request.
[  774.004293] EXT4-fs (nbd5): mounted filesystem with ordered data mode. Opts: (null)
[  774.067077] block nbd5: NBD_DISCONNECT
[  774.067193] block nbd5: Disconnected due to user request.
[  774.390404] block nbd0: Device being setup by another task
[  774.438414] block nbd0: Device being setup by another task
[  774.462413] block nbd1: Device being setup by another task
[  774.502413] block nbd0: Device being setup by another task
[  774.530420] block nbd1: Device being setup by another task
[  774.554418] block nbd2: Device being setup by another task
[  774.570402] EXT4-fs (nbd1): mounted filesystem with ordered data mode. Opts: (null)
[  774.632048] block nbd1: NBD_DISCONNECT
[  774.632164] block nbd1: Disconnected due to user request.
[  774.658462] block nbd0: Device being setup by another task
[  774.706942] EXT4-fs (nbd2): mounted filesystem with ordered data mode. Opts: (null)
[  774.779236] block nbd2: NBD_DISCONNECT
[  774.779363] block nbd2: Disconnected due to user request.
[  774.802568] EXT4-fs (nbd3): mounted filesystem with ordered data mode. Opts: (null)
[  774.845434] EXT4-fs (nbd1): mounted filesystem with ordered data mode. Opts: (null)
[  774.873040] block nbd3: NBD_DISCONNECT
[  774.873121] block nbd3: Disconnected due to user request.
[  774.901480] block nbd1: NBD_DISCONNECT
[  774.901571] block nbd1: Disconnected due to user request.
[  799.389988] block nbd0: shutting down sockets
[  799.390231] block nbd0: NBD_DISCONNECT
[  799.390234] block nbd0: Send disconnect failed -32
[  808.334624] block nbd0: Device being setup by another task
[  808.370615] block nbd0: Device being setup by another task
[  808.390637] block nbd1: Device being setup by another task
[  808.391024] block nbd0: Device being setup by another task
[  808.418648] block nbd0: Device being setup by another task
[  808.418732] block nbd1: Device being setup by another task
[  808.450644] block nbd1: Device being setup by another task
[  808.450660] block nbd2: Device being setup by another task
[  808.454828] block nbd1: NBD_DISCONNECT
[  808.455049] block nbd1: Disconnected due to user request.
[  808.482740] block nbd2: Device being setup by another task
[  808.510745] block nbd3: Device being setup by another task
[  808.561331] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null)
[  808.638189] block nbd0: NBD_DISCONNECT
[  808.638294] block nbd0: Disconnected due to user request.
[  808.642314] EXT4-fs (nbd2): mounted filesystem with ordered data mode. Opts: (null)
[  808.658654] EXT4-fs (nbd3): mounted filesystem with ordered data mode. Opts: (null)
[  808.688102] EXT4-fs (nbd4): mounted filesystem with ordered data mode. Opts: (null)
[  808.771666] block nbd2: NBD_DISCONNECT
[  808.776188] block nbd2: Disconnected due to user request.
[  808.825206] block nbd3: NBD_DISCONNECT
[  808.828381] block nbd3: Disconnected due to user request.
[  808.829112] block nbd4: NBD_DISCONNECT
[  808.829226] block nbd4: Disconnected due to user request.
[  809.240528] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null)
[  809.296358] block nbd0: NBD_DISCONNECT
[  809.296435] block nbd0: Disconnected due to user request.
[  809.454688] block nbd0: Device being setup by another task
[  809.459387] print_req_error: 75 callbacks suppressed
[  809.459390] blk_update_request: I/O error, dev nbd1, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[  809.467287] buffer_io_error: 102 callbacks suppressed
[  809.467290] Buffer I/O error on dev nbd1, logical block 0, async page read
[  809.472096] blk_update_request: I/O error, dev nbd1, sector 1 op 0x0:(READ) flags 0x0 phys_seg 7 prio class 0
[  809.479790] Buffer I/O error on dev nbd1, logical block 1, async page read
[  809.484608] Buffer I/O error on dev nbd1, logical block 2, async page read
[  809.489393] Buffer I/O error on dev nbd1, logical block 3, async page read
[  809.494239] Buffer I/O error on dev nbd1, logical block 4, async page read
[  809.496411] block nbd0: NBD_DISCONNECT
[  809.499166] Buffer I/O error on dev nbd1, logical block 5, async page read
[  809.500908] block nbd0: Disconnected due to user request.
[  809.504044] Buffer I/O error on dev nbd1, logical block 6, async page read
[  809.504462] blk_update_request: I/O error, dev nbd0, sector 2096640 op 0x0:(READ) flags 0x80700 phys_seg 14 prio class 0
[  809.508909] Buffer I/O error on dev nbd1, logical block 7, async page read
[  809.516926] blk_update_request: I/O error, dev nbd0, sector 2096760 op 0x0:(READ) flags 0x80700 phys_seg 16 prio class 0
[  809.521558] blk_update_request: I/O error, dev nbd1, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[  809.529775] blk_update_request: I/O error, dev nbd0, sector 2096904 op 0x0:(READ) flags 0x80700 phys_seg 15 prio class 0
[  809.537990] Buffer I/O error on dev nbd1, logical block 0, async page read
[  809.537999] blk_update_request: I/O error, dev nbd1, sector 1 op 0x0:(READ) flags 0x0 phys_seg 7 prio class 0
[  809.538000] Buffer I/O error on dev nbd1, logical block 1, async page read
[  809.538145] blk_update_request: I/O error, dev nbd1, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[  809.545629] blk_update_request: I/O error, dev nbd0, sector 2097032 op 0x0:(READ) flags 0x80700 phys_seg 7 prio class 0
[  809.550039] blk_update_request: I/O error, dev nbd1, sector 1 op 0x0:(READ) flags 0x0 phys_seg 7 prio class 0
[  809.550246] ldm_validate_partition_table(): Disk read failed.
[  809.558002] ldm_validate_partition_table(): Disk read failed.
[  809.562455] Dev nbd1: unable to read RDB block 0
[  809.569537] Dev nbd0: unable to read RDB block 0
[  809.577931]  nbd1: unable to read partition table
[  809.584819]  nbd0: unable to read partition table
[  809.606637] block nbd0: Device being setup by another task
[  809.610866] ldm_validate_partition_table(): Disk read failed.
[  809.611001] Dev nbd0: unable to read RDB block 0
[  809.614792]  nbd0: unable to read partition table
[  809.634671] block nbd0: Device being setup by another task
[  809.638896] ldm_validate_partition_table(): Disk read failed.
[  809.639015] Dev nbd0: unable to read RDB block 0
[  809.642761]  nbd0: unable to read partition table
[  838.786766] block nbd1: Possible stuck request 0000000096609e97: control (read@1073737728,4096B). Runtime 30 seconds
[  869.506854] block nbd1: Possible stuck request 0000000096609e97: control (read@1073737728,4096B). Runtime 60 seconds
[  900.226904] block nbd1: Possible stuck request 0000000096609e97: control (read@1073737728,4096B). Runtime 90 seconds
[  930.946959] block nbd1: Possible stuck request 0000000096609e97: control (read@1073737728,4096B). Runtime 120 seconds
[  961.667811] block nbd1: Possible stuck request 0000000096609e97: control (read@1073737728,4096B). Runtime 150 seconds
[  967.811089] INFO: task rbd-nbd:29627 blocked for more than 120 seconds.
[  967.815503]       Not tainted 5.4.0-1080-aws #87~18.04.1-Ubuntu
[  967.819833] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  967.827183] rbd-nbd         D    0 29627  10851 0x00000004
[  967.827185] Call Trace:
[  967.827192]  __schedule+0x292/0x720
[  967.827198]  ? call_rcu+0x10/0x20
[  967.827201]  schedule+0x33/0xa0
[  967.827267]  blk_mq_freeze_queue_wait+0x4b/0xb0
[  967.827272]  ? __wake_up_pollfree+0x40/0x40
[  967.827274]  blk_mq_freeze_queue+0x1a/0x20
[  967.827278]  nbd_add_socket+0x5e/0x1f0 [nbd]
[  967.827281]  nbd_ioctl+0x287/0x3e0 [nbd]
[  967.827284]  ? do_filp_open+0xaf/0x110
[  967.827287]  blkdev_ioctl+0x91f/0x9f0
[  967.827290]  block_ioctl+0x3d/0x50
[  967.827294]  do_vfs_ioctl+0xa9/0x640
[  967.827297]  ? kmem_cache_free+0x294/0x2b0
[  967.827299]  ? putname+0x4c/0x60
[  967.827300]  ksys_ioctl+0x75/0x80
[  967.827301]  __x64_sys_ioctl+0x1a/0x20
[  967.827305]  do_syscall_64+0x57/0x190
[  967.827308]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[  967.827311] RIP: 0033:0x7f613528d3ab
[  967.827316] Code: Bad RIP value.
[  967.827317] RSP: 002b:00007ffe17f38058 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[  967.827320] RAX: ffffffffffffffda RBX: 00007ffe17f38550 RCX: 00007f613528d3ab
[  967.827322] RDX: 0000000000000007 RSI: 000000000000ab00 RDI: 000000000000001f
[  967.827323] RBP: 000055ba43a8a207 R08: 000000000000001f R09: 00007ffe17f37ef0
[  967.827325] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000001
[  967.827326] R13: 0000000000000007 R14: 0000000000000001 R15: 00007ffe17f382c0
[  967.827334] INFO: task rbd-nbd:29672 blocked for more than 120 seconds.
[  967.831614]       Not tainted 5.4.0-1080-aws #87~18.04.1-Ubuntu
[  967.836022] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  967.842462] rbd-nbd         D    0 29672  10851 0x00004004
[  967.842466] Call Trace:
[  967.842474]  __schedule+0x292/0x720
[  967.842476]  schedule+0x33/0xa0
[  967.842480]  blk_queue_enter+0xfc/0x200
[  967.842486]  ? __wake_up_pollfree+0x40/0x40
[  967.842487]  generic_make_request+0xea/0x300
[  967.842492]  ? kmem_cache_alloc+0x180/0x230
[  967.842493]  submit_bio+0x46/0x1c0
[  967.842494]  ? submit_bio+0x46/0x1c0
[  967.842497]  ? bio_add_page+0x83/0x90
[  967.842500]  submit_bh_wbc+0x182/0x1b0
[  967.842502]  block_read_full_page+0x234/0x390
[  967.842504]  ? bdev_evict_inode+0xf0/0xf0
[  967.842506]  blkdev_readpage+0x18/0x20
[  967.842508]  do_read_cache_page+0x384/0x7c0
[  967.842512]  ? prep_new_page+0xfa/0x120
[  967.842514]  ? get_page_from_freelist+0x14ac/0x14c0
[  967.842516]  read_cache_page+0x12/0x20
[  967.842519]  read_dev_sector+0x27/0xd0
[  967.842521]  read_lba+0x130/0x220
[  967.842523]  efi_partition+0x131/0x770
[  967.842526]  ? snprintf+0x45/0x70
[  967.842528]  ? is_gpt_valid.part.7+0x420/0x420
[  967.842529]  check_partition+0x13f/0x250
[  967.842531]  ? check_partition+0x13f/0x250
[  967.842534]  rescan_partitions+0xaf/0x360
[  967.842536]  bdev_disk_changed+0x5a/0x60
[  967.842537]  __blkdev_get+0x4bd/0x570
[  967.842539]  blkdev_get+0xdd/0x130
[  967.842541]  ? blkdev_get_by_dev+0x50/0x50
[  967.842543]  blkdev_open+0x87/0xa0
[  967.842549]  do_dentry_open+0x1df/0x3a0
[  967.842553]  vfs_open+0x2f/0x40
[  967.842556]  path_openat+0x2f9/0x16a0
[  967.842563]  ? ilookup5_nowait+0x7e/0xa0
[  967.842568]  ? try_to_wake_up+0x63/0x5f0
[  967.842571]  ? do_blockdev_direct_IO+0x3f0/0x2ea0
[  967.842573]  ? wake_up_q+0x44/0x80
[  967.842576]  do_filp_open+0x9b/0x110
[  967.842579]  ? __switch_to_asm+0x34/0x70
[  967.842583]  ? __check_object_size+0xdb/0x1b0
[  967.842586]  ? __alloc_fd+0xb2/0x170
[  967.842589]  do_sys_open+0x1ba/0x2e0
[  967.842592]  ? do_sys_open+0x1ba/0x2e0
[  967.842595]  __x64_sys_openat+0x20/0x30
[  967.842600]  do_syscall_64+0x57/0x190
[  967.842603]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[  967.842605] RIP: 0033:0x7fcda64a1b34
[  967.842613] Code: Bad RIP value.
[  967.842614] RSP: 002b:00007fff65648600 EFLAGS: 00000293 ORIG_RAX: 0000000000000101
[  967.842619] RAX: ffffffffffffffda RBX: 00007fff65648b70 RCX: 00007fcda64a1b34
[  967.842619] RDX: 0000000000000002 RSI: 00007fff656488e0 RDI: 00000000ffffff9c
[  967.842622] RBP: 00007fff656488e0 R08: 0000000000000000 R09: 00007fff65648510
[  967.842623] R10: 0000000000000000 R11: 0000000000000293 R12: 0000000000000002
[  967.842626] R13: 0000000000000007 R14: 0000000000000001 R15: 00007fff656488e0
[  967.842634] INFO: task rbd-nbd:29709 blocked for more than 120 seconds.
[  967.847001]       Not tainted 5.4.0-1080-aws #87~18.04.1-Ubuntu
[  967.851291] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  967.857819] rbd-nbd         D    0 29709  10851 0x00000004
[  967.857823] Call Trace:
[  967.857830]  __schedule+0x292/0x720
[  967.857832]  schedule+0x33/0xa0
[  967.857834]  schedule_preempt_disabled+0xe/0x10
[  967.857836]  __mutex_lock.isra.10+0x24c/0x4a0
[  967.857840]  ? disk_map_sector_rcu+0x70/0x70
[  967.857841]  __mutex_lock_slowpath+0x13/0x20
[  967.857843]  ? __mutex_lock_slowpath+0x13/0x20
[  967.857850]  mutex_lock+0x2f/0x40
[  967.857853]  __blkdev_get+0x79/0x570
[  967.857855]  blkdev_get+0xdd/0x130
[  967.857856]  ? blkdev_get_by_dev+0x50/0x50
[  967.857857]  blkdev_open+0x87/0xa0
[  967.857861]  do_dentry_open+0x1df/0x3a0
[  967.857862]  vfs_open+0x2f/0x40
[  967.857865]  path_openat+0x2f9/0x16a0
[  967.857867]  ? ilookup5_nowait+0x7e/0xa0
[  967.857869]  ? _cond_resched+0x19/0x40
[  967.857871]  ? do_blockdev_direct_IO+0x3f0/0x2ea0
[  967.857875]  ? refcount_dec_and_mutex_lock+0x17/0x60
[  967.857876]  do_filp_open+0x9b/0x110
[  967.857878]  ? __check_object_size+0xdb/0x1b0
[  967.857880]  ? __alloc_fd+0xb2/0x170
[  967.857881]  do_sys_open+0x1ba/0x2e0
[  967.857882]  ? do_sys_open+0x1ba/0x2e0
[  967.857885]  ? task_work_run+0x80/0xc0
[  967.857886]  __x64_sys_openat+0x20/0x30
[  967.857889]  do_syscall_64+0x57/0x190
[  967.857891]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[  967.857892] RIP: 0033:0x7f691d1a9b34
[  967.857896] Code: Bad RIP value.
[  967.857897] RSP: 002b:00007ffef0259f90 EFLAGS: 00000293 ORIG_RAX: 0000000000000101
[  967.857898] RAX: ffffffffffffffda RBX: 00007ffef025a500 RCX: 00007f691d1a9b34
[  967.857899] RDX: 0000000000000002 RSI: 00007ffef025a270 RDI: 00000000ffffff9c
[  967.857900] RBP: 00007ffef025a270 R08: 0000000000000000 R09: 00007ffef0259ea0
[  967.857900] R10: 0000000000000000 R11: 0000000000000293 R12: 0000000000000002
[  967.857901] R13: 0000000000000007 R14: 0000000000000001 R15: 00007ffef025a270
[  992.387070] block nbd1: Possible stuck request 0000000096609e97: control (read@1073737728,4096B). Runtime 180 seconds
[ 1023.107109] block nbd1: Possible stuck request 0000000096609e97: control (read@1073737728,4096B). Runtime 210 seconds
[ 1053.831157] block nbd1: Possible stuck request 0000000096609e97: control (read@1073737728,4096B). Runtime 240 seconds
[ 1084.547229] block nbd1: Possible stuck request 0000000096609e97: control (read@1073737728,4096B). Runtime 270 seconds
[ 1088.643297] INFO: task rbd-nbd:29627 blocked for more than 241 seconds.
[ 1088.648752]       Not tainted 5.4.0-1080-aws #87~18.04.1-Ubuntu
[ 1088.653408] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1088.661129] rbd-nbd         D    0 29627  10851 0x00000004
[ 1088.661132] Call Trace:
[ 1088.661140]  __schedule+0x292/0x720
[ 1088.661146]  ? call_rcu+0x10/0x20
[ 1088.661148]  schedule+0x33/0xa0
[ 1088.661152]  blk_mq_freeze_queue_wait+0x4b/0xb0
[ 1088.661154]  ? __wake_up_pollfree+0x40/0x40
[ 1088.661156]  blk_mq_freeze_queue+0x1a/0x20
[ 1088.661161]  nbd_add_socket+0x5e/0x1f0 [nbd]
[ 1088.661164]  nbd_ioctl+0x287/0x3e0 [nbd]
[ 1088.661167]  ? do_filp_open+0xaf/0x110
[ 1088.661170]  blkdev_ioctl+0x91f/0x9f0
[ 1088.661173]  block_ioctl+0x3d/0x50
[ 1088.661174]  do_vfs_ioctl+0xa9/0x640
[ 1088.661177]  ? kmem_cache_free+0x294/0x2b0
[ 1088.661179]  ? putname+0x4c/0x60
[ 1088.661180]  ksys_ioctl+0x75/0x80
[ 1088.661182]  __x64_sys_ioctl+0x1a/0x20
[ 1088.661186]  do_syscall_64+0x57/0x190
[ 1088.661188]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 1088.661190] RIP: 0033:0x7f613528d3ab
[ 1088.661194] Code: Bad RIP value.
[ 1088.661195] RSP: 002b:00007ffe17f38058 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[ 1088.661198] RAX: ffffffffffffffda RBX: 00007ffe17f38550 RCX: 00007f613528d3ab
[ 1088.661199] RDX: 0000000000000007 RSI: 000000000000ab00 RDI: 000000000000001f
[ 1088.661200] RBP: 000055ba43a8a207 R08: 000000000000001f R09: 00007ffe17f37ef0
[ 1088.661202] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000001
[ 1088.661202] R13: 0000000000000007 R14: 0000000000000001 R15: 00007ffe17f382c0
[ 1088.661209] INFO: task rbd-nbd:29672 blocked for more than 241 seconds.
[ 1088.666081]       Not tainted 5.4.0-1080-aws #87~18.04.1-Ubuntu
[ 1088.670935] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1088.678114] rbd-nbd         D    0 29672  10851 0x00004004
[ 1088.678116] Call Trace:
[ 1088.678124]  __schedule+0x292/0x720
[ 1088.678127]  schedule+0x33/0xa0
[ 1088.678130]  blk_queue_enter+0xfc/0x200
[ 1088.678134]  ? __wake_up_pollfree+0x40/0x40
[ 1088.678136]  generic_make_request+0xea/0x300
[ 1088.678138]  ? kmem_cache_alloc+0x180/0x230
[ 1088.678140]  submit_bio+0x46/0x1c0
[ 1088.678141]  ? submit_bio+0x46/0x1c0
[ 1088.678144]  ? bio_add_page+0x83/0x90
[ 1088.678146]  submit_bh_wbc+0x182/0x1b0
[ 1088.678148]  block_read_full_page+0x234/0x390
[ 1088.678150]  ? bdev_evict_inode+0xf0/0xf0
[ 1088.678151]  blkdev_readpage+0x18/0x20
[ 1088.678154]  do_read_cache_page+0x384/0x7c0
[ 1088.678158]  ? prep_new_page+0xfa/0x120
[ 1088.678160]  ? get_page_from_freelist+0x14ac/0x14c0
[ 1088.678161]  read_cache_page+0x12/0x20
[ 1088.678164]  read_dev_sector+0x27/0xd0
[ 1088.678165]  read_lba+0x130/0x220
[ 1088.678171]  efi_partition+0x131/0x770
[ 1088.678174]  ? snprintf+0x45/0x70
[ 1088.678175]  ? is_gpt_valid.part.7+0x420/0x420
[ 1088.678177]  check_partition+0x13f/0x250
[ 1088.678178]  ? check_partition+0x13f/0x250
[ 1088.678180]  rescan_partitions+0xaf/0x360
[ 1088.678183]  bdev_disk_changed+0x5a/0x60
[ 1088.678184]  __blkdev_get+0x4bd/0x570
[ 1088.678186]  blkdev_get+0xdd/0x130
[ 1088.678187]  ? blkdev_get_by_dev+0x50/0x50
[ 1088.678188]  blkdev_open+0x87/0xa0
[ 1088.678193]  do_dentry_open+0x1df/0x3a0
[ 1088.678197]  vfs_open+0x2f/0x40
[ 1088.678199]  path_openat+0x2f9/0x16a0
[ 1088.678202]  ? ilookup5_nowait+0x7e/0xa0
[ 1088.678206]  ? try_to_wake_up+0x63/0x5f0
[ 1088.678211]  ? do_blockdev_direct_IO+0x3f0/0x2ea0
[ 1088.678213]  ? wake_up_q+0x44/0x80
[ 1088.678216]  do_filp_open+0x9b/0x110
[ 1088.678220]  ? __switch_to_asm+0x34/0x70
[ 1088.678223]  ? __check_object_size+0xdb/0x1b0
[ 1088.678227]  ? __alloc_fd+0xb2/0x170
[ 1088.678230]  do_sys_open+0x1ba/0x2e0
[ 1088.678232]  ? do_sys_open+0x1ba/0x2e0
[ 1088.678235]  __x64_sys_openat+0x20/0x30
[ 1088.678240]  do_syscall_64+0x57/0x190
[ 1088.678246]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 1088.678249] RIP: 0033:0x7fcda64a1b34
[ 1088.678253] Code: Bad RIP value.
[ 1088.678254] RSP: 002b:00007fff65648600 EFLAGS: 00000293 ORIG_RAX: 0000000000000101
[ 1088.678257] RAX: ffffffffffffffda RBX: 00007fff65648b70 RCX: 00007fcda64a1b34
[ 1088.678258] RDX: 0000000000000002 RSI: 00007fff656488e0 RDI: 00000000ffffff9c
[ 1088.678262] RBP: 00007fff656488e0 R08: 0000000000000000 R09: 00007fff65648510
[ 1088.678263] R10: 0000000000000000 R11: 0000000000000293 R12: 0000000000000002
[ 1088.678264] R13: 0000000000000007 R14: 0000000000000001 R15: 00007fff656488e0
[ 1088.678272] INFO: task rbd-nbd:29709 blocked for more than 241 seconds.
[ 1088.684053]       Not tainted 5.4.0-1080-aws #87~18.04.1-Ubuntu
[ 1088.688437] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1088.695659] rbd-nbd         D    0 29709  10851 0x00000004
[ 1088.695661] Call Trace:
[ 1088.695668]  __schedule+0x292/0x720
[ 1088.695670]  schedule+0x33/0xa0
[ 1088.695671]  schedule_preempt_disabled+0xe/0x10
[ 1088.695673]  __mutex_lock.isra.10+0x24c/0x4a0
[ 1088.695677]  ? disk_map_sector_rcu+0x70/0x70
[ 1088.695678]  __mutex_lock_slowpath+0x13/0x20
[ 1088.695680]  ? __mutex_lock_slowpath+0x13/0x20
[ 1088.695681]  mutex_lock+0x2f/0x40
[ 1088.695684]  __blkdev_get+0x79/0x570
[ 1088.695686]  blkdev_get+0xdd/0x130
[ 1088.695687]  ? blkdev_get_by_dev+0x50/0x50
[ 1088.695688]  blkdev_open+0x87/0xa0
[ 1088.695692]  do_dentry_open+0x1df/0x3a0
[ 1088.695693]  vfs_open+0x2f/0x40
[ 1088.695696]  path_openat+0x2f9/0x16a0
[ 1088.695698]  ? ilookup5_nowait+0x7e/0xa0
[ 1088.695699]  ? _cond_resched+0x19/0x40
[ 1088.695701]  ? do_blockdev_direct_IO+0x3f0/0x2ea0
[ 1088.695705]  ? refcount_dec_and_mutex_lock+0x17/0x60
[ 1088.695706]  do_filp_open+0x9b/0x110
[ 1088.695708]  ? __check_object_size+0xdb/0x1b0
[ 1088.695709]  ? __alloc_fd+0xb2/0x170
[ 1088.695710]  do_sys_open+0x1ba/0x2e0
[ 1088.695711]  ? do_sys_open+0x1ba/0x2e0
[ 1088.695715]  ? task_work_run+0x80/0xc0
[ 1088.695716]  __x64_sys_openat+0x20/0x30
[ 1088.695719]  do_syscall_64+0x57/0x190
[ 1088.695720]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 1088.695722] RIP: 0033:0x7f691d1a9b34
[ 1088.695726] Code: Bad RIP value.
[ 1088.695726] RSP: 002b:00007ffef0259f90 EFLAGS: 00000293 ORIG_RAX: 0000000000000101
[ 1088.695728] RAX: ffffffffffffffda RBX: 00007ffef025a500 RCX: 00007f691d1a9b34
[ 1088.695728] RDX: 0000000000000002 RSI: 00007ffef025a270 RDI: 00000000ffffff9c
[ 1088.695729] RBP: 00007ffef025a270 R08: 0000000000000000 R09: 00007ffef0259ea0
[ 1088.695729] R10: 0000000000000000 R11: 0000000000000293 R12: 0000000000000002
[ 1088.695730] R13: 0000000000000007 R14: 0000000000000001 R15: 00007ffef025a270
[ 1115.267260] block nbd1: Possible stuck request 0000000096609e97: control (read@1073737728,4096B). Runtime 300 seconds
[ 1145.987432] block nbd1: Possible stuck request 0000000096609e97: control (read@1073737728,4096B). Runtime 330 seconds
[ 1176.707323] block nbd1: Possible stuck request 0000000096609e97: control (read@1073737728,4096B). Runtime 360 seconds
[ 1207.427381] block nbd1: Possible stuck request 0000000096609e97: control (read@1073737728,4096B). Runtime 390 seconds
[ 1209.475423] INFO: task systemd-udevd:29247 blocked for more than 120 seconds.
[ 1209.480564]       Not tainted 5.4.0-1080-aws #87~18.04.1-Ubuntu
[ 1209.485086] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1209.492627] systemd-udevd   D    0 29247  20620 0x80004106
[ 1209.492629] Call Trace:
[ 1209.492637]  __schedule+0x292/0x720
[ 1209.492639]  schedule+0x33/0xa0
[ 1209.492641]  schedule_preempt_disabled+0xe/0x10
[ 1209.492643]  __mutex_lock.isra.10+0x24c/0x4a0
[ 1209.492645]  __mutex_lock_slowpath+0x13/0x20
[ 1209.492647]  ? __mutex_lock_slowpath+0x13/0x20
[ 1209.492649]  mutex_lock+0x2f/0x40
[ 1209.492652]  blkdev_put+0x20/0xd0
[ 1209.492654]  blkdev_close+0x25/0x30
[ 1209.492657]  __fput+0xc6/0x260
[ 1209.492659]  ____fput+0xe/0x10
[ 1209.492662]  task_work_run+0x9d/0xc0
[ 1209.492665]  do_exit+0x389/0xb90
[ 1209.492666]  do_group_exit+0x43/0xa0
[ 1209.492685]  get_signal+0x14f/0x860
[ 1209.492690]  ? new_sync_read+0x122/0x1b0
[ 1209.492693]  do_signal+0x34/0x6e0
[ 1209.492695]  ? __vfs_read+0x29/0x40
[ 1209.492696]  ? vfs_read+0x8e/0x130
[ 1209.492700]  exit_to_usermode_loop+0x90/0x130
[ 1209.492701]  do_syscall_64+0x170/0x190
[ 1209.492703]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 1209.492705] RIP: 0033:0x7f0b9208c031
[ 1209.492711] Code: Bad RIP value.
[ 1209.492712] RSP: 002b:00007ffea3463538 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
[ 1209.492714] RAX: fffffffffffffffc RBX: 000055fe28c45f20 RCX: 00007f0b9208c031
[ 1209.492715] RDX: 0000000000000200 RSI: 000055fe28c3fca8 RDI: 000000000000000f
[ 1209.492715] RBP: 000000003ffffe00 R08: 000055fe28c3fc80 R09: 00007f0b92367c40
[ 1209.492716] R10: 0000000000000007 R11: 0000000000000246 R12: 0000000000000200
[ 1209.492717] R13: 000055fe28c45f70 R14: 000055fe28c3fc98 R15: 000055fe28c3fc80
[ 1209.492721] INFO: task rbd-nbd:29627 blocked for more than 362 seconds.
[ 1209.497788]       Not tainted 5.4.0-1080-aws #87~18.04.1-Ubuntu
[ 1209.502510] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1209.509634] rbd-nbd         D    0 29627  10851 0x00000004
[ 1209.509638] Call Trace:
[ 1209.509645]  __schedule+0x292/0x720
[ 1209.509651]  ? call_rcu+0x10/0x20
[ 1209.509653]  schedule+0x33/0xa0
[ 1209.509657]  blk_mq_freeze_queue_wait+0x4b/0xb0
[ 1209.509659]  ? __wake_up_pollfree+0x40/0x40
[ 1209.509661]  blk_mq_freeze_queue+0x1a/0x20
[ 1209.509666]  nbd_add_socket+0x5e/0x1f0 [nbd]
[ 1209.509669]  nbd_ioctl+0x287/0x3e0 [nbd]
[ 1209.509672]  ? do_filp_open+0xaf/0x110
[ 1209.509687]  blkdev_ioctl+0x91f/0x9f0
[ 1209.509690]  block_ioctl+0x3d/0x50
[ 1209.509691]  do_vfs_ioctl+0xa9/0x640
[ 1209.509694]  ? kmem_cache_free+0x294/0x2b0
[ 1209.509696]  ? putname+0x4c/0x60
[ 1209.509697]  ksys_ioctl+0x75/0x80
[ 1209.509699]  __x64_sys_ioctl+0x1a/0x20
[ 1209.509702]  do_syscall_64+0x57/0x190
[ 1209.509704]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 1209.509705] RIP: 0033:0x7f613528d3ab
[ 1209.509710] Code: Bad RIP value.
[ 1209.509710] RSP: 002b:00007ffe17f38058 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[ 1209.509712] RAX: ffffffffffffffda RBX: 00007ffe17f38550 RCX: 00007f613528d3ab
[ 1209.509712] RDX: 0000000000000007 RSI: 000000000000ab00 RDI: 000000000000001f
[ 1209.509713] RBP: 000055ba43a8a207 R08: 000000000000001f R09: 00007ffe17f37ef0
[ 1209.509714] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000001
[ 1209.509715] R13: 0000000000000007 R14: 0000000000000001 R15: 00007ffe17f382c0
[ 1209.509721] INFO: task rbd-nbd:29672 blocked for more than 362 seconds.
[ 1209.514752]       Not tainted 5.4.0-1080-aws #87~18.04.1-Ubuntu
[ 1209.519211] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1209.526306] rbd-nbd         D    0 29672  10851 0x00004004
[ 1209.526309] Call Trace:
[ 1209.526316]  __schedule+0x292/0x720
[ 1209.526319]  schedule+0x33/0xa0
[ 1209.526322]  blk_queue_enter+0xfc/0x200
[ 1209.526325]  ? __wake_up_pollfree+0x40/0x40
[ 1209.526326]  generic_make_request+0xea/0x300
[ 1209.526329]  ? kmem_cache_alloc+0x180/0x230
[ 1209.526330]  submit_bio+0x46/0x1c0
[ 1209.526331]  ? submit_bio+0x46/0x1c0
[ 1209.526333]  ? bio_add_page+0x83/0x90
[ 1209.526335]  submit_bh_wbc+0x182/0x1b0
[ 1209.526336]  block_read_full_page+0x234/0x390
[ 1209.526338]  ? bdev_evict_inode+0xf0/0xf0
[ 1209.526340]  blkdev_readpage+0x18/0x20
[ 1209.526342]  do_read_cache_page+0x384/0x7c0
[ 1209.526345]  ? prep_new_page+0xfa/0x120
[ 1209.526347]  ? get_page_from_freelist+0x14ac/0x14c0
[ 1209.526348]  read_cache_page+0x12/0x20
[ 1209.526351]  read_dev_sector+0x27/0xd0
[ 1209.526353]  read_lba+0x130/0x220
[ 1209.526354]  efi_partition+0x131/0x770
[ 1209.526357]  ? snprintf+0x45/0x70
[ 1209.526358]  ? is_gpt_valid.part.7+0x420/0x420
[ 1209.526359]  check_partition+0x13f/0x250
[ 1209.526361]  ? check_partition+0x13f/0x250
[ 1209.526362]  rescan_partitions+0xaf/0x360
[ 1209.526363]  bdev_disk_changed+0x5a/0x60
[ 1209.526365]  __blkdev_get+0x4bd/0x570
[ 1209.526366]  blkdev_get+0xdd/0x130
[ 1209.526367]  ? blkdev_get_by_dev+0x50/0x50
[ 1209.526368]  blkdev_open+0x87/0xa0
[ 1209.526372]  do_dentry_open+0x1df/0x3a0
[ 1209.526374]  vfs_open+0x2f/0x40
[ 1209.526488]  path_openat+0x2f9/0x16a0
[ 1209.526491]  ? ilookup5_nowait+0x7e/0xa0
[ 1209.526494]  ? try_to_wake_up+0x63/0x5f0
[ 1209.526495]  ? do_blockdev_direct_IO+0x3f0/0x2ea0
[ 1209.526497]  ? wake_up_q+0x44/0x80
[ 1209.526498]  do_filp_open+0x9b/0x110
[ 1209.526500]  ? __switch_to_asm+0x34/0x70
[ 1209.526502]  ? __check_object_size+0xdb/0x1b0
[ 1209.526503]  ? __alloc_fd+0xb2/0x170
[ 1209.526504]  do_sys_open+0x1ba/0x2e0
[ 1209.526505]  ? do_sys_open+0x1ba/0x2e0
[ 1209.526506]  __x64_sys_openat+0x20/0x30
[ 1209.526510]  do_syscall_64+0x57/0x190
[ 1209.526511]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 1209.526513] RIP: 0033:0x7fcda64a1b34
[ 1209.526517] Code: Bad RIP value.
[ 1209.526518] RSP: 002b:00007fff65648600 EFLAGS: 00000293 ORIG_RAX: 0000000000000101
[ 1209.526519] RAX: ffffffffffffffda RBX: 00007fff65648b70 RCX: 00007fcda64a1b34
[ 1209.526520] RDX: 0000000000000002 RSI: 00007fff656488e0 RDI: 00000000ffffff9c
[ 1209.526521] RBP: 00007fff656488e0 R08: 0000000000000000 R09: 00007fff65648510
[ 1209.526522] R10: 0000000000000000 R11: 0000000000000293 R12: 0000000000000002
[ 1209.526522] R13: 0000000000000007 R14: 0000000000000001 R15: 00007fff656488e0
[ 1209.526528] INFO: task rbd-nbd:29709 blocked for more than 362 seconds.
[ 1209.531364]       Not tainted 5.4.0-1080-aws #87~18.04.1-Ubuntu
[ 1209.535847] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1209.542935] rbd-nbd         D    0 29709  10851 0x00000004
[ 1209.542938] Call Trace:
[ 1209.542946]  __schedule+0x292/0x720
[ 1209.542948]  schedule+0x33/0xa0
[ 1209.542951]  schedule_preempt_disabled+0xe/0x10
[ 1209.542953]  __mutex_lock.isra.10+0x24c/0x4a0
[ 1209.542957]  ? disk_map_sector_rcu+0x70/0x70
[ 1209.542959]  __mutex_lock_slowpath+0x13/0x20
[ 1209.542960]  ? __mutex_lock_slowpath+0x13/0x20
[ 1209.542962]  mutex_lock+0x2f/0x40
[ 1209.542965]  __blkdev_get+0x79/0x570
[ 1209.542967]  blkdev_get+0xdd/0x130
[ 1209.542969]  ? blkdev_get_by_dev+0x50/0x50
[ 1209.542970]  blkdev_open+0x87/0xa0
[ 1209.542974]  do_dentry_open+0x1df/0x3a0
[ 1209.542975]  vfs_open+0x2f/0x40
[ 1209.542978]  path_openat+0x2f9/0x16a0
[ 1209.542980]  ? ilookup5_nowait+0x7e/0xa0
[ 1209.542982]  ? _cond_resched+0x19/0x40
[ 1209.542983]  ? do_blockdev_direct_IO+0x3f0/0x2ea0
[ 1209.542987]  ? refcount_dec_and_mutex_lock+0x17/0x60
[ 1209.542989]  do_filp_open+0x9b/0x110
[ 1209.542991]  ? __check_object_size+0xdb/0x1b0
[ 1209.542992]  ? __alloc_fd+0xb2/0x170
[ 1209.542993]  do_sys_open+0x1ba/0x2e0
[ 1209.542994]  ? do_sys_open+0x1ba/0x2e0
[ 1209.542997]  ? task_work_run+0x80/0xc0
[ 1209.542998]  __x64_sys_openat+0x20/0x30
[ 1209.543001]  do_syscall_64+0x57/0x190
[ 1209.543004]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 1209.543005] RIP: 0033:0x7f691d1a9b34
[ 1209.543009] Code: Bad RIP value.
[ 1209.543010] RSP: 002b:00007ffef0259f90 EFLAGS: 00000293 ORIG_RAX: 0000000000000101
[ 1209.543011] RAX: ffffffffffffffda RBX: 00007ffef025a500 RCX: 00007f691d1a9b34
[ 1209.543012] RDX: 0000000000000002 RSI: 00007ffef025a270 RDI: 00000000ffffff9c
[ 1209.543012] RBP: 00007ffef025a270 R08: 0000000000000000 R09: 00007ffef0259ea0
[ 1209.543013] R10: 0000000000000000 R11: 0000000000000293 R12: 0000000000000002
[ 1209.543013] R13: 0000000000000007 R14: 0000000000000001 R15: 00007ffef025a270
[ 1238.147433] block nbd1: Possible stuck request 0000000096609e97: control (read@1073737728,4096B). Runtime 420 seconds
[ 1268.867433] block nbd1: Possible stuck request 0000000096609e97: control (read@1073737728,4096B). Runtime 450 seconds
[ 1299.587478] block nbd1: Possible stuck request 0000000096609e97: control (read@1073737728,4096B). Runtime 480 seconds
[ 1330.307544] block nbd1: Possible stuck request 0000000096609e97: control (read@1073737728,4096B). Runtime 510 seconds
[ 1361.027582] block nbd1: Possible stuck request 0000000096609e97: control (read@1073737728,4096B). Runtime 540 seconds

Is this related? I happened to stumble upon this while doing other work.

for pid in $(pgrep -f nbd); do ps -o cmd --no-header -p $pid; cat /proc/$pid/stack;done
[kworker/u71:0-knbd0-recv]
[<0>] worker_thread+0xd1/0x3e0
[<0>] kthread+0x11b/0x140
[<0>] ret_from_fork+0x22/0x30
[knbd0-recv]
[<0>] rescuer_thread+0x2dc/0x3a0
[<0>] kthread+0x11b/0x140
[<0>] ret_from_fork+0x22/0x30
dd if=/dev/random of=/dev/nbd0 bs=4096 count=32 oflag=sync
[<0>] submit_bio_wait+0x79/0xc0
[<0>] blkdev_issue_flush+0x5b/0x80
[<0>] blkdev_fsync+0x2f/0x50
[<0>] blkdev_write_iter+0x104/0x150
[<0>] new_sync_write+0x11c/0x1b0
[<0>] vfs_write+0x1ce/0x260
[<0>] ksys_write+0x5f/0xe0
[<0>] do_syscall_64+0x33/0x80
[<0>] entry_SYSCALL_64_after_hwframe+0x61/0xc6
rbd-nbd --device /dev/nbd0 map deadlock
[<0>] unix_stream_read_generic+0x595/0x870
[<0>] unix_stream_recvmsg+0x53/0x80
[<0>] sock_read_iter+0x92/0x100
[<0>] new_sync_read+0x192/0x1b0
[<0>] vfs_read+0xf8/0x180
[<0>] ksys_read+0xab/0xe0
[<0>] do_syscall_64+0x33/0x80
[<0>] entry_SYSCALL_64_after_hwframe+0x61/0xc6
rbd-nbd --device /dev/nbd0 map deadlock
[<0>] blk_mq_freeze_queue_wait+0x66/0xa0
[<0>] nbd_add_socket+0x74/0x1f0 [nbd]
[<0>] nbd_ioctl+0x2dc/0x42c [nbd]
[<0>] block_ioctl+0x39/0x40
[<0>] __x64_sys_ioctl+0x8b/0xc0
[<0>] do_syscall_64+0x33/0x80
[<0>] entry_SYSCALL_64_after_hwframe+0x61/0xc6
rbd-nbd --device /dev/nbd0 unmap deadlock
[<0>] blk_queue_enter+0x112/0x230
[<0>] submit_bio_noacct+0x3a1/0x420
[<0>] submit_bh_wbc+0x16a/0x1a0
[<0>] block_read_full_page+0x2e4/0x380
[<0>] do_read_cache_page+0x2e7/0x480
[<0>] read_part_sector+0x32/0xf0
[<0>] adfspart_check_ICS+0x32/0x350
[<0>] blk_add_partitions+0x133/0x350
[<0>] bdev_disk_changed+0x79/0x100
[<0>] __blkdev_get+0x244/0x640
[<0>] blkdev_get+0x40/0xc0
[<0>] do_dentry_open+0x14b/0x360
[<0>] path_openat+0xb86/0x1080
[<0>] do_filp_open+0x88/0x130
[<0>] do_sys_openat2+0x9b/0x150
[<0>] __x64_sys_openat+0x54/0xa0
[<0>] do_syscall_64+0x33/0x80
[<0>] entry_SYSCALL_64_after_hwframe+0x61/0xc6

../qa/workunits/rbd_nbd/deadlock.sh
+ rbd create deadlock --size 32
2022-12-21T21:39:54.189+0000 7f892cdf8580 -1 WARNING: all dangerous and experimental features are enabled.
2022-12-21T21:39:54.189+0000 7f892cdf8580 -1 WARNING: all dangerous and experimental features are enabled.
2022-12-21T21:39:54.197+0000 7f892cdf8580 -1 WARNING: all dangerous and experimental features are enabled.
+ trap unmap EXIT
+ rbd-nbd --device /dev/nbd0 map deadlock
+ true
+ lsblk --noempty /dev/nbd0
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
nbd0  43:0    0  32M  0 disk 
+ r=0
+ '[' 0 -eq 32 ']'
+ '[' 0 -eq 0 ']'
+ break
+ sleep 0.1
+ dd if=/dev/random of=/dev/nbd0 bs=4096 count=1024 oflag=sync
dd: error writing '/dev/nbd0': Input/output error
1+0 records in
0+0 records out
0 bytes copied, 0.000300712 s, 0.0 kB/s
2022-12-21T21:39:54.325+0000 7f267c3251c0 -1 WARNING: all dangerous and experimental features are enabled.
2022-12-21T21:39:54.333+0000 7f267c3251c0 -1 WARNING: all dangerous and experimental features are enabled.
2022-12-21T21:39:54.337+0000 7f267c3251c0 -1 WARNING: all dangerous and experimental features are enabled.
+ unmap
+ rbd-nbd --device /dev/nbd0 unmap deadlock

#!/bin/bash -x

rbd create deadlock --size 32

rbd-nbd --device /dev/nbd0 map deadlock &
unmap()
{
  rbd-nbd --device /dev/nbd0 unmap deadlock
  rbd rm deadlock
}
trap unmap EXIT
while true; do
  lsblk --noempty /dev/nbd0
  r=$?
  [ $r -eq 32 ] && continue
  [ $r -eq 0 ] && break
done
dd if=/dev/random of=/dev/nbd0 bs=4096 count=1024 oflag=sync &
sleep 0.1