S3: Disable strong integrity checksums #12264
Conversation
ebyhr
force-pushed
the
ebi/s3-integrity-check
branch
from
e629e83 to
83185e7
Compare
| // TODO Remove me once all of the S3-compatible storage support strong integrity checks | ||
| @SuppressWarnings("deprecation") | ||
| static AwsRequestOverrideConfiguration disableStrongIntegrityChecksums() { | ||
| return AwsRequestOverrideConfiguration.builder().signer(AwsS3V4Signer.create()).build(); |
I'm not sure this is the right way to disable it? Feels like this is just a side effect of setting a new signer but I don't think we want that.
Oh I see, is the issue in compatibility is that older versions of Minio/3rd party object storage solutions expect Content-MD5 and in the new SDK we are not sending that and so the service rejects the request? Still feels like there should be a different way to force setting MD5
Setting WHEN_REQUIRED for checksum calculation/validation doesn't resolve as far as I tested. Calculating MD5 for PutObjectRequest looks feasible, but I'm not sure how to do it for DeleteObjectsRequest.
@wendigo Do you happen to know any other approaches?
We have the same issue on the PyIceberg: apache/iceberg-python#1546 How about making this configurable?
@ebyhr no. The only workaround that I have found is to force previous signature and it works against Dell ECS, old Minio versions and Ozone. I haven't tested Vast since I don't have access to it
| @@ -149,4 +151,10 @@ static void configurePermission( | |||
| Function<ObjectCannedACL, S3Request.Builder> aclSetter) { | |||
| aclSetter.apply(s3FileIOProperties.acl()); | |||
| } | |||
|
|
|||
| // TODO Remove me once all of the S3-compatible storage support strong integrity checks | |||
I understand the intent but this doesn't feel like a practical way of going about this just because there's many out there and "all" just seems like a moving goalpost.
| @@ -29,15 +29,18 @@ | |||
| import software.amazon.awssdk.services.s3.S3ClientBuilder; | |||
|
|
|||
| public class MinioUtil { | |||
| public static final String LATEST_TAG = "latest"; | |||
| // This version doesn't support strong integrity checks | |||
| static final String LEGACY_TAG = "RELEASE.2024-12-18T13-15-44Z"; | |||
can we try and roll back all of the changes in the PR and then only update the TAG to RELEASE.2025-01-20T14-49-07Z (see also minio/minio#20845 (comment)). I've tried it locally and that fixed the issue described in #12237 when running those tests locally with Docker Desktop
@nastra it does, but it won't still work with some other S3-compatible storages like Vast, Dell ECS, so upgrading Minio to compatible version doesn't solve the issue
@nastra We intentionally specified pre-2025 here. Using a newer tag hides the actual problem.
|
@nastra that will work :) |
|
@ebyhr How hard would it be for us to get some integration tests with one of these systems into the Iceberg project? Seems like we should have some coverage for these other S3-Compat systems. I'd also be ok with a separate project that we just use as a canary before release. |
Hi @RussellSpitzer, |
The recent AWS SDK bump introduced strong integrity checksums, and broke compatibility with many S3-compatible object storages (pre-2025 Minio, Vast, Dell EC etc).
In Trino project, we received the error report (Missing required header for this request: Content-Md5) from several users and had to disable the check temporarily. We recommend disabling it in Iceberg as well. I faced this issue when I tried upgrading Iceberg library to 1.8.0 in Trino.
Relates to trinodb/trino#24954 & trinodb/trino#24713