destroyed my CI/CD Pipeline this morning 😆

It is also impossible to build on ANY alpine linux version/architecture. Please remove this dependency as soon as possible.

It is also impossible to build on ANY alpine linux version/architecture.

Update to most recent Alpine with Rust >= 1.45. Alpine latest has rust (1.47.0-r2).

Please remove this dependency as soon as possible.

Not going to happen. Rust dependency will stay to replace C code with an actual safe language.

You can disable Rust integration in 3.4.x (see FAQ). Starting with 3.5 cryptography will have mandatory Rust code.

So the only "reason" for this change is to use a "actual safe language"? ...ahhh...yeah...
But blowing up systems with unnecessary software packages is a more secure way, that's for sure...
(my 2 ¢)

I guess this means we'll have to fork Cryptography, and render the original package irrelevant.

So the only "reason" for this change is to use a "actual safe language"? ...ahhh...yeah...

Yes, memory safety is a very compelling reason to use Rust. C is a bad language to implement parsers for e.g. ASN.1. Check out @alex and @reaperhulk side-project https://twitter.com/LazyFishBarrel

But blowing up systems with unnecessary software packages is a more secure way, that's for sure...
(my 2 ¢)

The new Rust code adds exactly 0 (zero) runtime packages to Cryptography. Rust, Cargo, pyo3, its dependencies, and setuptools_rust are build-time dependencies only.

Pinned to second last version to get our pipelines up and running again, at least until I've figured out what should be done long term

Memory safety is of course a compelling reason, but you should also keep in mind that may developers out there are using your package, which is great by the way, and are now forced to upgrade all their containers, ci/cd pipelines, packages and so on. A little piece of information upfront or a deprecation warning would have helped a lot. Or maybe I just overlooked something.

Pinned to second last version to get our pipelines up and running again, at least until I've figured out what should be done long term

Yes, that's exactly what I did.

I think that it is clear that you have broken semantic versioning with both the python 2 change, and the requirement for rust at build time. From the changelog:

BACKWARDS INCOMPATIBLE: Support for Python 2 has been removed.
We now ship manylinux2014 wheels and no longer ship manylinux1 wheels. Users should upgrade to the latest pip to ensure this doesn’t cause issues downloading wheels on their platform.
cryptography now incorporates Rust code. Users building cryptography themselves will need to have the Rust toolchain installed. Users who use an officially produced wheel will not need to make any changes. The minimum supported Rust version is 1.45.0.
cryptography now has PEP 484 type hints on nearly all of of its public APIs. Users can begin using them to type check their code with mypy.

This should probably be a 4.0 release it would cause a lot less pain for the community. I suppose many have cryptography>=3.0,<4 in setup.py or requirements.txt.

Update
I was unaware that you don't use semantic versioning. I have found in the docs https://cryptography.io/en/latest/api-stability.html#versioning. However it was a little confusing given the style. As many of our dependencies require this we will try not to pin a version and install rust/use manywheel packages.

A little piece of information upfront or a deprecation warning would have helped a lot. Or maybe I just overlooked something.

Looks like you have missed all announcements and FAQ entries.

Rust bindings kicked off in July 2020 with #5357. Alex started a thread on the cryptography developer mailing list in December to get feedback from packagers and users. The FAQ also contains instrutions how you can disable Rust bindings.

PS: I've been preparing Fedora packaging for almost three months. My Fedora Rawhide packages of python-cryptography 3.4 landed about 12 hours after the release -- and I had to jump through several extra hoops because Fedora does not permit vendoring of Rust crates.

Yes, memory safety is a very compelling reason to use Rust. C is a bad language to implement parsers for e.g. ASN.1. Check out @alex and @reaperhulk side-project https://twitter.com/LazyFishBarrel

I'm sure all the users who aren't able to use cryptography anymore will appreciate the memory safety.

Rust bindings kicked off in July 2020 with #5357. Alex started a thread on the cryptography developer mailing list in December to get feedback from packagers and users. The FAQ also contains instrutions how you can disable Rust bindings.

Do you seriously expect people who maintain a few hundred Python packages to follow development mailing lists of every single one of them?

I'm sure all the users who aren't able to use cryptography anymore will appreciate the memory safety.

The majority of users either uses binary wheels (macOS x86_64, glibc Linux x86_64 + aarch64, Windows X86 + X86_64) or Linux distro packages. Binary wheels don't require an additional Rust libraries. Only users on Alpine (musl), BSD, other hardware platforms, and distro packagers are affected.

Do you seriously expect people who maintain a few hundred Python packages to follow development mailing lists of every single one of them?

Do you have constructive suggestions how to communicate changes additionally to project mailing lists, Github issue tracker, project IRC channel, documentation (changelog, FAQ), and Twitter channels?

@tiran

I wonder if there's a way to produce a more helpful error message, perhaps with a link to some documentation about the new required build deps. The current message most people will hit is

      =============================DEBUG ASSISTANCE=============================
      If you are seeing a compilation error please try the following steps to
      successfully install cryptography:
      1) Upgrade to the latest pip and try again. This will fix errors for most
         users. See: https://pip.pypa.io/en/stable/installing/#upgrading-pip
      2) Read https://cryptography.io/en/latest/installation.html for specific
         instructions for your platform.
      3) Check our frequently asked questions for more information:
         https://cryptography.io/en/latest/faq.html
      4) Ensure you have a recent Rust toolchain installed.
      =============================DEBUG ASSISTANCE=============================

Which does sort of attempt to guide people the right way but could be more specific for common cases like Alpine.

Trying to follow the docs as an Alpine user who doesn't know about Rust is kind of confusing right now. As the Alpine specific steps only tell you to install gcc but as you point out above latest Alpine actually has a new enough rust in APK.

Pinned to second last version to get our pipelines up and running again, at least until I've figured out what should be done long term

Yes, that's exactly what I did.

Yep, forced to do the same (AWS Glue broke when importing this for requests lib)

Switching to Rust and locking out dozens of platforms is worse for security, because now those users have no choice but to rely on outdated security-critical software OR to port Rust to their platform, which is a gargantuan effort (hi, someone who has tried - and failed - to port Rust to a new platform here). A lot of those people can't afford to "upgrade" to newer hardware - much of which brings its own security concerns, like UEFI.

C is not evil. C is not a bad programming language. C is a universally supported language. Cargo culting Rust is harmful to the ecosystem.

There were questions on how the announcement/notifications could be handled better. Well, it's fair to say that this change affects the Python ecosystem as a whole, and so release notification should have been posted to the https://mail.python.org/mailman3/lists/python-announce-list.python.org/ mailing list. And indeed it was! But I'd say it was pretty meek: https://mail.python.org/archives/list/python-announce-list@python.org/thread/J6VZ22LY5IS3WG5SMICUSWQQOK5SAFTQ/

Given the effect it caused, it should have a subject:

PyCA cryptography 3.3.2 and 3.4 released - SYSTEM BREAKAGE ahead

and should have started with:

###################### WARNING! #########################
Installing PyCA cryptography from source (as happens with
older pip releases) now requires Rust to be installed. If
you use old pip (if you didn't check, you likely do) and
don't have Rust installed, all packages depending on
PyCA cryptography will be broken!
###################### WARNING! #########################

And I'm only half-joking here, your recent release announcement before/with such a big change really contain too few exclamation marks and all-caps.

And I second what other people said - dear cryptography developers, you are doing the great work of maintaining security package as volunteers! But not only that, you also BREAK SYSTEMS of other people! Please take not less pride in that than in your development work. And if you don't want to take pride in that (why? it's fun), then maybe indeed spend a bit less time hasting to write more Rust, and a bit more time writing "scary" (read: disclosing the truth to users) (pre)release announcements.

Thanks!

C is in fact very bad and I have a mountain of research to prove it: https://www.usenix.org/conference/enigma2021/presentation/gaynor

C is in fact very bad and I have a mountain of research to prove it: https://www.usenix.org/conference/enigma2021/presentation/gaynor

https://en.wikipedia.org/wiki/Confirmation_bias

You read my slides very quickly! I can tell you took them seriously and didn't just drop that link immediately.

You're right - I don't take your research seriously!

I recommend not using this software package then, regardless of what languages its written in. (Of course, I suspect that's already the case -- you're not a user, you just showed up to complain because this is your hobby horse.)

I am a cryptography user, as a matter of fact, and a packager for affected systems. But thanks for the ad-hominem speculation, appreciate it.

My apologies. I regret engaging with you at all.

First: this is going to be the second to last last comment on this thread. I think it's run its course. Folks should please feel empowered to file new issues, but I'd request that you only file issues for either a) new issues, b) concrete improvements not already discussed here.

Second: Thank you to all the folks who have engaged constructively here. Our release is better for your contributions.

Third: The last comment on this thread will be a summary of everything discussed here, I'll leave that after I finish writing it.

I now want to summarize the state of various discussions here:

Outdated Rust installations

We now take advantage of an option in setuptools-rust to provide a clear error message when a user's Rust installation is too old.

In the future, we'll be somewhat conservative in increasing our MSRV.

Update: We have lowered the MSRV to 1.41 in #5823 and this change was released in 3.4.5.

No Rust installation at all

We've now documented very aggressively how to obtain a Rust installation, and to update pip (which will solve the problem for folks on manylinux-compatible distributions).

No future steps are expected.

Alpine

Wheels

It looks like there's folks actively interested in specifying many-musl. Excellent! Once that's defined and supported in pip, we'll ship wheels for that platform.

Update September 20, 2021: The musllinux specification and implementation is complete and cryptography was the first project to upload x86_64 and aarch64 wheels. You need pip >= 21.2.4 for musllinux support.

Bootstrapping Rust on supported platforms

We'll reach out off-list to see how we can support this effort.

LTS

We'll engage on the thread that @tiran filed.

SemVer

We'll consider it, however, I am skeptical for the simple reason that this change would not have required a major version bump (no public APIs were changed). Therefore it's clear that what folks wanted was a major version bump regardless of semver.

Comms

In the future, when we know about a change like this well in advance, we'll include it in our CHANGELOG as soon as we know about it.

If a platform for proactively reaching out to PyPI users exists, we'll use it.

Alpha, S390, HPPA, HPUX, AIX, etc.

Best of luck.

I'd encourage you to contribute platform support to LLVM or contribute to gccrs.

My work to promote language-level memory safety will continue unabated.

Miscellany

I'm sure I missed something (though I did just re-read this thread in its entirety).