Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-13574

NotLeaderOrFollowerException thrown for a successful send

Details

    • Bug
    • Status: Open
    • Minor
    • Resolution: Unresolved
    • 3.0.0
    • None
    • clients
    • openjdk version "11.0.13" 2021-10-19

    Description

      With org.apache.kafka/kafka-clients 3.0.0, under rare circumstances involving multiple node and network failures, I've observed a call to `producer.send()` throw `NotLeaderOrFollowerException` for a message which later appears in `consumer.poll()` return values.

      I don't have a reliable repro case for this yet, but the case I hit involved retries=1000, acks=all, and idempotence enabled. I suspect what might be happening here is that an initial attempt to send the message makes it to the server and is committed, but the acknowledgement is lost e.g. due to timeout; the Kafka producer then automatically retries the send attempt, and on that retry hits a NotLeaderOrFollowerException, which is thrown back to the caller. If we interpret NotLeaderOrFollowerException as a definite failure, then this would constitute an aborted read.

      I've seen issues like this in a number of databases around client or server-internal retry mechanisms, and I think the thing to do is: rather than throwing the most recent error, throw the most indefinite. That way clients know that their request may have actually succeeded, and they won't (e.g.) attempt to re-submit a non-idempotent request again.

      As a side note: is there... perhaps documentation on which errors in Kafka are supposed to be definite vs indefinite? NotLeaderOrFollowerException is a subclass of RetriableException, but it looks like RetriableException is more about transient vs permanent errors than whether it's safe to retry.

      Attachments

        Activity

          Ascending order - Click to sort in descending order
          hachikuji Jason Gustafson added a comment -

          aphyr Thanks for the bug report. My guess is that you might be hitting a race condition such as the following during a leader change:

          1. Leader takes the write at offset N and parks the request in purgatory to await replication.
          2. Follower acks offset N-1 and writes the data up to offset N.
          3. Follower becomes new leader before it is able to ack offset N.
          4. Old leader sees the leader change and returns NOT_LEADER to the still-parked produce request.

          In this scenario, the new leader has the data and it may become committed even though the produce request returned NOT_LEADER. There might be a better error to return in this scenario, but we do want the producer to realize that it needs to find the new leader. I guess we could also hold the request a bit longer until we know whether or not it was committed. Failing that, we should make it clear that NOT_LEADER does not necessarily indicate a failure to write the data.

          junrao Any thoughts?

          hachikuji Jason Gustafson added a comment - aphyr Thanks for the bug report. My guess is that you might be hitting a race condition such as the following during a leader change: 1. Leader takes the write at offset N and parks the request in purgatory to await replication. 2. Follower acks offset N-1 and writes the data up to offset N. 3. Follower becomes new leader before it is able to ack offset N. 4. Old leader sees the leader change and returns NOT_LEADER to the still-parked produce request. In this scenario, the new leader has the data and it may become committed even though the produce request returned NOT_LEADER. There might be a better error to return in this scenario, but we do want the producer to realize that it needs to find the new leader. I guess we could also hold the request a bit longer until we know whether or not it was committed. Failing that, we should make it clear that NOT_LEADER does not necessarily indicate a failure to write the data. junrao Any thoughts?
          junrao Jun Rao added a comment -

          aphyr : Thanks for reporting this issue. hachikuji : Thanks for the investigation.

          NOT_LEADER_OR_FOLLOWER is considered an indefinite error, which means that the produce request may or may not have succeeded. We could wait a bit longer to return a more precise response. However, we need to make sure that the record committed at offset N is indeed the one from the producer to be acknowledged. To do that, we could save the leader epoch after the produce records are appended to the leader's log. When checking the purgatory for completion, we could compare the records' committed leader epoch (probably through leader epoch cache) with the expected one.

          junrao Jun Rao added a comment - aphyr : Thanks for reporting this issue. hachikuji : Thanks for the investigation. NOT_LEADER_OR_FOLLOWER is considered an indefinite error, which means that the produce request may or may not have succeeded. We could wait a bit longer to return a more precise response. However, we need to make sure that the record committed at offset N is indeed the one from the producer to be acknowledged. To do that, we could save the leader epoch after the produce records are appended to the leader's log. When checking the purgatory for completion, we could compare the records' committed leader epoch (probably through leader epoch cache) with the expected one.
          aphyr Kyle Kingsbury added a comment -

          > NOT_LEADER_OR_FOLLOWER is considered an indefinite error

          Aha! In that case this isn't a bug at all, and we can close it!

          Is there documentation for which errors are definite vs indefinite? I've been relying on the javadocs and asking kafka users how they interpret these errors, and that may have led me astray here.

          aphyr Kyle Kingsbury added a comment - > NOT_LEADER_OR_FOLLOWER is considered an indefinite error Aha! In that case this isn't a bug at all, and we can close it! Is there documentation for which errors are definite vs indefinite? I've been relying on the javadocs and asking kafka users how they interpret these errors, and that may have led me astray here.
          aphyr Kyle Kingsbury added a comment -

          Another case like this–I'm seeing transactions fail with `InvalidTxnStateException: The producer attempted a transactional operation in an invalid state.`, which sounds like it should be a definite failure, but... sometimes appears to actually commit. Is this error intended to be definite? Is this documented anywhere?

          aphyr Kyle Kingsbury added a comment - Another case like this–I'm seeing transactions fail with `InvalidTxnStateException: The producer attempted a transactional operation in an invalid state.`, which sounds like it should be a definite failure, but... sometimes appears to actually commit. Is this error intended to be definite? Is this documented anywhere?
          julienlau Laurenceau Julien added a comment -

          We are 2 years later this bug report that assess exactly once processing is broken.

          I see some ideas, but I see no beginning of a solution. Is there any workaround or fix ? 

          Do you guys think that this is not important ?

          Maybe some warning notice should be added on the documentation, because people choosing to pay the price of exactly-once generally care a lot about consistency !

          julienlau Laurenceau Julien added a comment - We are 2 years later this bug report that assess exactly once processing is broken. I see some ideas, but I see no beginning of a solution. Is there any workaround or fix ?  Do you guys think that this is not important ? Maybe some warning notice should be added on the documentation, because people choosing to pay the price of exactly-once generally care a lot about consistency !

          People

            Unassigned Unassigned
            aphyr Kyle Kingsbury
            Votes:
            1 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated: