How to configure a Google Pubsub subscription with dead-letter enabled?

I have a simple setup with one topic and one subscription. There are some strict limits on the API used so the ultimate goal is to avoid spamming as much as possible so as to avoid being banned.

I have a dead letter topic with max_delivery_attempts set to 5 (the minimum possible value) so that messages get purged after 5 nacks or timeouts.

There are also parameters like ack_deadline_seconds and message_retention_duration which I might not understand completely.

If I understand correctly, ack_deadline_seconds defines how much time should pass before a message is considered a failure. Roughly speaking, this is similar to being nacked.

message_retention_duration defines how long messages are stored in the backlog. This is what I don’t understand. The default value is 10 days, which sounds like an overkill.

Do I understand correctly, such a high value doesn’t make much sense given that I have a dead letter topic? All messages will be sent to the topic without even having a chance to be in the backlog for so long.

In general, how do I come up with a reasonable set of values for these parameters?

>Solution :

The fields ack_deadline_seconds and message_retention_duration control very different things. The ack_deadline_seconds is the amount of time a subscriber that has received a message has to ack the message before the message should be redelivered. If your subscriber repeatedly didn’t ack this message, then with max_delivery_attempts set to 5, It would take approximately 5 * ack_deadline_seconds for the message to end up in the dead letter topic (and to be acked by Cloud Pub/Sub so that it is not delivered to your subscriber again).

The value you use for ack_deadline_seconds is a tradeoff between the possibility of duplicates if a subscriber takes longer than this to process a message vs. the delayed processing of a message if a subscriber received a message, but went down before it could process the message. Note that the Cloud Pub/Sub client libraries allow for a little more granularity of control of this tradeoff by allowing you to set a deadline and the amount of time for which the deadline should be extended each time. The flow control settings section of the Pub/Sub settings post goes into a little more detail on this.

The message_retention_duration field controls how long messages are kept for delivery to a subscriber. This is important if there is a time when your subscriber is not running. Even if you have a dead letter queue set up, if no deliveries are attempted because your subscriber is down, the messages in the backlog are not written to the dead letter topic. In other words, if your subscriber is down for longer than message_retention_duration, then messages published before that will be deleted.

You should choose a value for message_retention_duration based on a few different things:

1. How long do you expect it to be possible for your subscriber to be down? You’ll want the duration to be at least this long.
2. Is there an age beyond which messages are no longer relevant to your subscriber? If this is the case, you can set message_retention_duration to this value.