Route53: character limit of 32000 exceeded about octodns-route53

[ross]

@ross do you think it make sense to lower the max_changes to 500 or something, so no one run into the same problem. I have lower it 500 and run it on my local env works fine now.

Thought/thinking about that. Problem is that even at 500 it'd be possible, not even hard with sizeable TXT records, to hit 32k. 100 might be safer and we may be lucky enough that no one would ever hit it. I think my preferred approach would be to look at the code briefly to see what it'd take to actually measure things. We already have a bunch of code in there to deal with the 1k item limit so it may not be too hard to compute the expected size of the request as well. If it's not easy I think making the default 100 items would probably be the way to go.

Thanks for the report, debugging, and help.

[joewilliams]

RDATA character limit of 32000 exceeded seems to suggest this might be a individual record with 32000 characters in the data. Is this the case? If so, it possible to supply us with the record(s) that are like this?

[aikram24]

@joewilliams I don't think so its individual record. So I got a length of each RDATA and put them in a list and when I add them it equal to 33721. Following is list contain length of each RData:
[59, 68, 29, 18, 25, 44, 18, 50, 28, 68, 64, 24, 38, 3, 31, 19, 45, 34, 15, 3, 54, 43, 19, 35, 19, 48, 34, 47, 43, 18, 44, 32, 49, 49, 17, 17, 18, 33, 67, 25, 17, 55, 18, 35, 17, 64, 15, 18, 15, 50, 19, 48, 40, 35, 17, 48, 29, 34, 75, 58, 30, 52, 32, 30, 16, 49, 35, 35, 31, 49, 67, 19, 36, 48, 25, 21, 40, 17, 69, 35, 29, 15, 33, 21, 17, 34, 49, 20, 41, 30, 30, 65, 35, 17, 16, 37, 43, 20, 16, 65, 47, 39, 70, 17, 65, 49, 51, 39, 61, 30, 17, 30, 18, 49, 40, 17, 35, 30, 50, 17, 19, 32, 27, 43, 35, 34, 20, 17, 16, 19, 31, 37, 35, 32, 51, 42, 20, 30, 27, 58, 51, 35, 74, 69, 64, 35, 32, 24, 49, 42, 46, 25, 57, 31, 24, 71, 24, 33, 19, 16, 36, 17, 29, 57, 17, 52, 20, 49, 16, 24, 19, 67, 33, 15, 34, 18, 64, 26, 49, 29, 67, 16, 35, 54, 29, 16, 34, 16, 32, 47, 47, 25, 35, 28, 16, 22, 47, 19, 78, 40, 17, 68, 64, 15, 18, 30, 16, 51, 19, 58, 16, 68, 17, 35, 56, 42, 27, 30, 16, 72, 41, 25, 24, 18, 70, 32, 24, 18, 33, 35, 46, 18, 49, 28, 41, 56, 51, 51, 43, 17, 48, 64, 18, 44, 19, 24, 40, 42, 25, 35, 35, 25, 18, 60, 80, 35, 30, 15, 19, 30, 15, 20, 19, 35, 57, 62, 19, 34, 25, 22, 30, 35, 35, 25, 48, 30, 50, 17, 20, 3, 24, 35, 35, 49, 35, 18, 3, 59, 32, 25, 34, 51, 30, 42, 37, 35, 18, 49, 37, 28, 16, 70, 68, 18, 44, 43, 30, 77, 30, 70, 19, 34, 32, 30, 24, 3, 43, 50, 67, 56, 30, 48, 17, 20, 49, 19, 35, 19, 62, 26, 18, 17, 31, 34, 35, 49, 17, 50, 49, 23, 49, 43, 70, 40, 35, 30, 65, 19, 26, 19, 24, 35, 34, 78, 17, 14, 44, 62, 32, 17, 31, 16, 49, 35, 34, 3, 34, 25, 16, 35, 49, 16, 23, 19, 31, 17, 25, 15, 51, 35, 65, 17, 66, 20, 46, 30, 55, 35, 16, 67, 17, 49, 49, 30, 33, 17, 71, 35, 21, 40, 16, 77, 51, 40, 18, 29, 17, 19, 59, 16, 20, 40, 49, 67, 19, 24, 67, 26, 17, 17, 26, 68, 26, 19, 20, 31, 35, 30, 26, 49, 35, 49, 30, 18, 34, 34, 50, 50, 35, 33, 24, 20, 51, 16, 48, 38, 20, 17, 29, 34, 50, 15, 19, 42, 44, 25, 28, 49, 25, 25, 18, 51, 44, 65, 34, 32, 30, 62, 29, 19, 25, 16, 36, 35, 17, 16, 27, 68, 62, 35, 38, 20, 35, 67, 28, 33, 41, 27, 29, 52, 25, 19, 24, 26, 34, 30, 43, 17, 16, 16, 70, 58, 27, 17, 20, 35, 35, 16, 45, 20, 43, 19, 36, 59, 34, 58, 46, 28, 16, 28, 27, 72, 62, 19, 20, 18, 49, 25, 60, 19, 55, 17, 68, 24, 23, 28, 16, 52, 17, 34, 67, 34, 42, 35, 37, 31, 35, 19, 32, 37, 15, 35, 51, 35, 47, 19, 16, 17, 27, 77, 17, 34, 50, 19, 35, 16, 34, 34, 19, 35, 49, 17, 18, 35, 17, 29, 49, 35, 66, 30, 17, 49, 78, 46, 33, 16, 17, 65, 16, 59, 47, 20, 53, 56, 20, 19, 19, 46, 52, 35, 18, 19, 18, 31, 47, 18, 19, 44, 16, 60, 47, 27, 35, 30, 67, 20, 15, 78, 62, 35, 17, 59, 20, 27, 17, 46, 72, 16, 24, 64, 35, 17, 29, 27, 55, 15, 41, 59, 35, 61, 16, 19, 18, 20, 17, 41, 49, 49, 17, 30, 18, 20, 16, 25, 34, 54, 51, 17, 18, 17, 27, 35, 19, 18, 29, 59, 34, 65, 18, 34, 35, 17, 35, 68, 43, 72, 35, 68, 47, 50, 16, 62, 20, 30, 19, 35, 72, 19, 19, 18, 31, 30, 47, 35, 30, 29, 51, 17, 16, 73, 40, 52, 16, 49, 31, 57, 18, 34, 19, 59, 51, 15, 35, 14, 68, 21, 24, 35, 16, 35, 42, 20, 50, 43, 67, 40, 35, 35, 59, 34, 48, 25, 16, 18, 49, 59, 48, 17, 26, 30, 38, 35, 17, 20, 34, 38, 37, 33, 34, 34, 48, 20, 29, 33, 43, 70, 69, 18, 42, 24, 34, 56, 16, 32, 79, 35, 25, 17, 39, 25, 16, 35, 15, 35, 28, 37, 37, 34, 26, 55, 31, 17, 35, 19, 48, 35, 24, 33, 25, 16, 30, 50, 31, 17, 70, 17, 59, 35, 40, 17, 35, 19, 18, 19, 38, 24, 35, 35, 30, 48, 31, 24, 43, 16, 30, 16, 33, 19, 35, 15, 23, 20, 33, 28, 17, 18, 18, 39, 30, 24, 30, 71, 30, 25, 35, 35, 19, 20, 31, 34, 35, 18, 19, 70, 43, 16, 49, 46, 35, 28, 68, 27, 77, 35, 50, 49, 17, 34, 18, 57, 18, 59, 44, 22, 30, 16, 51, 29, 17, 3, 34, 17, 25, 54, 34, 59, 17, 35, 35, 28, 19, 26, 23, 3, 28, 31, 34, 44, 19, 20, 35, 48, 26, 35, 33, 35, 17, 30, 36, 19, 47, 49, 16, 17, 35, 51, 77, 16, 17, 18, 33, 38, 35, 35, 68, 34, 16, 20, 35, 17, 45, 17, 20, 33, 22, 38, 65, 35, 48, 57, 30, 35, 35, 18, 35, 39, 16, 77, 43, 25, 35, 25, 57, 46, 32, 28, 44, 69, 34, 62, 24, 18, 59, 27, 16, 15, 18, 44, 19, 27, 35, 40, 51, 35, 49, 71, 30, 24, 35, 17, 17, 58, 16, 30, 48, 20, 29]

[ross]

Fun, looks like there's a limit on data size in addition to the number of records.

@joewilliams actually added an undocumented parameter, max_changes, to help in unit testing the batch size code that will probably function as a work-around here. The default is 1000, anything less than that should break it up into smaller chunks. 500 is probably as good as any place to start.

providers:
  route53:
    ...
    max_changes: 500

[ross]

Looks like the limits are at least documented. 32k is a pretty arbitrary seeming limit, but I guess they had to pick something.

[aikram24]

@ross do you think it make sense to lower the max_changes to 500 or something, so no one run into the same problem. I have lower it 500 and run it on my local env works fine now.

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 7 days.