Temporary failure in name resolution

@fin_chen looks like everyone here was right. I was able to clear out our resolvers' cache across our data centers for mailgun.org and this should be all set now. Here's an example from the resolver @millisa tested on earlier.

# dig smtp.mailgun.com @72.14.179.5

; <<>> DiG 9.10.3-P4-Debian <<>> smtp.mailgun.com @72.14.179.5
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 62896
;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;smtp.mailgun.com.        IN  A

;; ANSWER SECTION:
smtp.mailgun.com.    295 IN  CNAME   smtp.mailgun.org.
smtp.mailgun.org.    39  IN  A   44.231.238.210
smtp.mailgun.org.    39  IN  A   54.184.250.231
smtp.mailgun.org.    39  IN  A   52.34.221.85

;; Query time: 0 msec
;; SERVER: 72.14.179.5#53(72.14.179.5)
;; WHEN: Fri Aug 27 23:15:19 EDT 2021
;; MSG SIZE  rcvd: 123

This was being talked about this morning in the linode irc channel.

I started seeing mail failures a little before 8/27 4pm GMT for smtp.mailgun.org when using Linode's resolvers. Same deal - the record wouldn't resolve. The SOA record didn't either (and as of 9 hours later still doesn't from Linode's resolvers).

The time I started seeing failures matches up with the Updated Date in the mailgun.org whois record… This is an unlikely coincidence. It seems very likely mailgun broke something at their registrar/nameservice. In the short term, adding google (8.8.8.8) or cloudflare (1.1.1.1) as a resolver will work around the problem until the linode resolvers start handing out good info again.

Domain Name: MAILGUN.ORG
Registry Domain ID: D159169556-LROR
Registrar WHOIS Server: whois.namecheap.com
Registrar URL: http://www.namecheap.com
Updated Date: 2021-08-27T15:45:45Z
Creation Date: 2010-05-14T00:26:11Z
Registry Expiry Date: 2022-05-14T00:26:11Z
Registrar Registration Expiration Date:
Registrar: NameCheap, Inc.
Registrar IANA ID: 1068
Registrar Abuse Contact Email: abuse@namecheap.com
Registrar Abuse Contact Phone: +1.6613102107
Reseller:
Domain Status: clientTransferProhibited https://icann.org/epp#clientTransferProhibited
Registrant Organization: Privacy service provided by Withheld for Privacy ehf
Registrant State/Province: Capital Region
Registrant Country: IS
Name Server: NS-1482.AWSDNS-57.ORG
Name Server: NS-133.AWSDNS-16.COM
Name Server: NS-586.AWSDNS-09.NET
Name Server: NS-1614.AWSDNS-09.CO.UK
DNSSEC: unsigned
URL of the ICANN Whois Inaccuracy Complaint Form https://www.icann.org/wicf/)
>>> Last update of WHOIS database: 2021-08-28T01:13:17Z <<<

It looks like mailgun briefly tried to enable dnssec… These DS records are still cached on the linode resolvers for another 13 or so hours…

As of 1:43am 8/28 GMT:

$ dig +nocmd +noall +answer -t ds mailgun.org @72.14.179.5
mailgun.org.        48497   IN  DS  49611 8 2 28F2F10427480AC6FB98D7544D61FE8D866EB5FB33688BC7C3CB8DC6 5D39C916
$ 
$ dig +nocmd +noall +answer -t ds mailgun.org @72.14.188.5
mailgun.org.        48441   IN  DS  49611 8 2 28F2F10427480AC6FB98D7544D61FE8D866EB5FB33688BC7C3CB8DC6 5D39C916
$ 
$ dig +nocmd +noall +answer -t ds mailgun.org @173.255.199.5
mailgun.org.        48455   IN  DS  49611 8 2 28F2F10427480AC6FB98D7544D61FE8D866EB5FB33688BC7C3CB8DC6 5D39C916
$ 
$ dig +nocmd +noall +answer -t ds mailgun.org @66.228.53.5
mailgun.org.        48472   IN  DS  49611 8 2 28F2F10427480AC6FB98D7544D61FE8D866EB5FB33688BC7C3CB8DC6 5D39C916
$ 
$ dig +nocmd +noall +answer -t ds mailgun.org @96.126.122.5
mailgun.org.        48509   IN  DS  49611 8 2 28F2F10427480AC6FB98D7544D61FE8D866EB5FB33688BC7C3CB8DC6 5D39C916
$ 
$ dig +nocmd +noall +answer -t ds mailgun.org @96.126.124.5
mailgun.org.        48486   IN  DS  49611 8 2 28F2F10427480AC6FB98D7544D61FE8D866EB5FB33688BC7C3CB8DC6 5D39C916
$ 
$ dig +nocmd +noall +answer -t ds mailgun.org @96.126.127.5
mailgun.org.        48443   IN  DS  49611 8 2 28F2F10427480AC6FB98D7544D61FE8D866EB5FB33688BC7C3CB8DC6 5D39C916
$ 
$ dig +nocmd +noall +answer -t ds mailgun.org @198.58.107.5
mailgun.org.        48453   IN  DS  49611 8 2 28F2F10427480AC6FB98D7544D61FE8D866EB5FB33688BC7C3CB8DC6 5D39C916
$ 
$ dig +nocmd +noall +answer -t ds mailgun.org @198.58.111.5
mailgun.org.        48436   IN  DS  49611 8 2 28F2F10427480AC6FB98D7544D61FE8D866EB5FB33688BC7C3CB8DC6 5D39C916
$ 
$ dig +nocmd +noall +answer -t ds mailgun.org @23.239.24.5
mailgun.org.        48446   IN  DS  49611 8 2 28F2F10427480AC6FB98D7544D61FE8D866EB5FB33688BC7C3CB8DC6 5D39C916

Credit to Peng in #linode for spotting.

mailgun.org recently botched something related to DNSSEC. I guess they added an incorrect DS record (enabling DNSSEC) and then quickly deleted it to turn it off again, but the .org TLD allows DS records to be cached for up to 1 day.

Should start working again in no more than about 14 hours.

It's nothing specific to Linode (or Vultr). Any validating resolver would have been similarly affected. Some resolver operators may have taken time out of their days to manually remove the records from the cache or disable DNSSEC for the domain; some have low maximum TTLs anyway, or may coincidentally have seen and cached the problematic record while it existed.

Querying one of Linode's resolvers in Atlanta:

$ dig mailgun.org ds @2600:3c02::b

; <<>> DiG 9.17.17-2+ubuntu20.04.1+isc+1-Ubuntu <<>> mailgun.org ds @2600:3c02::b
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 61763
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;mailgun.org.                   IN      DS

;; ANSWER SECTION:
mailgun.org.            48860   IN      DS      49611 8 2 28F2F10427480AC6FB98D7544D61FE8D866EB5FB33688BC7C3CB8DC6 5D39C916

;; Query time: 7 msec
;; SERVER: 2600:3c02::b#53(2600:3c02::b) (UDP)
;; WHEN: Sat Aug 28 01:39:27 UTC 2021
;; MSG SIZE  rcvd: 88

it went normal on our machine also.

Thanks for the help!