DNS resolution not working on macOS with "bind: operation not permitted" #237

thom-nic · 2020-03-25T14:21:43Z

Puma dev's DNS resolution doesn't appear to work at all anymore on MacOS Catalina.

I've done puma-dev -cleanup and even uninstalled/reinstalled from homebrew, then sudo puma-dev -setup; puma-dev -install again.

dig @127.0.0.1 -p 9253 blah.test

; <<>> DiG 9.10.6 <<>> @127.0.0.1 -p 9253 blah.test
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached

However everything looks good otherwise. Puma is running, resolver file in place, responds to HTTP but not DNS on port

$ ls -l /etc/resolver 
total 8
-rw-r--r--  1 thom  staff    55B Mar 24 10:22 test

$ cat /etc/resolver/test 
# Generated by puma-dev
nameserver 127.0.0.1
port 9253

$ ps ax|grep puma
39752   ??  S      0:00.04 /usr/local/Cellar/puma-dev/0.13/bin/puma-dev -launchd -dir ~/.puma-dev -d test -timeout 15m0s

$ cat ~/.puma-dev/blah
2999

$ curl -vH "Host: blah.test" localhost
*   Trying ::1...
* TCP_NODELAY set
* Connection failed
* connect to ::1 port 80 failed: Connection refused
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 80 (#0)
> GET / HTTP/1.1
> Host: blah.test
> User-Agent: curl/7.64.1
> Accept: */*
> 
< HTTP/1.1 500 Internal Server Error
< Date: Wed, 25 Mar 2020 13:18:25 GMT
< Content-Length: 0
< 
* Connection #0 to host localhost left intact
* Closing connection 0

$ tail -n20 ~/Library/Logs/puma-dev.log
2020/03/24 10:22:09 Existing valid puma-dev CA keypair found. Assuming previously trusted.
* Directory for apps: /Users/thom/.puma-dev
* Domains: test
* DNS Server port: 9253
* HTTP Server port: inherited from launchd
* HTTPS Server port: inherited from launchd
! Puma dev listening on http and https
* Generated proxy connection for 'blah' to http://127.0.0.1:2999
2020/03/25 09:18:21 http: proxy error: dial tcp 127.0.0.1:2999: connect: connection refused

Summary: HTTP proxy works, but DNS does not.

The text was updated successfully, but these errors were encountered:

nonrational · 2020-03-25T19:56:33Z

I wrote a quick test to ensure the DNS server works when it can listen properly, but it seems like 9253 isn't reachable at all on your machine.

Would you mind running sw_vers; nc -zv localhost 9253 and providing the result?

thom-nic · 2020-03-25T20:27:22Z

$ sw_vers; nc -zv localhost 9253
ProductName:	Mac OS X
ProductVersion:	10.15.3
BuildVersion:	19D76
nc: connectx to localhost port 9253 (tcp) failed: Connection refused
nc: connectx to localhost port 9253 (tcp) failed: Connection refused

ETA: Shouldn't 9253 be a UDP socket?
Edit 2: sudo lsof -i :9253 also returns nothing

nonrational · 2020-03-27T00:11:24Z

ETA: Shouldn't 9253 be a UDP socket?

puma-dev's DNS is intended to serve both TCP and UDP. https://github.com/puma/puma-dev/blob/master/dev/dns.go#L75-L85

Running two copies of puma-dev in the foreground, puma-dev bails on the http/s port, but it doesn't bail on the DNS port.

So, I expect puma-dev is failing to bind to 9253 and failing silently.

# keep this one running 
$ puma-dev
2020/03/26 15:41:02 Existing valid puma-dev CA keypair found. Assuming previously trusted.
* Directory for apps: /Users/norton/.puma-dev
* Domains: test
* DNS Server port: 9253
* HTTP Server port: 9280
* HTTPS Server port: 9283
! Puma dev listening on http and https

# attempt no. 1
$ puma-dev
2020/03/26 19:42:16 Existing valid puma-dev CA keypair found. Assuming previously trusted.
* Directory for apps: /Users/norton/.puma-dev
* Domains: test
* DNS Server port: 9253
* HTTP Server port: 9280
* HTTPS Server port: 9283
! Puma dev listening on http and https
2020/03/26 19:42:16 Error listening: listen tcp 127.0.0.1:9280: bind: address already in use

# attempt no. 2
$ puma-dev -http-port 9000 -https-port 90001
2020/03/26 19:42:48 Existing valid puma-dev CA keypair found. Assuming previously trusted.
* Directory for apps: /Users/norton/.puma-dev
* Domains: test
* DNS Server port: 9253
* HTTP Server port: 9000
* HTTPS Server port: 90001
! Puma dev listening on http and https

Also, this "works," which is disconcerting.

$ puma-dev -dns-port -1
2020/03/26 20:09:00 Existing valid puma-dev CA keypair found. Assuming previously trusted.
* Directory for apps: /Users/norton/.puma-dev
* Domains: test
* DNS Server port: -1
* HTTP Server port: 9280
* HTTPS Server port: 9283
! Puma dev listening on http and https

nonrational · 2020-03-27T00:11:29Z

@thom-nic will you try running the following?

puma-dev -dns-port 32000
sudo lsof -i :32000

edit: ~~I realized I'm using GNU nc. Not sure what the equivalent BSD nc flags are. One sec.~~ Your lsof approach is better for checking port binding.

nonrational · 2020-03-27T13:45:48Z

Confirmed that DNS port bind will fail silently. I have a WIP branch that will expose the error messages and bail.

thom-nic · 2020-03-27T14:00:21Z

For some reason when I try to run on OSX, puma-dev can't seem to bind to any HTTP/S port...?

puma-dev -dns-port 32000 -http-port 34999 -https-port 34998
2020/03/27 09:57:38 Existing valid puma-dev CA keypair found. Assuming previously trusted.
* Directory for apps: /Users/thom/.puma-dev
* Domains: test
* DNS Server port: 32000
* HTTP Server port: 34999
* HTTPS Server port: 34998
! Puma dev listening on http and https
2020/03/27 09:57:38 Error listening: listen tcp 127.0.0.1:34999: bind: operation not permitted

This may be part of the root cause of the issue I'm facing. Although the HTTP bind doesn't fail when running as a launch daemon, but then again in that case HTTP/S are being forwarded via launchd socket.

nonrational · 2020-03-27T14:05:16Z

Can you bind to those ports with other user processes e.g. rackup ?

thom-nic · 2020-03-27T14:29:00Z

I tried:

$ python3 -m http.server 34999
Serving HTTP on 0.0.0.0 port 34999 (http://0.0.0.0:34999/) ...

This prompt popped up:

(I clicked yes)
EDIT Firewall panel:

Oddly enough, nodejs isn't listed here and I do most of my development on node. Unsure why I'm not prompted for that. Or why I'm not prompted when I attempt to run puma-dev in the foreground.

EDIT 2: I manually added puma-dev in the firewall list and it still fails with the same error.

dangerous · 2020-04-08T02:52:25Z

I have the same issue, the firewall is stopping puma-dev from binding to the DNS port, however thought it might be worth mentioning that I am still using Mojave (10.14.6).

dangerous · 2020-04-08T02:56:50Z

I suspect it's because puma-dev is not signed

nonrational · 2020-04-08T17:22:33Z

I can't replicate the blocked behavior on my 10.14 machine. Haven't tried on 10.15 yet.

Can you run the following and report what you see?

$ /usr/libexec/ApplicationFirewall/socketfilterfw --getappblocked $(which puma-dev)
The application is not part of the firewall

And maybe try

$ sudo /usr/libexec/ApplicationFirewall/socketfilterfw --add $(which puma-dev)
Application at path ( /path/to/puma/dev/puma-dev ) added to firewall
$ sudo /usr/libexec/ApplicationFirewall/socketfilterfw --unblockapp $(which puma-dev)
Incoming connection to the application is permitted

swiknaba · 2020-04-15T14:02:29Z

Just for the record: it is also broken on my machine running macOS 10.14.6 (Mojave) after installing the following update from Apple:

XProtectPlistConfigData:

  Version:	2118
  Source:	Apple
  Install Date:	11.04.20, 18:23

the launchd still spawns, but routing traffic from .test domains does not resolve to localhost, but does a request to my regular DNS provider.

Didn't find much about that update from Apple, but it seems to be about blocking possibly malicious processes: https://support.apple.com/en-gb/HT207005

nonrational · 2020-04-15T20:48:23Z

@swiknaba I'm upgrading my mojave machine w/ latest security patches now and will attempt to reproduce.

For anyone who can reproduce the issue and is comfortable at the command-line, I'd ask that you try to disable gatekeeper temporarily and see if that fixes the port binding issue. That, at least, will allow us to test the unsigned hypothesis.

sudo spctl --master-disable will disable gatekeeper (and bring back the previous "allow from anywhere" option in system preferences).

sudo spctl --master-enable will then re-enable gatekeeper, once you're done testing.

nonrational · 2020-04-15T21:43:34Z

Finished latest security patches, still not having any issues with port bindings with firewall and gatekeeper enabled. ☹️

NoahFisher · 2020-04-21T16:38:34Z

+1 Same issue for me.

$ sw_vers; puma-dev
ProductName:    Mac OS X
ProductVersion: 10.15.4
BuildVersion:   19E287
2020/04/21 09:35:25 Existing valid puma-dev CA keypair found. Assuming previously trusted.
* Directory for apps: /Users/noah/.puma-dev
* Domains: test
* DNS Server port: 9253
* HTTP Server port: 9280
* HTTPS Server port: 9283
! Puma dev listening on http and https
2020/04/21 09:35:25 Error listening: listen tcp 127.0.0.1:9280: bind: operation not permitted

nonrational · 2020-04-21T17:19:25Z

~~I'm going to rename this issue to correctly reflect the port-binding issue.~~

@NoahFisher note that you had a problem binding to HTTPS, not DNS. And, in puma-dev v0.13, issues binding to dns/https fail silently, so my guess it you weren't able to bind to any of the 3 required ports. if you can install / build from master, you'll get better error messages.

Given I can't recreate this issue, having trouble debugging. Any additional diagnostic information is greatly appreciated.

NoahFisher · 2020-04-22T00:40:40Z

@nonrational - thanks for the reply. I just built from source and included the error output below.

Is there any other information that would be helpful to provide?

~/go/bin $ ./puma-dev
2020/04/21 17:38:01 Existing valid puma-dev CA keypair found. Assuming previously trusted.
* Directory for apps: /Users/noah/.puma-dev
* Domains: test
* DNS Server port: 9253
* HTTP Server port: 9280
* HTTPS Server port: 9283
! Puma dev running...
2020/04/21 17:38:01 ! HTTP Server failed: listen tcp 127.0.0.1:9280: bind: operation not permitted
! DNS Server failed: listen tcp 127.0.0.1:9253: bind: operation not permitted

~/go/bin $ ./puma-dev -V
Version: devel (go1.14.2)

nonrational · 2020-04-22T16:55:18Z

I had a working theory that SIP is involved, but if you're getting the same issue running a binary from your home directory, that rules that out.

@NoahFisher grasping at straws here, but

did you upgrade to catalina from mojave? or did you start with a fresh install?
what mac hardware are you using?
does your user have admin privileges?
have you tried the spctl approach discussed above? what were the results?
is your mac firewall on?
have you tried non-standard ports? e.g. puma-dev -http-port=13080 -https-port=13443 -dns-port=13053

I'm not sure what the difference between "connection refused" and "operation not permitted" error messages are, but I suspect that every participant in this thread is having a slightly different issue. 🙃

NoahFisher · 2020-04-22T20:37:27Z

I suspect that every participant in this thread is having a slightly different issue.

@nonrational - fair point! I tried on another machine today and was not able to reproduce.

I tried restarting my machine today and the issue is resolved. 🤷 Thanks for you help!

nonrational · 2020-04-23T18:36:43Z

Maddeningly, I just started getting this error in tests after upgrading to Catalina. 😡 Bright side, might be able to discover root cause, at least for the "not permitted" error. Still not sure about the "connection refused."

Edit: And of course it doesn't fail when being run in a debugger.
Edit2: Disabling gatekeeper didn't change anything, but turning off the macOS firewall fixed the error.
Edit3: https://github.com/golang/go/blob/83d25c61e45f2d38e1015d6eef50cccc0dc0d9b3/src/net/fd_unix.go#L59 this is the line that generates the error.

jormon · 2020-05-10T00:09:02Z

@nonrational I use zsh so at first I couldn't get your sudo /usr/libexec/ApplicationFirewall/socketfilterfw --unblockapp $(which puma-dev) to work because I think it was getting confused by the symlink action (installed via brew, so lots of symlinks). I replaced the $(which puma-dev) with the actual location of the binary and suddently (after a few puma-dev restarts) everything is working again.

For those using homebrew, the binary is somewhere like this: /usr/local/Cellar/puma-dev/0.13/bin/puma-dev but YMMV based on homebrew install and version of puma-dev, etc.

ProductName:    Mac OS X
ProductVersion: 10.15.4
BuildVersion:   19E287

stevenkaras · 2020-08-14T19:00:13Z

I just hit this as well, and wanted to report that the suggested firewall whitelisting worked, but only with the physical path to puma-dev (homebrew sets a symlink from /usr/local/bin/puma-dev to /usr/local/Cellar/puma-dev/0.14/bin/puma-dev). Just going to leave this here in case anyone else hits the same issue:

sudo /usr/libexec/ApplicationFirewall/socketfilterfw --add /usr/local/Cellar/puma-dev/0.14/bin/puma-dev
sudo /usr/libexec/ApplicationFirewall/socketfilterfw --unblockapp /usr/local/Cellar/puma-dev/0.14/bin/puma-dev

nonrational · 2020-08-16T13:51:39Z

Ah. Excellent find @stevenkaras! Adding a bit of portability for new versions, since 0.15 is right around the corner...

PUMA_DEV_BIN_PATH="$(brew --prefix puma/puma/puma-dev)/bin/puma-dev" # /usr/local/Cellar/puma-dev/0.14/bin/puma-dev
sudo /usr/libexec/ApplicationFirewall/socketfilterfw --add "$PUMA_DEV_BIN_PATH"
sudo /usr/libexec/ApplicationFirewall/socketfilterfw --unblockapp "$PUMA_DEV_BIN_PATH"

richardonrails · 2021-05-07T21:19:39Z

I've had puma-dev working without issue for a long time. Today it stopped working :( (possibly related to me doing brew update and brew upgrade today)

I've tried uninstalling (/etc/resolver/ becomes empty) then reinstalling:

$ puma-dev -V
Version: 0.16.0 (go1.16.2)

$ sw_vers
ProductName:	macOS
ProductVersion:	11.3.1
BuildVersion:	20E241

$ sudo puma-dev -setup
Password:
* Configuring /etc/resolver to be owned by me
* Changing '/etc/resolver/test' to be owned by me

$ puma-dev -install   
2021/05/07 17:09:43 Existing valid puma-dev CA keypair found. Assuming previously trusted.
* Use '/usr/local/bin/puma-dev' as the location of puma-dev
* Installed puma-dev on ports: http 80, https 443

But then in ~/Library/Logs/puma-dev.log:

2021/05/07 17:09:43 Existing valid puma-dev CA keypair found. Assuming previously trusted.
* Directory for apps: /Users/me/.puma-dev
* Domains: test
* DNS Server port: 9253
* HTTP Server port: inherited from launchd
* HTTPS Server port: inherited from launchd
! Puma dev running...
! DNS Server failed: listen tcp 127.0.0.1:9253: bind: operation not permitted

Note that here it shows the DNS port not binding.

However a bit different if I run in the foreground:

$ puma-dev
2021/05/07 17:13:35 Existing valid puma-dev CA keypair found. Assuming previously trusted.
* Directory for apps: /Users/me/.puma-dev
* Domains: test
* DNS Server port: 9253
* HTTP Server port: 9280
* HTTPS Server port: 9283
! Puma dev running...
2021/05/07 17:13:35 ! HTTP Server failed: listen tcp 127.0.0.1:9280: bind: operation not permitted

Here the HTTP port isn't binding.

I have no issue binding an HTTP port e.g. with python3 -m http.server 34999. Only puma-dev has an issue it seems.

I have tried repeating all this after running this:

PUMA_DEV_BIN_PATH="$(brew --prefix puma/puma/puma-dev)/bin/puma-dev"
sudo /usr/libexec/ApplicationFirewall/socketfilterfw --add "$PUMA_DEV_BIN_PATH"
sudo /usr/libexec/ApplicationFirewall/socketfilterfw --unblockapp "$PUMA_DEV_BIN_PATH"

Didn't help.

Same issue with custom ports:

puma-dev -dns-port 32000 -http-port 34999 -https-port 34998
2021/05/07 17:18:25 Existing valid puma-dev CA keypair found. Assuming previously trusted.
* Directory for apps: /Users/me/.puma-dev
* Domains: test
* DNS Server port: 32000
* HTTP Server port: 34999
* HTTPS Server port: 34998
! Puma dev running...
! HTTPS Server failed: listen tcp 127.0.0.1:34998: bind: operation not permitted
2021/05/07 17:18:25 ! HTTP Server failed: listen tcp 127.0.0.1:34999: bind: operation not permitted

Any advice or suggestions?

richardonrails · 2021-05-09T00:11:04Z

After a while I also see a few lines like this in puma-dev.log:

2021/05/07 20:46:05 http: TLS handshake error from 127.0.0.1:57366: EOF
2021/05/07 20:46:06 http: TLS handshake error from 127.0.0.1:57367: EOF
2021/05/07 20:46:06 http: TLS handshake error from 127.0.0.1:57368: tls: client using inappropriate protocol fallback
2021/05/07 20:47:38 http: TLS handshake error from 127.0.0.1:57418: EOF
2021/05/07 20:48:04 http: TLS handshake error from 127.0.0.1:57434: EOF
2021/05/07 20:48:04 http: TLS handshake error from 127.0.0.1:57435: EOF

richardonrails · 2021-05-19T16:51:52Z

And....... of course, after restarting my computer a SECOND time, it all works again. (I restarted once before posting everything above). Sorry for the noise.

dentarg · 2023-09-26T18:30:52Z

This is so much wtf. macOS just gives up after a while? (This is on macOS Ventura)

Yeah, I was playing around with puma-dev, installing and uninstalling it, building from source etc. (e.g. taking #322 for a spin).

I can run the net/http example program just fine: https://pkg.go.dev/net/http#hdr-Servers, but if I replace the whole Serve method

puma-dev/dev/http_darwin.go

Lines 51 to 75 in db9ec15

    
           func (h *HTTPServer) Serve(launchdSocket string) error { 
        
           	serv := http.Server{ 
        
           		Addr:    h.Address, 
        
           		Handler: h, 
        
           	} 
        
           	if launchdSocket == "" { 
        
           		return serv.ListenAndServe() 
        
           	} 
        
           	listeners, err := launch.SocketListeners(launchdSocket) 
        
           	if err != nil { 
        
           		return err 
        
           	} 
        
           	var t tomb.Tomb 
        
           	for _, l := range listeners { 
        
           		t.Go(func() error { 
        
           			return serv.Serve(l) 
        
           		}) 
        
           	} 
        
           	return t.Wait() 
        
           }

with http.ListenAndServe(":8080", nil), it still fails 🤯 2023/09/26 20:14:56 listen tcp :8080: bind: operation not permitted

Meanwhile, on my other macOS laptop, I just checked out puma-dev from source, built it and ran puma-dev -debug and it is working just fine. This is macOS Monterey though.

I guess I will be soon rebooting my Ventura laptop...

nonrational self-assigned this Mar 27, 2020

nonrational added the bug label Mar 27, 2020

nonrational mentioned this issue Mar 27, 2020

print & log port bind failures, dns-related test coverage, test improvements #239

Merged

6 tasks

nonrational changed the title ~~DNS resolver does not appear to be working on MacOS Catalina~~ DNS resolver does not appear to be working (intermittently) on MacOS Catalina Aug 13, 2020

nonrational mentioned this issue Oct 2, 2020

Unable to troubleshoot why puma-dev is not working #261

Closed

nonrational changed the title ~~DNS resolver does not appear to be working (intermittently) on MacOS Catalina~~ DNS resolution not working on macOS with "bind: operation not permitted" Oct 2, 2020

nonrational mentioned this issue Dec 1, 2020

Test fails on master #264

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DNS resolution not working on macOS with "bind: operation not permitted" #237

DNS resolution not working on macOS with "bind: operation not permitted" #237

thom-nic commented Mar 25, 2020

nonrational commented Mar 25, 2020 •

edited

Loading

thom-nic commented Mar 25, 2020 •

edited

Loading

nonrational commented Mar 27, 2020

nonrational commented Mar 27, 2020 •

edited

Loading

nonrational commented Mar 27, 2020

thom-nic commented Mar 27, 2020

nonrational commented Mar 27, 2020

thom-nic commented Mar 27, 2020 •

edited

Loading

dangerous commented Apr 8, 2020

dangerous commented Apr 8, 2020

nonrational commented Apr 8, 2020 •

edited

Loading

swiknaba commented Apr 15, 2020

nonrational commented Apr 15, 2020

nonrational commented Apr 15, 2020

NoahFisher commented Apr 21, 2020

nonrational commented Apr 21, 2020 •

edited

Loading

NoahFisher commented Apr 22, 2020

nonrational commented Apr 22, 2020

NoahFisher commented Apr 22, 2020

nonrational commented Apr 23, 2020 •

edited

Loading

jormon commented May 10, 2020 •

edited

Loading

stevenkaras commented Aug 14, 2020

nonrational commented Aug 16, 2020

richardonrails commented May 7, 2021 •

edited

Loading

richardonrails commented May 9, 2021

richardonrails commented May 19, 2021

dentarg commented Sep 26, 2023

DNS resolution not working on macOS with "bind: operation not permitted" #237

DNS resolution not working on macOS with "bind: operation not permitted" #237

Comments

thom-nic commented Mar 25, 2020

nonrational commented Mar 25, 2020 • edited Loading

thom-nic commented Mar 25, 2020 • edited Loading

nonrational commented Mar 27, 2020

nonrational commented Mar 27, 2020 • edited Loading

nonrational commented Mar 27, 2020

thom-nic commented Mar 27, 2020

nonrational commented Mar 27, 2020

thom-nic commented Mar 27, 2020 • edited Loading

dangerous commented Apr 8, 2020

dangerous commented Apr 8, 2020

nonrational commented Apr 8, 2020 • edited Loading

swiknaba commented Apr 15, 2020

nonrational commented Apr 15, 2020

nonrational commented Apr 15, 2020

NoahFisher commented Apr 21, 2020

nonrational commented Apr 21, 2020 • edited Loading

NoahFisher commented Apr 22, 2020

nonrational commented Apr 22, 2020

NoahFisher commented Apr 22, 2020

nonrational commented Apr 23, 2020 • edited Loading

jormon commented May 10, 2020 • edited Loading

stevenkaras commented Aug 14, 2020

nonrational commented Aug 16, 2020

richardonrails commented May 7, 2021 • edited Loading

richardonrails commented May 9, 2021

richardonrails commented May 19, 2021

dentarg commented Sep 26, 2023

nonrational commented Mar 25, 2020 •

edited

Loading

thom-nic commented Mar 25, 2020 •

edited

Loading

nonrational commented Mar 27, 2020 •

edited

Loading

thom-nic commented Mar 27, 2020 •

edited

Loading

nonrational commented Apr 8, 2020 •

edited

Loading

nonrational commented Apr 21, 2020 •

edited

Loading

nonrational commented Apr 23, 2020 •

edited

Loading

jormon commented May 10, 2020 •

edited

Loading

richardonrails commented May 7, 2021 •

edited

Loading