dhcpcd core dumps on FreeBSD 15-RELEASE

[athanp]

Dhcpcd (even with the latest #555 patch) not works on FreeBSD 15-RELEASE. It fails with the following errors.
The most worrying thing is that older versions (10.2.4) do not work either on 15R.

dhcpcd[1414]: main: control_open: Connection refused
dhcpcd[1417]: read_config: /usr/local/etc/dhcpcd.conf: Invalid argument
dhcpcd[1417]: SIOCSRTRFLUSH_IN6 22: Invalid argument
dhcpcd[1417]: SIOCSPFXFLUSH_IN6: Invalid argument

My production configuration that worked flawlessly for years

quiet
ipv6only
noipv6rs
release
background
vendorclassid
slaac private

clientid
allowinterfaces ng0
nohook test,hostname,resolv.conf,ntp.conf

option interface_mtu
option classless_static_routes
require dhcp_server_identifier

interface ng0
  ipv6rs
  ia_pd 1/::/56 bridge0/0/64

P.S. I'm seriously considering whether it's time to build dhcpcd binary with --disable-privsep, as it seems to cause more problems than it solves, at least on FreeBSD...

[rsmarples]

@athanp I've pushed a new change to the branch which adds some diagnosis for when this goes wrong.

And I still can't replicate it on a FreeBSD-15 vm :(

[athanp]

And I still can't replicate it on a FreeBSD-15 vm :(

@rsmarples I believe it's due to different setups.
I use mpd5 as a pppoe client that creates the Netgraph interface ng0 on which dhcpcd runs.
In seven similar setups that I use in production, dhcpcd exhibits exactly the same issue.
Even 10.2.4 that used to work flawlessly with FreeBSD 14.3 and earlier, exhibits now similar malfunctions on 15-Release under the same operating conditions. Perhaps something has changed in R15.

As I mentioned, I had to build dhcpcd with --disable-privsep in order to continue uninterrupted operation with R15.
This way everything works smoothly as always, and I'm seriously considering continuing to run it like this on production, at least until the issue is permanently fixed.

P.S. I will test your changes and keep you informed of the findings.

[rsmarples]

@athanp I've pushed another change to the branch, could you test it please?

I understand you have to disable privsep for production right now, but privsep (or more specifically capsicum) was a requirement for dhcpcd to be included in FreeBSD-base .... but that seems to have stalled again.

[perkelix]

@athanp What about the patch proposed here?

[athanp]

@athanp What about the patch proposed here?

This issue exists even with that patch applied, even with previous dhcpcd versions e.g 10.2.x
I had to compile it with --disable-privsep to be able to continue using it.

[rsmarples]

@athanp if you have the time and want to help provide some test debug, could you compile with CPPFLAGS+=-DPRIVSEP_DEBUG and post some logs.
It will be very very verbose. You could also email them to me at roy@marples.name if you prefer not to post here.

[asagratini]

I compiled this branch with CPPFLAGS+=-DPRIVSEP_DEBUG on FreeBSD 15.0 and I am attaching both stdout and dhcpcd logs to this response

dhcpcd.log

stdout.log

Here's my dhcpcd.conf:

logfile /var/log/dhcpcd/dhcpcd.log
noipv6rs
nohook resolv.conf, hostname, ntp.conf, test
duid
require dhcp_server_identifier
slaac private
option interface_mtu
denyinterfaces backplane data guests iot iptv ixl0 ixl1 lan lo0 pfsync0 sync # Don't touch other interfaces at all
interface voip
	noipv6
	nodhcp6
	nogateway

interface internet
	noipv4
	ipv6only
	ipv6rs								# enable routing solicitation for internet
	ia_na 1								# request a NA network address
	ia_pd 1 lan/0/64/0 guests/1/64/0 iot/2/64/0 backplane/3/64/0	# request a PD /64 prefix delegation and assign it to

Let me know if you need anything else, thank you

[rsmarples]

I'm using FreeBSD 15.0-RELEASE (GENERIC) releng/15.0-n280995-7aedc8de6446 and I just cannot replicate this.
I have this running on a QEMU VM on my NetBSD host and I've created a lot of tap devices and renamed them to match the above and use the same config and it just works for me.

Is it possible for anyone to downgrade to FreeBSD-14 to see if the issue is there still so we can work out if it's only FreeBSD-15 affected?

[asagratini]

no worries! Everything seems to be working as expected on FreeBSD 14.3, as also pointed out by @athanp....

To give you a bit more context on my network set up, here's the configuration for the voip interface:

[ale@freebsd15 ~]$ ifconfig lan
lan: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 9000
	options=4c100b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,TXCSUM_IPV6,HWSTATS,MEXTPG>
	ether 64:9d:99:b2:8a:a6
	hwaddr 00:00:00:00:00:00
	inet 10.255.200.3 netmask 0xffffff00 broadcast 10.255.200.255
	inet 10.255.200.1 netmask 0xffffffff broadcast 10.255.200.1 vhid 10
	inet6 fe80::669d:99ff:feb2:8aa6%lan prefixlen 64 scopeid 0x5
	laggproto lacp lagghash l2,l3,l4
	laggport: ixl0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
	laggport: ixl1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
	groups: lagg
	carp: BACKUP vhid 10 advbase 1 advskew 10
	      peer 224.0.0.18 peer6 ff02::12
	media: Ethernet autoselect
	status: active
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

while the interface called internet is created using mpd5 (it uses netgraph, if this is relevant) as pppoe client to authenticate with my ISP.

Here's the relevant configuration for the network interface used by mpd:

[ale@freebsd15 ~]$ ifconfig data
data: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 9000
	options=4400000<TXCSUM_IPV6,MEXTPG>
	ether 64:9d:99:b2:8a:a6
	groups: vlan
	vlan: 6 vlanproto: 802.1q vlanpcp: 1 parent interface: lan
	media: Ethernet autoselect
	status: active
	nd6 options=1<PERFORMNUD>

and here is my mpd.conf file:

isp:
  create bundle static B1
  set bundle enable ipv6cp
  set bundle enable ipcp
  set iface name internet
  set iface enable tcpmssfix
  set iface route default
  set iface up-script /usr/local/etc/mpd5/up.sh
  #set iface down-script /usr/local/etc/mpd5/down.sh
  set ipcp ranges 0.0.0.0/0 0.0.0.0/0
  create link static L1 pppoe
  set link action bundle B1
  set auth authname redact3d
  set auth password r3dacted
  set link max-redial 0
  set link mtu 1492
  set link mru 1492
  set link keep-alive 10 60
  set pppoe iface data
  set pppoe service ""
  open
default:
  load isp

I know we can compile the software with --disable-privsep option, but it does not seem a solution to me :)
Let me know if you need anything else, thank you

[rsmarples]

Hmmm, both of you use pppoe via mpd5. Is that the common cause? I'll see if I can setup a PPPoE server and mirror the config.
Likely take some time as I really hate debugging PPPoE and I'm not good at setting this up.

[asagratini]

exactly, this is what I see in my dhcpcd.log file if only the voip interface is available on the 15.0 system:

Dec 28 13:37:46 [87128]: dhcpcd-10.3.0 starting
Dec 28 13:37:46 [87688]: DUID 00:01:00:01:30:e1:7b:63:64:9d:99:b2:8a:a6
Dec 28 13:37:46 [87688]: ps_root_readerrorcb: recvmsg returned 44, expecting 60
Dec 28 13:37:46 [87688]: ps_root_readerrorcb: recvmsg returned 44, expecting 60
Dec 28 13:37:46 [87688]: ps_root_readerrorcb: recvmsg returned 44, expecting 60
Dec 28 13:37:46 [87688]: ps_root_readerrorcb: recvmsg returned 44, expecting 60
Dec 28 13:37:46 [87688]: ps_root_readerrorcb: recvmsg returned 32, expecting 60
Dec 28 13:37:46 [87688]: ps_root_readerrorcb: recvmsg returned 28, expecting 24
Dec 28 13:37:46 [87688]: SIOCSRTRFLUSH_IN6 89: Bad message
Dec 28 13:37:46 [87688]: script_runreason: Inappropriate ioctl for device
Dec 28 13:37:46 [87688]: ps_root_readerrorcb: recvmsg returned 24, expecting 634
Dec 28 13:37:46 [87688]: script_runreason: Bad message
Dec 28 13:37:46 [87688]: voip: IAID 99:b2:8a:a6
Dec 28 13:37:46 [87688]: ps_root_readerrorcb: recvmsg returned 610, expecting 3414963751808951367
Dec 28 13:37:46 [87688]: ipv6_readsecret: cannot read secret: Bad message
Dec 28 13:37:46 [87688]: ps_root_readerrorcb: recvmsg returned 24, expecting 32
Dec 28 13:37:46 [87688]: ipv6_readsecret: cannot write secret: Bad message
Dec 28 13:37:46 [87688]: voip: ipv6_start: Bad message
Dec 28 13:37:46 [87688]: voip: /var/db/dhcpcd/voip.lease: Invalid argument
Dec 28 13:37:46 [87688]: voip: soliciting a DHCP lease
Dec 28 13:37:47 [87688]: voip: offered 10.25.14.88 from 172.24.244.52
Dec 28 13:37:47 [87688]: voip: leased 10.25.14.88 for 43345 seconds
Dec 28 13:37:47 [87688]: ps_root_readerrorcb: recvmsg returned 24, expecting 312
Dec 28 13:37:47 [87688]: dhcp_writefile: /var/db/dhcpcd/voip.lease: Bad message
Dec 28 13:37:47 [87688]: ps_root_readerrorcb: recvmsg returned 288, expecting 24
Dec 28 13:37:47 [87688]: rt_build: if_initrt: Bad message
Dec 28 13:37:47 [87688]: voip: adding route to 10.25.0.0/18
Dec 28 13:37:47 [87688]: if_route (ADD): Device not configured
Dec 28 13:37:47 [87688]: voip: adding default route via 10.25.0.1
Dec 28 13:38:20 [87688]: voip: pid 0 deleted default route via 10.25.0.1

so that possibly means things are working as expected, if I read it right?

[rsmarples]

@asagratini ah, @athanp said that 10.2.4 was working fine on FreeBSD-14, can you be more specific please? Have to you tested 10.3.0 and/or the privsep-recvmsg branch we are testing here?

[asagratini]

right, all the information I shared is when using 10.3.0 version with the privsep-recvmsg branch, apologies if I did not mention this earlier in the thread

[rsmarples]

exactly, this is what I see in my dhcpcd.log file if only the voip interface is available on the 15.0 system:

Dec 28 13:37:46 [87128]: dhcpcd-10.3.0 starting
Dec 28 13:37:46 [87688]: DUID 00:01:00:01:30:e1:7b:63:64:9d:99:b2:8a:a6
Dec 28 13:37:46 [87688]: ps_root_readerrorcb: recvmsg returned 44, expecting 60
Dec 28 13:37:46 [87688]: ps_root_readerrorcb: recvmsg returned 44, expecting 60
Dec 28 13:37:46 [87688]: ps_root_readerrorcb: recvmsg returned 44, expecting 60
Dec 28 13:37:46 [87688]: ps_root_readerrorcb: recvmsg returned 44, expecting 60
Dec 28 13:37:46 [87688]: ps_root_readerrorcb: recvmsg returned 32, expecting 60
Dec 28 13:37:46 [87688]: ps_root_readerrorcb: recvmsg returned 28, expecting 24
Dec 28 13:37:46 [87688]: SIOCSRTRFLUSH_IN6 89: Bad message
Dec 28 13:37:46 [87688]: script_runreason: Inappropriate ioctl for device
Dec 28 13:37:46 [87688]: ps_root_readerrorcb: recvmsg returned 24, expecting 634
Dec 28 13:37:46 [87688]: script_runreason: Bad message
Dec 28 13:37:46 [87688]: voip: IAID 99:b2:8a:a6
Dec 28 13:37:46 [87688]: ps_root_readerrorcb: recvmsg returned 610, expecting 3414963751808951367
Dec 28 13:37:46 [87688]: ipv6_readsecret: cannot read secret: Bad message
Dec 28 13:37:46 [87688]: ps_root_readerrorcb: recvmsg returned 24, expecting 32
Dec 28 13:37:46 [87688]: ipv6_readsecret: cannot write secret: Bad message
Dec 28 13:37:46 [87688]: voip: ipv6_start: Bad message
Dec 28 13:37:46 [87688]: voip: /var/db/dhcpcd/voip.lease: Invalid argument
Dec 28 13:37:46 [87688]: voip: soliciting a DHCP lease
Dec 28 13:37:47 [87688]: voip: offered 10.25.14.88 from 172.24.244.52
Dec 28 13:37:47 [87688]: voip: leased 10.25.14.88 for 43345 seconds
Dec 28 13:37:47 [87688]: ps_root_readerrorcb: recvmsg returned 24, expecting 312
Dec 28 13:37:47 [87688]: dhcp_writefile: /var/db/dhcpcd/voip.lease: Bad message
Dec 28 13:37:47 [87688]: ps_root_readerrorcb: recvmsg returned 288, expecting 24
Dec 28 13:37:47 [87688]: rt_build: if_initrt: Bad message
Dec 28 13:37:47 [87688]: voip: adding route to 10.25.0.0/18
Dec 28 13:37:47 [87688]: if_route (ADD): Device not configured
Dec 28 13:37:47 [87688]: voip: adding default route via 10.25.0.1
Dec 28 13:38:20 [87688]: voip: pid 0 deleted default route via 10.25.0.1

so that possibly means things are working as expected, if I read it right?

No, things are not working that is just co-incidence. Example errors below:

Dec 28 13:37:46 [87688]: ps_root_readerrorcb: recvmsg returned 44, expecting 60
Dec 28 13:37:46 [87688]: SIOCSRTRFLUSH_IN6 89: Bad message
Dec 28 13:37:46 [87688]: voip: /var/db/dhcpcd/voip.lease: Invalid argument

None of these should happen.

[athanp]

I'm sorry I couldn't help with debug info and the rest, but I'm away from my lab on a Christmas vacation trip.
What is certain is that with a direct ethernet connection (no PPPoE) it works without problems, while with PPPoE it does not.
In the latter case, it was necessary to build it with --disable-privsep, which works without any problems with the mpd5 pppoe client and the ng0 created interface.