RS485 Hardware Support Brokes Other Uarts Recivers

[theoar]

Describe the bug

Enabling RS485 support for any UART interferes other UART Receiver. When you enable hw rts/cts support for one UART and start to transmit data from that UART other UART Receiver starts to choke - not all data that were send to that UART can be read via C API read function. The amout/frequency of missing bytes grows when the values of delay_rts_before_send" or "delay_rts_after_send" (struct serial_rs485) are increased. The issue appears on RS485 configuration on different UARTs. Enabling RS485 for one UART can interfere more than one UART Receiver.

Steps to reproduce the behaviour

Configure UART_X to support hardware RTS/CTS signals RS485.
Send data continuously from UART_X.
Send data continuously to UART_Y from external device (or from other UART on board).
Compare data written from UART_Y to data that were recieved by UART_Y - a lot of bytes will be missing.

Device (s)

Raspberry Pi CM4

System

Kernel: 6.1.13 (and above)

Logs

No response

Additional context

No response

[pelwell]

Your steps to reproduce does not mention RS485. Is that an omission, or is RS485 irrelevant?

[theoar]

Yes, its related to only enabling RS485 RTS/CTS support without any existing external hardware RS485 transceiver.

[6by9]

The implementation of delay_rts_before_send and delay_rts_after_send for PL011 looks pretty horrid.

If delay_rts_before_send is set, then it mdelays in pl011_rs485_tx_start at https://github.com/raspberrypi/linux/blob/rpi-6.1.y/drivers/tty/serial/amba-pl011.c#L1460
That's called from pl011_tx_chars, potentially called from pl011_int (https://github.com/raspberrypi/linux/blob/rpi-6.1.y/drivers/tty/serial/amba-pl011.c#L1583). Busy waiting for several milliseconds in an interrupt handler is going to be bad, and even worse as the uarts share an interrupt line. Delay too long and the FIFOs will overflow.

Likewise ``delay_rts_after_sendcalls mdelay in pl011_rs485_tx_stop```, called from ```pl011_stop_tx```, again called from ```pl011_tx_chars```, and in this case is almost guaranteed to be an interrupt context.

You say "Send data continuously from UART_X." Are you sending as long writes, keeping the FIFOs full, or writing a byte at a time? The former should avoid any of the delays as RTS should be kept asserted. The latter will be pinging it up and down with each write.

[6by9]

More information of the use case left on https://forums.raspberrypi.com/viewtopic.php?p=2128712 (now locked to avoid duplication of effort).

[pelwell]

Also bear in mind that unless UART_Y has flow control enabled, data delivery is best-effort only. Even with no hideous delays in interrupt context there may be some combination of system activity that causes data loss.

[6by9]

Also bear in mind that unless UART_Y has flow control enabled, data delivery is best-effort only. Even with no hideous delays in interrupt context there may be some combination of system activity that causes data loss.

Really? I was expecting the kernel to have decent buffering on a tty.

Flow control historically has been required for where the downstream device has a more limited onward bandwidth than the serial link between host and device, eg modem doing compression, or bluetooth with the radio channel. Yes going back further there was the case that the OS wasn't ready for more data, but that's not been a major concern for decades.

Seeing as there is no direct flow control capability on FIFO full in the uart, flow control is relying on the host to respond to a FIFO full event to update flow control signals, which makes it a chicken and egg situation.

theoar doesn't state the baud rate they're using.
115200 baud is a byte every 86usecs. PL011 supposedly has a 32byte FIFO in each direction, so it would be full in 2.7ms. Actually that's better than I was expecting (16550 only had a 16byte FIFO), but does put some numbers around the maximum delays that can potentially be tolerated.

IMHO the RTS delays ought to be handled by a workqueue or similar, rather than busy waiting in the driver. The timing will be a little looser, and care would be needed in case of the state changing whilst delaying, but it'd be a far nicer implementation from a system impact perspective.
Looking at the 8250 driver it is using an hrtimer to handle the RTS delays - that seems a better bet. It may be complicated by the use of DMA to feed the PL011 UART though.

[theoar]

I have 3 different devices connected to uarts: two rtu modbus devices, one 115200 badu (dev1), second 57600 (dev2) and the third (dev3) device is just custom protocol uart with 115200 badu. After enabling RS485 for dev2 I started to getting a lot of "timeouts" on modbus transmission with dev1. According to linked source code I just don't understand why even setting "delay_rts_*_send" to 0 didn't fix my issue (btw.: on logic analizer I saw that when the delay is set to 0 the real delay is about ~20us).

Edit: probably that code make the transmission sloppy: https://github.com/raspberrypi/linux/blob/19a1b03529363945fbbb4b9160fe8645809a9dce/drivers/tty/serial/amba-pl011.c#L1298C45-L1298C45. That explains why setting "delay_rts_*_send=0" reduced problem but didn't eliminated it. Also lowering transmission speed (<19200 badu) for RS485 transmission makes other transmissions to work even worse.

[6by9]

Holy jeepers - again busy waiting for the last byte to be sent is pretty grim, and potentially leads to the interrupt handler being blocked for a byte period. Actually worse as the interrupt looks to be that the TX FIFO has hit the configured threshold, so not necessarily only one byte.

Been there, done that, with 16550's many years ago. It only generates an interrupt on transmitter hold register empty, not transmitter empty, so each message got an extra byte stashed on the end. RTS could then be dropped when THRE was signalled, and therefore the last byte never made it onto the RS485 line. That doesn't work brilliantly with TX FIFOs though.

[nbuchwitz]

@linosanfilippo-kunbus / @l1k any thoughts on this?

[l1k]

Also bear in mind that unless UART_Y has flow control enabled, data delivery is best-effort only.

There is no flow control with RS-485.

PL011 supposedly has a 32byte FIFO in each direction

No, the PL011 on all Raspberry Pi SoCs up to and including BCM2711 is unfortunately an ancient revision with only 16 byte FIFOs. It identifies as revision 2 in the UARTPeriphID2 register:

fe201000.serial: ttyAMA0 at MMIO 0xfe201000 (irq = 21, base_baud = 0) is a PL011 rev2

The upgraded revision with 32 byte FIFOs would identify as revision 3 according to section 1.2 of the Technical Reference Manual. The mini-UART is even worse with only 8 byte FIFOs. It's easy to experience an RX FIFO overrun on a busy system with these shallow FIFOs.

However, that's not necessarily the root cause here. Each of the five PL011 instances on the BCM2711 is controlled by a separate driver with distinct locking. There is no common lock, hence no interference between the PL011 instances.

When I enabled hardware support for UART2 (RTS to drive RS485 transceiver) and started to transmit data (I was just writing from that URAT2) UART3 and URAT5 started to choke. By a choke I mean that not all data that were send TO UART5 and UART3 were received by a Kernel/Driver/System. The problem disappears when i disable support for RS485 for UART2 or when I stop transmitting data from UART2.

Are you using full-duplex or half-duplex mode on the RS-485-enabled UART? In half-duplex mode, the UART's receiver is disabled while it is transmitting. Maybe the registers of the five PL011 blocks are not properly separated and disabling the receiver on one UART also disables it on other UARTs? Try enabling full-duplex mode. You will see your own echo when transmitting on the RS-485-enabled UART, but it will allow verifying if that's what's causing the reception issues on the other UARTs. You can enable full-duplex mode either through the devicetree (rs485-rx-during-tx property) or through a TIOCSRS485 ioctl at runtime (set the SER_RS485_RX_DURING_TX bit).

Another explanation would be that the RTS pins of the five different UARTs are not routed correctly on your board: If RTS is asserted by the RS-485-enabled UART and the same RTS line is used by RS-232-enabled UARTs for flow control, naturally that can cause reception issues on those UARTs. A shared RTS line could not only be caused by incorrect wiring on the board, but also incorrect pin-controller settings, so make sure the RTS pins of the UARTs are configured correctly.

If all else fails, try using the mini-UART (UART 1) for RS-485. It's completely distinct from the five PL011s and RS-485 support for it was added to the kernel 3 years ago. It only has 8-byte FIFOs but that might not be an issue unless the baudrate is high.

[theoar]

I have checked both half and full duplex, but it didn't make any difference.
RTS/CTS was configured via RPi overlays e.g. "uart1,ctsrts" etc.
I will try to check mini-UART combined with other UARTS in a spare time/

[linosanfilippo-kunbus]

@linosanfilippo-kunbus / @l1k any thoughts on this?

I will try to have a look at this this weekend

[nbuchwitz]

@theoar if you still have the issue / setup at hand, it would be appreciated if you could test if #7180 brings any improvements?