OCI/ROCK Container: Avahi sees printer as remote service not as local service

[tillkamppeter]

When running the (immutable) OCI container of ipp-usb, from PR #106, from https://github.com/im0vishal/ipp-usb/tree/immutable-ipp-usb, commit 44b2745 ("Updated ReadMe to include runtime flags for data persistence"), started with the command lines

sudo docker volume create ipp-usb-data
sudo docker run -d --network host -v /dev/bus/usb:/dev/bus/usb:ro -v ipp-usb-data:/var/lib/ipp-usb --device-cgroup-rule='c 189:* rmw' --name ipp-usb ipp-usb:latest

the avahi-daemon on the host system sees ipp-usb as a remote service and not as a local service. So running ippfind --local does not list the printer but ippfind --remote lists it.

Especially client tools then do not replace the network host name (XXX.local) by localhost, and if ipp-usb is providing the printer only on the local machine (lo network interface) the printer cannot be accessed as the URI ipp://XXX.local:60000/ipp/print is advertised and used, while ipp://localhost:60000/ipp/print works.

This can be caused by the container running its own avahi-daemon and so the host's avahidaemon receiving the service from the container's avahi-daemon and not directly.

In avahi (I am using 0.8.0), the function avahi_server_is_service_local() in the file avahi-core/server.c of the Avahi source code is actually determining whether a service is local or not.

The function is left by the return 0; in these lines:

    e = find_entry(s, interface, protocol, key);
    avahi_key_unref(key);
    if (!e)
        return 0;

and therefore the service is see as not local. This I found out by applying the attached patch to the Avahi source code and looking for XXX in the /var/log/syslog file and I saw XXX6 but not XXX10 right afterwards.

check-local.patch.txt

[tillkamppeter]

@im0vishal @rudra-iitm Could you also have a look here?

[alexpevzner]

the avahi-daemon on the host system sees ipp-usb as a remote service and not as a local service. So running ippfind --local does not list the printer but ippfind --remote lists it.

It seems to be Avahi bug. In the loopback-only mode, ipp-usb uses the "localhost` name and the loopback interface index for announcing itself. Regarding of that, if system has at least one real network interface, Avahi silently replaces "loopback" with the "XXX.local" hostname.

[tillkamppeter]

The fact that Avahi considers a service local or remote is independent of the host name it reports.

If I run ipp-usb as classic installation (DEB package) or as the Snap, the DNS-SD record as I see with avahi-browse or avahi-discover also reports the XXX.local host name but ippfind --local lists the service anyway. With this, Avahi reporting the service as local, CUPS/libcups correct the XXX.local host name back to localhost when accessing.

Only if I use the OCI container the service gets listed by ippfind --remote and not by ippfind --local.

[alexpevzner]

What exactly it means, does Avahi considers service local or not?

[tillkamppeter]

If you use the Avahi client library to browse available services, you have to supply a browse callback function which Avahi calls for each service and you have to do something with Avahi's data sets inside you callback function, like saving the data in a data structure or displaying the data on the screen. One of the data items Avahi supplies is a bit field of flags, and there is one flag indicating that a service is local. Avahi tells by that that the service is running on the local machine and not on a remote machine in the network. And AFAIK this bit already existed before the lo support was introduced (the lo support is actually a tiny 2-line patch created by me).

In the source code of ippfind (tools/ippfind.c in the CUPS source) you find

#ifdef HAVE_AVAHI
/*
 * 'browse_callback()' - Browse devices.
 */

static void
browse_callback(
    AvahiServiceBrowser    *browser,    /* I - Browser */
    AvahiIfIndex           interface,   /* I - Interface index (unused) */
    AvahiProtocol          protocol,    /* I - Network protocol (unused) */
    AvahiBrowserEvent      event,       /* I - What happened */
    const char             *name,       /* I - Service name */
    const char             *type,       /* I - Registration type */
    const char             *domain,     /* I - Domain */
    AvahiLookupResultFlags flags,       /* I - Flags */
    void                   *context)    /* I - Services array */
{
  AvahiClient   *client = avahi_service_browser_get_client(browser);
                                        /* Client information */
  ippfind_srv_t *service;               /* Service information */

[...]

    case AVAHI_BROWSER_NEW:
       /*
        * This object is new on the network. Create a device entry for it if
        * it doesn't yet exist.
        */

        service = get_service((cups_array_t *)context, name, type, domain);

        if (flags & AVAHI_LOOKUP_RESULT_LOCAL)
          service->is_local = 1;
        break;
[...]

Here you see that a lot of data is provided to the callback function by Avahi and there is this bit field (flags), and the callback function is grabbing the AVAHI_LOOKUP_RESULT_LOCAL bit and saving the fact in the data structure which ippfind uses to store the list of available services.

This has nothing to do with the host name which Avahi reports (and Avahi seems to only report it when resolving a service, not when only browsing). Also the bit existed before we introduced lo and then there was never localhost reported as host name.

My feeling is that the OCI container is introducing its own avahi-daemon and this makes the service looking remote.

[alexpevzner]

My feeling is that the OCI container is introducing its own avahi-daemon and this makes the service looking remote.

Me too. After quick looking to the Avahi source, I think Avahi sets this flag only for the entries it manages by itself (i.e., for the services, registered in this instance of the Avahi daemon).

[tillkamppeter]

So @im0vishal @rudra-iitm could you modify all the OpenPrinting OCI container images (CUPS, ipp-usb, Printer Applications) so that they do not run their own avahi-daemon but register their services (printers) on the host's avahi-daemon, like the Snaps do.

[alexpevzner]

What if we tweak ippfind to consider services that use one of the 127.0.0.1 or ::1 addresses as local, regardless of the AVAHI_LOOKUP_RESULT_LOCAL flag?

[tillkamppeter]

@alexpevzner Now I realize what "local" actually is for Avahi, it is just a service registered on the avahi-daemon with which we are actually talking ...

[tillkamppeter]

What if we tweak ippfind to consider services that use one of the 127.0.0.1 or ::1 addresses as local, regardless of the AVAHI_LOOKUP_RESULT_LOCAL flag?

I see now that Avahi also reports the network interface to the browse callback. One could check whether it is lo and consider this local, too. But this would need to be done for every client, not only for ippfind. So we need to fix at several places.

Ideal would be that Avahi does this extra check (and also finds out about the hosts own name to be able to consider services from non-"lo" interfaces as local).

Could you report a bug on Avahi?

[alexpevzner]

@tillkamppeter,

after some reflection on that...

1. Registration on localhost will not work properly, if there are multiple instances of Avahi. Unlike the normal network, loopback registration doesn't actually imply any multicast service discovery. There is no such thing as multicast on the loopback interface. Instead, server registration just adds a record into the Avahi list of registered services; client query return this record.

If there are multiple instances of Avahi running on a same host (in containers), they don't communicate to each other and if client and server communicate to the different instances of Abahi, they will not find each other.

2. Considering service to be "local" if it is bound to some of local IP addresses will not work properly in all cases.

In a simple case, when there is no containers or if all containers share the same network configuration, it may work as expected. However, if network namespaces are in use, network interfaces/addresses may become "local" to one namespace and "remote" to other namespace of the same machine, so even the semantics of addresses to be local or remote becomes unclear and non-trivial.

So probably, the only thing we can do here is to update our containers to connect to the host-global Avahi instance instead of running their own.