SST1RSoXSDB has been timing out on loads of large scans through Tiled. This was first observed in December 2024 and discussed internally extensively with several work-arounds.
Summarizing briefly many discussions with NSLS2 DSSI/Tiled, the problem is that Dask's default configuration creates one thread per cpu core on the machine. Particularly on shared boxes with loads of cores, this easily swamps the load balancer and server cluster on the public-facing tiled.nsls2.bnl.gov. The fix outlined by Phil Maffetone is quite simple:
from dask.distributed import Client
from tiled.client import from_uri
c = from_uri("http://tiled.nsls2.bnl.gov")['csx']['raw']
run = c[202847]
da = run.primary['data']['axis1_image']
client = Client(
n_workers=config["n_workers"],
threads_per_worker=config["threads_per_worker"],
processes=False,
)
arr = da.read() # Or da[:,:, :]
client.close()
# Do other work with arr
This should be integrated into SST1RSoXSDB, essentially creating a dask.distributed.Client in __init__ with some reasonable number of workers (perhaps the lesser of 12 threads or the number of cores). The client should be .close()'d on class destruction.
SST1RSoXSDBhas been timing out on loads of large scans through Tiled. This was first observed in December 2024 and discussed internally extensively with several work-arounds.Summarizing briefly many discussions with NSLS2 DSSI/Tiled, the problem is that Dask's default configuration creates one thread per cpu core on the machine. Particularly on shared boxes with loads of cores, this easily swamps the load balancer and server cluster on the public-facing
tiled.nsls2.bnl.gov. The fix outlined by Phil Maffetone is quite simple:This should be integrated into
SST1RSoXSDB, essentially creating adask.distributed.Clientin__init__with some reasonable number of workers (perhaps the lesser of 12 threads or the number of cores). The client should be.close()'d on class destruction.