Add batchset for rendering a hierarchy of containers (single file)

[manics]

Alternative to #50

This is a single YAML/JSON file that allows rendering settings to be defined on projects, datasets, or particular datasets inside a project.

[snoopycrimecop]

Conflicting PR. Removed from build OMERO-plugins-push#640. See the console output for more details.
Possible conflicts:

• PR Add batchset for rendering a folder of datasets #50 manics 'Add batchset for rendering a folder of datasets'
  • src/omero_cli_render.py
• PR test command: also print image id #51 dominikl 'test command: also print image id'
  • src/omero_cli_render.py

--conflicts Conflict resolved in build OMERO-plugins-push#655. See the console output for more details.

[sbesson]

From the IDR perspective, we are regularly facing submissions where a higher level of granularity is required than the one allowed at the moment. Some form of core batch feature would be very useful.

My previous attempt to tackle this had been based on a distribution of YML files in a folder hierarchy matching the container hierarchy - see https://github.com/IDR/idr0067-king-yeastmeiosis/tree/master/experimentA. Looking at the description, I find this proposal where the hierarchy is defined within a single YAML much more elegant and easier to generate and manipulate. 👍

A few immediate comments on the proposed specification extension. I will start testing this feature with a few synthetic and real examples.

In addition, I have an IDR submission which could help driving this feature. I might need to implement two additional features once we agree on the initial extension : 1- support for images element (like idr0067 mentioned above) and some reciprocal form of batchget command allowing to programmatically retrieve rendering settings that have been set manually.

[sbesson]

Is this because the version is considered as global? Is it defined at the bottom of the YML file?

[manics]

I thought it would simplify things if we avoided mixed versioning though that's probably not a major problem to support. You're right that it should be specified somewhere though, so the question is should we require the version

• as part of every renderdef
• as part of the top-level version for this file i.e. version indicates the version of the renderdef spec + the version of this hierarchical definition
• in a dedicated top-level renderdef_version key
  And if renderdef has it's own version indicator how do we version this hierarchy spec?

[sbesson]

The new hierarchy makes sense and the renderdef key has the advantage of containerizing the render metadata. Looking at the examples, I was wondering whether a scheme where each node would be either a containers key (projects, datasets, ...) OR directly a render element had been tested/would be an option e.g.

projects:
  - name: Project
    channels:
    ...
  - name: Project B
    channels:
    ...

projects:
  datasets:
    - name: Dataset A
      channels:
      ...
    - name: Dataset B
      channels:
      ...

[manics]

I think keeping the renderdef self-contained will make this spec easier to update, otherwise when adding a key to either the container definition or the renderdef spec you'll need to consider future conflicts or potential confusion.

In addition keeping the renderdef under it's own key means this could potentially be extended to handle other metadata e.g. for setting tags or other annotations.

[manics]

Adding support for other containers (screen, plate, well) and images definitely makes sense. Ideally we'd have a simple graph-query method to obtain things like

• Project[name=p1]/Dataset[name=d1]
• Project[name=p2]/Dataset[name=d2]/Image[name=i2]
• Plate[name=p3]/Image[name=i3]

but we don't, so it'll require some repetitive HQL- maybe that HQL could go into the BlitzGateway though?

[manics]

I added jsonschema to validate the renderdef, and it's lead to a mysterious pytest error.

Traceback (most recent call last):
  File "setup.py", line 133, in <module>
    tests_require=['pytest', 'restview', 'mox3'],
  File "/opt/omero/server/venv3/lib64/python3.6/site-packages/setuptools/__init__.py", line 153, in setup
    return distutils.core.setup(**attrs)
  File "/usr/lib64/python3.6/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/usr/lib64/python3.6/distutils/dist.py", line 955, in run_commands
    self.run_command(cmd)
  File "/usr/lib64/python3.6/distutils/dist.py", line 974, in run_command
    cmd_obj.run()
  File "/opt/omero/server/venv3/lib64/python3.6/site-packages/setuptools/command/test.py", line 232, in run
    self.run_tests()
  File "setup.py", line 81, in run_tests
    import pytest
  File "/opt/omero/server/venv3/lib64/python3.6/site-packages/pytest.py", line 13, in <module>
    from _pytest.fixtures import fixture, yield_fixture
  File "/opt/omero/server/venv3/lib64/python3.6/site-packages/_pytest/fixtures.py", line 842, in <module>
    class FixtureFunctionMarker(object):
  File "/opt/omero/server/venv3/lib64/python3.6/site-packages/_pytest/fixtures.py", line 844, in FixtureFunctionMarker
    params = attr.ib(convert=attr.converters.optional(tuple))
TypeError: attrib() got an unexpected keyword argument 'convert'

Feel free to force push the last commit away!