Ensure we use GDAL 3.11.1 in Python bindings after 24.04 upgrade by spwoodcock · Pull Request #1958 · OpenDroneMap/ODM

spwoodcock · 2025-11-20T10:54:30Z

Follow on from #1956 and #1904

Closes #1953

Fix to ensure 24.04 upgrade is fully functional.

After building latest, I tested on the brighton beach dataset:

cd ODM
docker build . -t opendronemap/odm:latest
cd ..
git clone https://github.com/pierotofy/drone_dataset_brighton_beach.git
cd drone_dataset_brighton_beach

docker run -it --rm -v "$(pwd)/images:/code/images" -v "$(pwd)/odm_orthophoto:/code/odm_orthophoto" -v "$(pwd)/odm_texturing:/code/odm_texturing" docker.io/opendronemap/odm:latest --dtm --rerun-all

Output:

[INFO]    running gdaldem color-relief "/code/odm_dem/dtm.preview.tif" "/code/opendm/tiles/color_relief.txt" "/code/odm_dem/dtm.previewcolor.tif" -alpha -co ALPHA=YES
0...10...20...30...40...50...60...70...80...90...100 - done.
[INFO]    running gdaldem hillshade "/code/odm_dem/dtm.preview.tif" "/code/odm_dem/dtm.previewhillshade.tif" -z 1.0 -s 1.0 -az 315.0 -alt 45.0
0...10...20...30...40...50...60...70...80...90...100 - done.
[INFO]    running "/code/venv/bin/python3" "/code/opendm/tiles/hsv_merge.py" "/code/odm_dem/dtm.previewcolor.tif" "/code/odm_dem/dtm.previewhillshade.tif" "/code/odm_dem/dtm.previewcolored_hillshade.tif"
/code/venv/lib/python3.12/site-packages/osgeo/gdal.py:330: FutureWarning: Neither gdal.UseExceptions() nor gdal.DontUseExceptions() has been explicitly called. In GDAL 4.0, exceptions will be enabled by default.
  warnings.warn(
0...10...20...30...40...50...60...70...80...90...100 - done.
[INFO]    running gdal_translate -outsize 1400 0 -of png "/code/odm_dem/dtm.previewcolored_hillshade.tif" "/code/opensfm/stats/dtm.png" --config GDAL_CACHEMAX 37.35%
Input file size is 1400, 1379
0...10...20...30...40...50...60...70...80...90...100 - done.
[INFO]    Exporting report to /code/odm_report/report.pdf
2025-11-20 10:53:25,986 DEBUG: ICC profile found for image /tmp/tmprpo2ga6l/dsm_gradient.png - It will be inserted in the PDF document
2025-11-20 10:53:26,098 DEBUG: ICC profile found for image /tmp/tmplzyyha5e/overlap_diagram_legend.png - It will be inserted in the PDF document
2025-11-20 10:53:27,375 DEBUG: Final size summary of the biggest document sections:
2025-11-20 10:53:27,375 DEBUG: - pages: 4.3KiB
2025-11-20 10:53:27,375 DEBUG: - fonts: 201.0B
2025-11-20 10:53:27,375 DEBUG: - images: 8.7MiB
2025-11-20 10:53:27,375 DEBUG: - iccp: 950.0B
[INFO]    Finished odm_report stage
[INFO]    Running odm_postprocess stage
[INFO]    Post Processing
[INFO]    Finished odm_postprocess stage
[INFO]    No more stages to run
...
[INFO]    ODM app finished - Thu Nov 20 10:53:27  2025

Also replaced some dependencies with Python stdlib implementation:

dateutil --> datetime
repoze --> functools lru_cache
pytz --> datetime (only recommended for Python 3.9, as stdlib has all functionality
Removed used joblib
Removed shebang from run.py to make invocation more flexible and cross platform

* Changes to support building on Ubuntu 24.04 and windows-2022 Github runner. * Update Python to 3.12 * Install dependencies in virtual environment * Run python scripts from the virtual environment) * Update dependencies * Ubuntu dependencies in snap/snapcraft24.yaml * Python dependencies in requirements.txt * Windows dependencies built in OpenDroneMap/windows-deps repo * Update CUDA * 12.8.1 on Windows * 13.0.0 on Ubuntu) * Run tests as part of docker build * Use exact commits to specify dependencies that are built from source, instead of branch names * Use upstream versions of Libraries: * PDAL * PDAL-Python * untwine * ExifRead * draco * Build windows builds with -j2 * Use Micasense's latest version of dls.py * Update failing unit tests to match current behavior

spwoodcock · 2025-11-20T10:58:08Z

@smathermather @NathanMOlson sorry to ping you - if you get a chance to test on any other dataset please do 🙏

But everything seemed to function pretty nicely on my end, so hopefully this fixes the recent issues 🤞

Once merged, I will delete tag 3.6.0 and create tag v3.6.0 + copy the release content.
This should trigger a build of the lastest and 3.6.0 image, fixing #1944 too.

spwoodcock · 2025-11-20T15:53:02Z

 ENV DEBIAN_FRONTEND=noninteractive \
-    PYTHONPATH="$PYTHONPATH:/code/SuperBuild/install/lib/python3.12/dist-packages:/code/SuperBuild/install/bin/opensfm" \
-    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/code/SuperBuild/install/lib"
+    PYTHONPATH="/code/SuperBuild/install/lib/python3.12/dist-packages:/code/SuperBuild/install/bin/opensfm" \


I removed the env var as its empty and redundant during the build

spwoodcock · 2025-11-20T15:53:35Z

 https://github.com/OpenDroneMap/windows-deps/releases/download/2.6.0/rasterio-1.4.3-cp312-cp312-win_amd64.whl ; sys_platform == 'win32'
-gdal[numpy]==3.8.4 ; sys_platform == 'linux'
-gdal[numpy]==3.8.4 ; sys_platform == 'darwin'
+gdal[numpy]==3.11.1 ; sys_platform == 'linux'


Ensure GDAL version matches on Mac / Linux / Windows

spwoodcock · 2025-11-20T15:54:24Z

+    elif [[ "$UBUNTU_VERSION" == *"24.04"* ]]; then
+        echo "Enabling ubuntugis-unstable PPA for Ubuntu 24.04"
+        sudo $APT_GET install -y -qq --no-install-recommends software-properties-common
+        sudo add-apt-repository -y ppa:ubuntugis/ubuntugis-unstable


The main thing here - we need to use the current unstable version of 3.11.1 for the underlying GDAL libs

smathermather · 2025-11-20T20:13:02Z

Initial tests complete and successful, now just running oats defaults against it as it has some tests I wouldn't think to run.

smathermather · 2025-11-21T13:49:55Z

cd ~/Documents/git/odm
git checkout fix/gdal-3.11.1
docker build -t opendronemap/odm:gdalll .
cd ~/Documents/git/oats
./run all --tags 3.5.6,gdalll

Oats testing complete. Will review and report back shortly.

smathermather · 2025-11-21T14:38:01Z

spwoodcock · 2025-11-21T15:32:17Z

Did ortho generation for some of the test datasets fail?

smathermather · 2025-11-21T15:36:55Z

Looks like on split merge stages alone it's having a problem with dateutil:

[WARNING] Found a alignment done progress file in: /datasets/code/opensfm/alignment_done.txt
[INFO]    Submodel submodel_0000 has already been aligned.
[INFO]    Submodel submodel_0001 has already been aligned.
[INFO]    ========================
[INFO]    Processing submodel_0000
[INFO]    ========================
[INFO]    running /code/run.py --dsm --orthophoto-cutline --dem-euclidean-map --skip-3dmodel --skip-report --project-path /datasets/code/submodels submodel_0000
Traceback (most recent call last):
  File "/code/run.py", line 10, in <module>
    from opendm import log
  File "/code/opendm/log.py", line 6, in <module>
    import dateutil.parser
ModuleNotFoundError: No module named 'dateutil'

===== Dumping Info for Geeks (developers need this to fix bugs) =====
Child returned 1
Traceback (most recent call last):
  File "/code/stages/odm_app.py", line 82, in execute
    self.first_stage.run()
  File "/code/opendm/types.py", line 470, in run
    self.next_stage.run(outputs)
  File "/code/opendm/types.py", line 449, in run
    self.process(self.args, outputs)
  File "/code/stages/splitmerge.py", line 189, in process
    system.run(" ".join(map(double_quote, map(str, argv))), env_vars=os.environ.copy())
  File "/code/opendm/system.py", line 112, in run
    raise SubprocessException("Child returned {}".format(retcode), retcode)
opendm.system.SubprocessException: Child returned 1

===== Done, human-readable information to follow... =====

[ERROR]   Uh oh! Processing stopped because of strange values in the reconstruction. This is often a sign that the input data has some issues or the software cannot deal with it. Ha
ve you followed best practices for data acquisition? See https://docs.opendronemap.org/flying/

~~But let me make sure I have the right error: tried changing things and now unsure which log applies.~~

smathermather · 2025-11-21T15:40:59Z

~~Should be r~~Replicable with:

git clone https://github.com/pierotofy/drone_dataset_brighton_beach.git
cd drone_dataset_brighton_beach
docker run -it --rm -v "$(pwd)/images:/code/images" -v "$(pwd)/odm_orthophoto:/code/odm_orthophoto" -v "$(pwd)/odm_texturing:/code/odm_texturing" imagename_as_built --dsm --pc-ept --split 8 --split-overlap 20 --rerun-all

spwoodcock · 2025-11-21T22:46:40Z

Python dateutil is unmaintained and I dont think compatible with Python 3.12.

Simple replacement in log.py using stdlib

from datetime import datetime

start_time = datetime.fromisoformat(last_stage['startTime'].replace("Z", "+00:00"))

smathermather · 2025-11-22T02:58:45Z

Makes sense. Now the build isn't working for other reasons for me, but probably it's just Friday evening stuff.

spwoodcock · 2025-11-22T05:40:26Z

I can update on Monday morning - afk for the weekend 👍
(unless anyone else is around to do the replacement)

What server specs do you use to run OATS, if I want to do some additional testing?

I'm assuming it wouldn't run on my framework 32GB RAM, right?

…lacement

spwoodcock · 2025-11-22T13:07:45Z

I found a minute to update - fixed the dateutil issue 👍

Will run tests again on Monday!
(hopefully the whole OATS suite is working now 😄)

smathermather · 2025-11-22T16:31:29Z

Thanks Sam! Definitely stay away from that keyboard for the weekend. I'll be endeavoring to do the same after capturing these notes before I do.

New error:

2025-11-22 16:01:22,962 INFO: 1 partial reconstructions in total.
[INFO]    Aligning submodels...
[INFO]    /code/submodels/submodel_0000/opensfm/reconstruction.aligned.json is now /code/submodels/submodel_0000/opensfm/reconstruction.json
[INFO]    /code/submodels/submodel_0001/opensfm/reconstruction.aligned.json is now /code/submodels/submodel_0001/opensfm/reconstruction.json
[INFO]    /code/submodels/submodel_0002/opensfm/reconstruction.aligned.json is now /code/submodels/submodel_0002/opensfm/reconstruction.json
[INFO]    ========================
[INFO]    Processing submodel_0000
[INFO]    ========================
[INFO]    running /code/run.py --dsm --orthophoto-cutline --dem-euclidean-map --skip-3dmodel --skip-report --project-path /code/submodels submodel_0000
Traceback (most recent call last):
  File "/code/run.py", line 10, in <module>
    from opendm import log
  File "/code/opendm/log.py", line 8, in <module>
    from repoze.lru import lru_cache
ModuleNotFoundError: No module named 'repoze'

===== Dumping Info for Geeks (developers need this to fix bugs) =====
Child returned 1
Traceback (most recent call last):
  File "/code/stages/odm_app.py", line 82, in execute
    self.first_stage.run()
  File "/code/opendm/types.py", line 470, in run
    self.next_stage.run(outputs)
  File "/code/opendm/types.py", line 449, in run
    self.process(self.args, outputs)
  File "/code/stages/splitmerge.py", line 189, in process
    system.run(" ".join(map(double_quote, map(str, argv))), env_vars=os.environ.copy())
  File "/code/opendm/system.py", line 112, in run
    raise SubprocessException("Child returned {}".format(retcode), retcode)
opendm.system.SubprocessException: Child returned 1

===== Done, human-readable information to follow... =====

[ERROR]   Uh oh! Processing stopped because of strange values in the reconstruction. This is often a sign that the input data has some issues or the software cannot deal with it. Have you followed best practices for data acquisition? See https://docs.opendronemap.org/flying/

If I look for repoze.lru through the codebase, I get:

git grep repoze .
opendm/gpu.py:from repoze.lru import lru_cache
opendm/gsd.py:from repoze.lru import lru_cache
opendm/log.py:#from repoze.lru import lru_cache
requirements.txt:repoze.lru==0.7

and

git grep lru .
Dockerfile:RUN bash configure.sh installruntimedepsonly \                                                                                           
configure.sh:installruntimedepsonly() {
configure.sh:    echo "  installruntimedepsonly"
configure.sh:if [[ $1 =~ ^(install|installruntimedepsonly|reinstall|uninstall|installreqs|clean)$ ]]; then
configure_macos.sh:if [[ $1 =~ ^(install|installruntimedepsonly|reinstall|uninstall|installreqs|clean)$ ]]; then
gpu.Dockerfile:RUN bash configure.sh installruntimedepsonly \
opendm/gpu.py:from repoze.lru import lru_cache
opendm/gpu.py:@lru_cache(maxsize=None)
opendm/gpu.py:@lru_cache(maxsize=None)
opendm/gsd.py:from repoze.lru import lru_cache
opendm/gsd.py:@lru_cache(maxsize=None)
opendm/log.py:#from repoze.lru import lru_cache
opendm/log.py:#@lru_cache(maxsize=None)
portable.Dockerfile:RUN bash configure.sh installruntimedepsonly \
requirements.txt:repoze.lru==0.7

Unsure the exact functionality of repoze.lru, but looking at the repo, the youngest commits are 8 years old, and the website linked at pypi looks unmaintained.

If, for sake of testing we remove lru in opendm/log.py, we get a failure to import vmem, which is a bit strange:

[INFO]    ========================
[INFO]    Processing submodel_0000
[INFO]    ========================
[INFO]    running /code/run.py --dsm --orthophoto-cutline --dem-euclidean-map --skip-3dmodel --skip-report --project-path /code/submodels submodel_0000
Traceback (most recent call last):
  File "/code/run.py", line 10, in <module>
    from opendm import log
  File "/code/opendm/log.py", line 11, in <module>
    from vmem import virtual_memory
ModuleNotFoundError: No module named 'vmem'

===== Dumping Info for Geeks (developers need this to fix bugs) =====
Child returned 1
Traceback (most recent call last):
  File "/code/stages/odm_app.py", line 82, in execute
    self.first_stage.run()
  File "/code/opendm/types.py", line 470, in run
    self.next_stage.run(outputs)
  File "/code/opendm/types.py", line 449, in run
    self.process(self.args, outputs)
  File "/code/stages/splitmerge.py", line 189, in process
    system.run(" ".join(map(double_quote, map(str, argv))), env_vars=os.environ.copy())
  File "/code/opendm/system.py", line 112, in run
    raise SubprocessException("Child returned {}".format(retcode), retcode)
opendm.system.SubprocessException: Child returned 1

===== Done, human-readable information to follow... =====

[ERROR]   Uh oh! Processing stopped because of strange values in the reconstruction. This is often a sign that the input data has some issues or the software cannot deal with it. Have you followed best practices for data acquisition? See https://docs.opendronemap.org/flying/

smathermather · 2025-11-22T16:39:42Z

What server specs do you use to run OATS, if I want to do some additional testing?

I'm assuming it wouldn't run on my framework 32GB RAM, right?

I haven't tested for minimum specs, but your framework should be more than enough. Biggest issue if you run all of the tests is time. I would like to refactor it such that it allows for a mode that doesn't allow more than 4 cores per test and runs as many jobs in parallel as possible. Parallel processing and remote / cloud processing is on the roadmap, so it would align with the project vision.

Running everything sequentially on my 20-core, 14th gen desktop, it takes roughly 5.5 hours, and barely touches the 64GB RAM.

spwoodcock · 2025-11-23T00:48:27Z

I like debugging these sorts of things!

I'm sure it can all be worked out on Monday =)

Looks like the splitmerge workflow is the only issue now, with outdated Python dependencies.

repose.lru is definitely obsolete. Its a variable caching mechanism that is built directly into python stdlib now, in the functools package.

~~vmem is also unmaintained. psutil is a good up to date solution~~
~~psutil.virtual_memory()~~

spwoodcock · 2025-11-23T03:38:19Z

Had 2 mins in an airport - made a quick update to remove these deprecated libs.

Haven't tested though - there may be others - will continue and try to replace as much as possible with stdlib, modern libs, or perhaps just copied code if the library usage is really small (on Monday).

I think we can strip out some deps, for example beautifulsoup4

spwoodcock · 2025-11-23T04:20:46Z

~~Ah damn, looking at the latest CI build, looks like OpenSfm log.py uses vmem too 😅~~

~~We may need to update a fork of OpenSfm to support Python 3.12 fully - slightly more involved~~

smathermather · 2025-11-23T06:34:44Z

ok had I looked into this more deeply from the start, I would have seen that vmem is actually an implementation by Piero to remove the complexity of psutil, while keeping cross plaform support. OpenSfM also relies on it.

Yeah, I was surprised to see that too. Nice compatibility library.

Still needs import line removed for lru, but otherwise gets us on to the next step:

git diff
diff --git a/opendm/log.py b/opendm/log.py
index 66891500..07469418 100644
--- a/opendm/log.py
+++ b/opendm/log.py
@@ -7,8 +7,6 @@ import shutil
 import multiprocessing
 from functools import lru_cache
 
-from repoze.lru import lru_cache
-
 from opendm.arghelpers import double_quote, args_to_dict
 from vmem import virtual_memory

Still having issues with vmem, and too late tonight to look further, so I will put this here and take a look tomorrow evening:

[INFO]    ========================
[INFO]    Processing submodel_0000
[INFO]    ========================
[INFO]    running /code/run.py --dsm --orthophoto-cutline --dem-euclidean-map --skip-3dmodel --skip-report --project-path /code/submodels submodel_0000
Traceback (most recent call last):
  File "/code/run.py", line 10, in <module>
    from opendm import log
  File "/code/opendm/log.py", line 11, in <module>
    from vmem import virtual_memory
ModuleNotFoundError: No module named 'vmem'

===== Dumping Info for Geeks (developers need this to fix bugs) =====
Child returned 1
Traceback (most recent call last):
  File "/code/stages/odm_app.py", line 82, in execute
    self.first_stage.run()
  File "/code/opendm/types.py", line 470, in run
    self.next_stage.run(outputs)
  File "/code/opendm/types.py", line 449, in run
    self.process(self.args, outputs)
  File "/code/stages/splitmerge.py", line 189, in process
    system.run(" ".join(map(double_quote, map(str, argv))), env_vars=os.environ.copy())
  File "/code/opendm/system.py", line 112, in run
    raise SubprocessException("Child returned {}".format(retcode), retcode)
opendm.system.SubprocessException: Child returned 1

===== Done, human-readable information to follow... =====

[ERROR]   Uh oh! Processing stopped because of strange values in the reconstruction. This is often a sign that the input data has some issues or the software cannot deal with it. Have you followed best practices for data acquisition? See https://docs.opendronemap.org/flying/

I haven't checked yet to see if there's a problem with vmem caught in the build / install process, which seems likely.

spwoodcock · 2025-11-24T18:57:00Z

I had a hunch the issue above was because the run.py invocation wasn't correctly using the venv with all python dependencies present.

Likely caused the the shebang on run.py that hardcodes the usage of #!/usr/bin/python3.
I removed this & am testing again now: build + run on brighton beach splitmerge.

…ython3'

spwoodcock · 2025-11-24T19:19:02Z

@@ -1 +1 @@
-3.5.6


I updated this so that running ODM display the correct message:

[INFO] Initializing ODM 3.6.0 - Mon Nov 24 18:43:31 2025

spwoodcock · 2025-11-24T19:21:03Z

-        parser = SettingsParser(description='ODM is a command line toolkit to generate maps, point clouds, 3D models and DEMs from drone, balloon or kite images.',
-                            usage='%s [options] <dataset name>' % usage_bin,
-                            yaml_file=open(context.settings_path))
+        parser = argparse.ArgumentParser(


appsettings is ancient and is basically a thin wrapper around argparse.ArgumentParser, with the ability to load YAML settings.

I replaced this with a simple YAML reader to maintain support for it.

spwoodcock · 2025-11-24T19:21:47Z

-                    self.utc_time = (timezone.localize(utc_time) - epoch).total_seconds() * 1000.0
+                    # Ensure utc_time is timezone-aware
+                    if utc_time.tzinfo is None:
+                        utc_time = utc_time.replace(tzinfo=timezone.utc)


Uses stdlib instead of outdated pytz pkg

smathermather · 2025-11-24T20:35:25Z

0...10...20...30...40...50...60...70...80...90...100 - done.
[INFO]    Computing vertical cutlines
[INFO]    Computing horizontal cutlines
[INFO]    Generating polygons... this could take a bit.
[INFO]    Merging polygons
[INFO]    Largest cutline found: 2808.4021670588368 m^2
2025-11-24 20:20:41,132 DEBUG: GDAL data found in package: path='/code/venv/lib/python3.12/site-packages/fiona/gdal_data'.
2025-11-24 20:20:41,132 DEBUG: PROJ data found in package: path='/code/venv/lib/python3.12/site-packages/fiona/proj_data'.
2025-11-24 20:20:41,134 DEBUG: File doesn't exist. Creating a new one...
2025-11-24 20:20:41,137 DEBUG: GDAL data found in package: path='/code/venv/lib/python3.12/site-packages/fiona/gdal_data'.
2025-11-24 20:20:41,137 DEBUG: PROJ data found in package: path='/code/venv/lib/python3.12/site-packages/fiona/proj_data'.
2025-11-24 20:20:41,138 DEBUG: GDAL data found in package: path='/code/venv/lib/python3.12/site-packages/fiona/gdal_data'.
2025-11-24 20:20:41,138 DEBUG: PROJ data found in package: path='/code/venv/lib/python3.12/site-packages/fiona/proj_data'.
2025-11-24 20:20:41,138 DEBUG: Created layer cutline
2025-11-24 20:20:41,138 DEBUG: Writing started
2025-11-24 20:20:41,138 DEBUG: GDAL data found in package: path='/code/venv/lib/python3.12/site-packages/fiona/gdal_data'.
2025-11-24 20:20:41,138 DEBUG: PROJ data found in package: path='/code/venv/lib/python3.12/site-packages/fiona/proj_data'.
2025-11-24 20:20:41,139 DEBUG: Transaction supported: 1
2025-11-24 20:20:41,139 DEBUG: Starting transaction (initial)
2025-11-24 20:20:41,140 DEBUG: Flushed data source cache
2025-11-24 20:20:41,140 DEBUG: Flushed buffer
2025-11-24 20:20:41,140 DEBUG: Stopped session
100 - done.
Traceback (most recent call last):
  File "/code/run.py", line 63, in <module>
    retcode = app.execute()
              ^^^^^^^^^^^^^
  File "/code/stages/odm_app.py", line 118, in execute
    raise e
  File "/code/stages/odm_app.py", line 82, in execute
    self.first_stage.run()
  File "/code/opendm/types.py", line 465, in run
    self.next_stage.run(outputs)
  File "/code/opendm/types.py", line 465, in run
    self.next_stage.run(outputs)
  File "/code/opendm/types.py", line 465, in run
    self.next_stage.run(outputs)
  [Previous line repeated 7 more times]
  File "/code/opendm/types.py", line 444, in run
    self.process(self.args, outputs)
  File "/code/stages/odm_orthophoto.py", line 122, in process
    compute_cutline(tree.odm_orthophoto_tif, 
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/opendm/cutline.py", line 177, in compute_cutline
    sink.write({
  File "/code/venv/lib/python3.12/site-packages/fiona/collection.py", line 551, in write
    self.writerecords([record])
  File "/code/venv/lib/python3.12/site-packages/fiona/collection.py", line 541, in writerecords
    self.session.writerecs(records, self)
  File "fiona/ogrext.pyx", line 1660, in fiona.ogrext.WritingSession.writerecs
fiona.errors.GeometryTypeValidationError: Record's geometry type does not match collection schema's geometry type: 'MultiPolygon' != 'Polygon'

===== Dumping Info for Geeks (developers need this to fix bugs) =====
Child returned 1
Traceback (most recent call last):
  File "/code/stages/odm_app.py", line 82, in execute
    self.first_stage.run()
  File "/code/opendm/types.py", line 465, in run
    self.next_stage.run(outputs)
  File "/code/opendm/types.py", line 444, in run
    self.process(self.args, outputs)
  File "/code/stages/splitmerge.py", line 192, in process
    system.run(" ".join(map(double_quote, cmd)), env_vars=os.environ.copy())
  File "/code/opendm/system.py", line 112, in run
    raise SubprocessException("Child returned {}".format(retcode), retcode)
opendm.system.SubprocessException: Child returned 1

===== Done, human-readable information to follow... =====

[ERROR]   Uh oh! Processing stopped because of strange values in the reconstruction. This is often a sign that the input data has some issues or the software cannot deal with it. Have you followed best practices for data acquisition? See https://docs.opendronemap.org/flying/

edit: Currently testing just changing line 168 in opendm/cutline to be issued as a MultiPolygon.

smathermather

Change line 168 in opendm/cutline to be issued as a MultiPolygon and we should be good to go.

…orts

spwoodcock · 2025-11-25T08:49:00Z

I just successfully ran the splitmerge brighton beach dataset 🎉

I'll give running oats a go & see where I get to.

But otherwise, seems pretty good to me!

spwoodcock · 2025-11-25T09:57:01Z

./run all --tags 3.5.6,3.6.0

A bit confused that I had a failure on 3.5.6 tag for Brighton:

Begin dataset: brighton
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------Free Memory: 10.014 GB
Free SWAP: 6.419 GB
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 ✓ default with dsm and dtm
 ✓ fast orthophoto
 ✓ planar with 3d tiles
 ✓ sampling
 ✗ GeoTIFF alignment
   (in test file tests/build/brighton_3.5.6.bats, line 29)
     `[ -e "$output_dir/opensfm/stats/codem/registration.json" ]' failed
 ✗ LAZ alignment
   (in test file tests/build/brighton_3.5.6.bats, line 34)
     `[ -e "$output_dir/opensfm/stats/codem/registration.json" ]' failed

As long as everything for 3.6.0 passed I guess we are good though!
Waiting on the tests currently

spwoodcock · 2025-11-25T12:40:48Z

Full Brighton output:

Begin dataset: brighton
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------Free Memory: 10.014 GB
Free SWAP: 6.419 GB
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 ✓ default with dsm and dtm
 ✓ fast orthophoto
 ✓ planar with 3d tiles
 ✓ sampling
 ✗ GeoTIFF alignment
   (in test file tests/build/brighton_3.5.6.bats, line 29)
     `[ -e "$output_dir/opensfm/stats/codem/registration.json" ]' failed
 ✗ LAZ alignment
   (in test file tests/build/brighton_3.5.6.bats, line 34)
     `[ -e "$output_dir/opensfm/stats/codem/registration.json" ]' failed

6 tests, 2 failures

End dataset: brighton
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------Free Memory: 9.503 GB
Free SWAP: 6.388 GB

real    31m10.066s
user    0m1.764s
sys     0m2.122s
========================================================================================================================================================================================================================================================================================================================================================


Begin dataset: brighton
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------Free Memory: 9.502 GB
Free SWAP: 6.388 GB
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 ✓ default with dsm and dtm
 ✓ fast orthophoto
 ✓ planar with 3d tiles
 ✓ sampling
 ✓ GeoTIFF alignment
 ✗ LAZ alignment
   (in test file tests/build/brighton_3.6.0.bats, line 34)
     `[ -e "$output_dir/opensfm/stats/codem/registration.json" ]' failed

6 tests, 1 failure

Looks like GeoTIFF alignment fails for 3.5.6, but passes for 3.6.0.
LAZ alignment fails for both - not sure if this is expected!

spwoodcock · 2025-11-25T12:53:43Z

I got about half way through, mostly looks good apart from the LAZ failure for brighton (in both 3.5.6 and 3.6.0).

Had to stop as I was running out of memory on my small machine and needed to use the processor for something else!
(I'll run this again on a server at some point)

Begin dataset: banana
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------Free Memory: 12.236 GB
Free SWAP: 6.410 GB
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 ✓ default with dsm

1 test, 0 failures

End dataset: banana
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------Free Memory: 11.708 GB
Free SWAP: 6.415 GB

real    4m40.295s
user    0m0.479s
sys     0m0.314s
========================================================================================================================================================================================================================================================================================================================================================


Begin dataset: banana
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------Free Memory: 11.708 GB
Free SWAP: 6.415 GB
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 ✓ default with dsm

1 test, 0 failures

End dataset: banana
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------Free Memory: 11.075 GB
Free SWAP: 6.416 GB

real    5m9.968s
user    0m0.198s
sys     0m0.210s
========================================================================================================================================================================================================================================================================================================================================================


Begin dataset: brighton_masked
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------Free Memory: 11.076 GB
Free SWAP: 6.416 GB
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 ✓ fast-orthophoto with mask

1 test, 0 failures

End dataset: brighton_masked
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------Free Memory: 10.720 GB
Free SWAP: 6.419 GB

real    1m21.517s
user    0m0.482s
sys     0m0.408s
========================================================================================================================================================================================================================================================================================================================================================


Begin dataset: brighton_masked
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------Free Memory: 10.720 GB
Free SWAP: 6.419 GB
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 ✓ fast-orthophoto with mask

1 test, 0 failures

End dataset: brighton_masked
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------Free Memory: 10.485 GB
Free SWAP: 6.419 GB

real    1m18.993s
user    0m0.213s
sys     0m0.237s
========================================================================================================================================================================================================================================================================================================================================================


Begin dataset: brighton_no_exif
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------Free Memory: 10.485 GB
Free SWAP: 6.419 GB
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 ✓ fast-orthophoto with geo and dsm

1 test, 0 failures

End dataset: brighton_no_exif
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------Free Memory: 10.214 GB
Free SWAP: 6.419 GB

real    1m24.733s
user    0m0.797s
sys     0m0.844s
========================================================================================================================================================================================================================================================================================================================================================


Begin dataset: brighton_no_exif
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------Free Memory: 10.224 GB
Free SWAP: 6.419 GB
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 ✓ fast-orthophoto with geo and dsm

1 test, 0 failures

End dataset: brighton_no_exif
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------Free Memory: 10.022 GB
Free SWAP: 6.419 GB

real    1m21.528s
user    0m0.226s
sys     0m0.215s
========================================================================================================================================================================================================================================================================================================================================================


Begin dataset: brighton
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------Free Memory: 10.014 GB
Free SWAP: 6.419 GB
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 ✓ default with dsm and dtm
 ✓ fast orthophoto
 ✓ planar with 3d tiles
 ✓ sampling
 ✗ GeoTIFF alignment
   (in test file tests/build/brighton_3.5.6.bats, line 29)
     `[ -e "$output_dir/opensfm/stats/codem/registration.json" ]' failed
 ✗ LAZ alignment
   (in test file tests/build/brighton_3.5.6.bats, line 34)
     `[ -e "$output_dir/opensfm/stats/codem/registration.json" ]' failed

6 tests, 2 failures

End dataset: brighton
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------Free Memory: 9.503 GB
Free SWAP: 6.388 GB

real    31m10.066s
user    0m1.764s
sys     0m2.122s
========================================================================================================================================================================================================================================================================================================================================================


Begin dataset: brighton
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------Free Memory: 9.502 GB
Free SWAP: 6.388 GB
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 ✓ default with dsm and dtm
 ✓ fast orthophoto
 ✓ planar with 3d tiles
 ✓ sampling
 ✓ GeoTIFF alignment
 ✗ LAZ alignment
   (in test file tests/build/brighton_3.6.0.bats, line 34)
     `[ -e "$output_dir/opensfm/stats/codem/registration.json" ]' failed

6 tests, 1 failure

End dataset: brighton
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------Free Memory: 9.705 GB
Free SWAP: 6.388 GB

real    29m52.230s
user    0m1.126s
sys     0m1.823s
========================================================================================================================================================================================================================================================================================================================================================


Begin dataset: brighton
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------Free Memory: 9.706 GB
Free SWAP: 6.388 GB
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 ✓ fast-orthophoto with mask

1 test, 0 failures

End dataset: brighton
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------Free Memory: 9.681 GB
Free SWAP: 6.388 GB

real    0m6.358s
user    0m0.246s
sys     0m0.103s
========================================================================================================================================================================================================================================================================================================================================================


Begin dataset: brighton
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------Free Memory: 9.672 GB
Free SWAP: 6.388 GB
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 ✓ fast-orthophoto with mask

1 test, 0 failures

End dataset: brighton
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------Free Memory: 9.682 GB
Free SWAP: 6.388 GB

real    0m6.132s
user    0m0.176s
sys     0m0.117s
========================================================================================================================================================================================================================================================================================================================================================


Begin dataset: brighton
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------Free Memory: 9.676 GB
Free SWAP: 6.388 GB
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 ✓ fast-orthophoto with geo and dsm

1 test, 0 failures

End dataset: brighton
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------Free Memory: 9.675 GB
Free SWAP: 6.388 GB

real    0m5.786s
user    0m0.162s
sys     0m0.148s
========================================================================================================================================================================================================================================================================================================================================================


Begin dataset: brighton
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------Free Memory: 9.675 GB
Free SWAP: 6.388 GB
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 ✓ fast-orthophoto with geo and dsm

1 test, 0 failures

End dataset: brighton
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------Free Memory: 9.666 GB
Free SWAP: 6.388 GB

real    0m6.057s
user    0m0.120s
sys     0m0.146s
========================================================================================================================================================================================================================================================================================================================================================


Begin dataset: brighton
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------Free Memory: 9.667 GB
Free SWAP: 6.388 GB
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 ✓ Default with spaces in filenames

1 test, 0 failures

End dataset: brighton
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------Free Memory: 9.313 GB
Free SWAP: 6.389 GB

real    5m28.544s
user    0m0.380s
sys     0m0.436s
========================================================================================================================================================================================================================================================================================================================================================


Begin dataset: brighton
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------Free Memory: 9.315 GB
Free SWAP: 6.389 GB
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 ✓ Default with spaces in filenames

1 test, 0 failures

End dataset: brighton
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------Free Memory: 8.894 GB
Free SWAP: 6.390 GB

real    9m17.250s
user    0m0.487s
sys     0m0.502s
========================================================================================================================================================================================================================================================================================================================================================


Begin dataset: brighton
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------Free Memory: 8.916 GB
Free SWAP: 6.390 GB
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 ✓ default

1 test, 0 failures

End dataset: brighton
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------Free Memory: 7.822 GB
Free SWAP: 6.390 GB

real    10m0.592s
user    0m1.811s
sys     0m1.845s
========================================================================================================================================================================================================================================================================================================================================================


Begin dataset: brighton
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------Free Memory: 7.815 GB
Free SWAP: 6.390 GB
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 ✓ default

1 test, 0 failures

End dataset: brighton
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------Free Memory: 6.830 GB
Free SWAP: 6.390 GB

real    9m15.043s
user    0m0.392s
sys     0m0.712s
========================================================================================================================================================================================================================================================================================================================================================


Begin dataset: brighton_spaces
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------Free Memory: 6.821 GB
Free SWAP: 6.390 GB
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 ✓ Default with spaces in filenames

1 test, 0 failures

End dataset: brighton_spaces
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------Free Memory: 6.831 GB
Free SWAP: 6.390 GB

real    0m9.174s
user    0m0.474s
sys     0m0.275s
========================================================================================================================================================================================================================================================================================================================================================


Begin dataset: brighton_spaces
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------Free Memory: 6.830 GB
Free SWAP: 6.390 GB
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 ✓ Default with spaces in filenames

1 test, 0 failures

End dataset: brighton_spaces
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------Free Memory: 6.798 GB
Free SWAP: 6.390 GB

real    0m9.809s
user    0m0.334s
sys     0m0.297s
========================================================================================================================================================================================================================================================================================================================================================


Begin dataset: brighton_split_merge
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------Free Memory: 6.833 GB
Free SWAP: 6.390 GB
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 ✓ default

1 test, 0 failures

End dataset: brighton_split_merge
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------Free Memory: 6.802 GB
Free SWAP: 6.390 GB

real    0m8.128s
user    0m0.325s
sys     0m0.323s
========================================================================================================================================================================================================================================================================================================================================================


Begin dataset: brighton_split_merge
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------Free Memory: 6.824 GB
Free SWAP: 6.390 GB
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 ✓ default

1 test, 0 failures

End dataset: brighton_split_merge
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------Free Memory: 6.815 GB
Free SWAP: 6.390 GB

real    0m9.494s
user    0m0.327s
sys     0m0.186s
========================================================================================================================================================================================================================================================================================================================================================


Begin dataset: dji_video
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------Free Memory: 6.805 GB
Free SWAP: 6.390 GB
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 ✓ Default options

1 test, 0 failures

End dataset: dji_video
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------Free Memory: 6.555 GB
Free SWAP: 6.391 GB

real    12m54.597s
user    0m0.714s
sys     0m0.576s
========================================================================================================================================================================================================================================================================================================================================================


Begin dataset: dji_video
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------Free Memory: 6.551 GB
Free SWAP: 6.391 GB
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 ✓ Default options

1 test, 0 failures

End dataset: dji_video
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------Free Memory: 7.741 GB
Free SWAP: 6.392 GB

real    10m43.039s
user    0m0.359s
sys     0m0.378s
========================================================================================================================================================================================================================================================================================================================================================


Begin dataset: micasense
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------Free Memory: 7.742 GB
Free SWAP: 6.392 GB
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 ✓ default with dsm
 ✓ primary band
 ✓ fast orthophoto

3 tests, 0 failures

End dataset: micasense
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------Free Memory: 5.558 GB
Free SWAP: 6.381 GB

real    37m47.448s
user    0m7.654s
sys     0m6.993s
========================================================================================================================================================================================================================================================================================================================================================


Begin dataset: micasense
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------Free Memory: 5.554 GB
Free SWAP: 6.381 GB
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 ✓ default with dsm
 ✓ primary band
 ✓ fast orthophoto

3 tests, 0 failures

End dataset: micasense
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------Free Memory: 5.484 GB
Free SWAP: 6.380 GB

real    33m25.778s
user    0m1.554s
sys     0m4.545s
========================================================================================================================================================================================================================================================================================================================================================


Begin dataset: mica_split
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------Free Memory: 5.484 GB
Free SWAP: 6.380 GB
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 ✓ default with dsm, fast ortho

1 test, 0 failures

End dataset: mica_split
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------Free Memory: 4.862 GB
Free SWAP: 6.381 GB

real    2m23.908s
user    0m1.555s
sys     0m0.919s
========================================================================================================================================================================================================================================================================================================================================================


Begin dataset: mica_split
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------Free Memory: 4.862 GB
Free SWAP: 6.381 GB
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 ✓ default with dsm, fast ortho

1 test, 0 failures

End dataset: mica_split
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------Free Memory: 4.296 GB
Free SWAP: 6.381 GB

real    2m16.025s
user    0m0.194s
sys     0m0.276s
========================================================================================================================================================================================================================================================================================================================================================

smathermather · 2025-11-25T14:00:22Z

Looks like GeoTIFF alignment fails for 3.5.6, but passes for 3.6.0.
LAZ alignment fails for both - not sure if this is expected!

I saw this too, so good to get that fix validated.

I'm rerunning all now. Should be done in 6 hours or so, at which point we'll approve and merge this if I see nothing, which I expect.

Saijin-Naib · 2025-11-25T17:36:50Z

This reminds me I need to fixup how I added mem/swap reporting and printing pretty linebreaks 😭

spwoodcock · 2025-11-25T20:06:59Z

+            usage='%s [options] <dataset name>' % usage_bin)
+
+        # Load defaults from settings.yaml
+        if context.settings_path and os.path.exists(context.settings_path):


Do we know any examples of people that use a settings.yaml instead if command line args?

Wondering if we need to keep this functionality

smathermather

We are good to go.

spwoodcock · 2025-11-26T14:18:50Z

I was hoping these changes would be the end of it, but I neglected the portable / GPU builds! (and forgot that they are the actual prod images). They all fail: https://github.com/OpenDroneMap/ODM/actions

Related to #1960

smathermather and others added 3 commits October 22, 2025 16:25

Merge branch '24.04'

dcbde07

build: ensure we are using gdal 3.11.1 in all environments

8c01287

spwoodcock requested a review from smathermather November 20, 2025 10:54

spwoodcock self-assigned this Nov 20, 2025

spwoodcock changed the title ~~Ensure we use GDAL 3.11.1 in Python bindings~~ Ensure we use GDAL 3.11.1 in Python bindings as 24.04 upgrade Nov 20, 2025

spwoodcock changed the title ~~Ensure we use GDAL 3.11.1 in Python bindings as 24.04 upgrade~~ Ensure we use GDAL 3.11.1 in Python bindings after 24.04 upgrade Nov 20, 2025

This was referenced Nov 20, 2025

openmvs build failure with CUDA13 in Ubuntu 24.04 / 3.6.0 #1957

Open

Add ccache for cmake caching + optimise dockerfile ordering for cache usage #1959

Open

spwoodcock commented Nov 20, 2025

View reviewed changes

spwoodcock mentioned this pull request Nov 20, 2025

renderdem generates invalid bounds → OOM during odm_dem stage #1949

Closed

smathermather linked an issue Nov 20, 2025 that may be closed by this pull request

renderdem generates invalid bounds → OOM during odm_dem stage #1949

Closed

spwoodcock added 2 commits November 22, 2025 12:59

fix(logger): remove dateutil usage, replace with stdlib datetime

19cdbf8

docs: add comment for beautifulsoup usage, prior to possible lxml rep…

09e015e

…lacement

fix(odm): replace repoze lru_cache impl with stdlib python

f249280

spwoodcock force-pushed the fix/gdal-3.11.1 branch from ea510da to f249280 Compare November 24, 2025 14:55

spwoodcock added 4 commits November 24, 2025 15:41

build(odm): cleanup python dependency deprecation & unused code

add3982

build: add matplotlib back in, as required by CODEM

546af62

fix(odm): remove shebang from run.py to ensure venv is used

68fbc27

fix(odm): update VERSION for cmd line to correct 3.6.0

8c0cace

fix(odm): invoke secondary invokation of odm cli is prepended with 'p…

4c61d3b

…ython3'

spwoodcock commented Nov 24, 2025

View reviewed changes

smathermather requested changes Nov 25, 2025

View reviewed changes

fix(odm): use MultiPolygon for cutline fiona usage, remove unused imp…

75a1e1f

…orts

spwoodcock requested a review from smathermather November 25, 2025 08:31

spwoodcock commented Nov 25, 2025

View reviewed changes

smathermather approved these changes Nov 26, 2025

View reviewed changes

spwoodcock merged commit 7127aa2 into master Nov 26, 2025
2 checks passed

spwoodcock deleted the fix/gdal-3.11.1 branch November 26, 2025 09:07

spwoodcock mentioned this pull request Nov 26, 2025

Do we need to maintain support for old x86_64 CPUs (pre-2010)? i.e. Should we require AVX? #1960

Open

thadwald mentioned this pull request Dec 26, 2025

Fix subprocess venv activation for split-merge submodel execution on Linux/Mac #1972

Closed

Troloze mentioned this pull request May 4, 2026

Weirdly consistent DEM generation failure #2024

Closed

Conversation

spwoodcock commented Nov 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

spwoodcock commented Nov 20, 2025

Uh oh!

spwoodcock Nov 20, 2025

Choose a reason for hiding this comment

Uh oh!

spwoodcock Nov 20, 2025

Choose a reason for hiding this comment

Uh oh!

spwoodcock Nov 20, 2025

Choose a reason for hiding this comment

Uh oh!

smathermather commented Nov 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

smathermather commented Nov 21, 2025

Uh oh!

smathermather commented Nov 21, 2025

Uh oh!

spwoodcock commented Nov 21, 2025

Uh oh!

smathermather commented Nov 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

smathermather commented Nov 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

spwoodcock commented Nov 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

smathermather commented Nov 22, 2025

Uh oh!

spwoodcock commented Nov 22, 2025

Uh oh!

spwoodcock commented Nov 22, 2025

Uh oh!

smathermather commented Nov 22, 2025

Uh oh!

smathermather commented Nov 22, 2025

Uh oh!

spwoodcock commented Nov 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

spwoodcock commented Nov 23, 2025

Uh oh!

spwoodcock commented Nov 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

smathermather commented Nov 23, 2025

Uh oh!

spwoodcock commented Nov 24, 2025

Uh oh!

spwoodcock Nov 24, 2025

Choose a reason for hiding this comment

Uh oh!

spwoodcock Nov 24, 2025

Choose a reason for hiding this comment

Uh oh!

spwoodcock Nov 24, 2025

Choose a reason for hiding this comment

Uh oh!

smathermather commented Nov 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

smathermather left a comment

Choose a reason for hiding this comment

Uh oh!

spwoodcock commented Nov 25, 2025

Uh oh!

spwoodcock commented Nov 25, 2025

Uh oh!

spwoodcock commented Nov 25, 2025

Uh oh!

spwoodcock commented Nov 25, 2025

Uh oh!

smathermather commented Nov 25, 2025

Uh oh!

spwoodcock commented Nov 20, 2025 •

edited

Loading

smathermather commented Nov 20, 2025 •

edited

Loading

smathermather commented Nov 21, 2025 •

edited

Loading

smathermather commented Nov 21, 2025 •

edited

Loading

spwoodcock commented Nov 21, 2025 •

edited

Loading

spwoodcock commented Nov 23, 2025 •

edited

Loading

spwoodcock commented Nov 23, 2025 •

edited

Loading

smathermather commented Nov 24, 2025 •

edited

Loading