Deploy from url split #1715

IvanBorislavovDimitrov · 2025-09-29T11:43:37Z

LMCROSSITXSADEPLOY-3285

theghost5800 · 2025-10-01T08:29:47Z

...oudfoundry/multiapps/controller/process/configuration/FileUploadThreadPoolConfiguration.java

    }

+    @Bean(name = "deployFromUrlExecutor")
+    public ExecutorService deployFromUrlExecutor() {


Why this is needed when we have asyncFileUploadExecutor?

The asyncFileUploadExecutor is used for the actual upload. It has a queue of 20 which means it can store 20 jobs in the queue before starting a new thread.
Such executor will not scale well under load it might not update the update_at column in the database and the job might be consider as stale.
The goal of deployFromUrlExecutor is to update regularly updated_at value in the table and trigger the other executor to schedule the upload.

multiapps-controller-persistence/src/main/resources/META-INF/persistence.xml

theghost5800 · 2025-10-01T12:43:35Z

...in/resources/org/cloudfoundry/multiapps/controller/persistence/db/changelog/db-changelog.xml


    <include file="/org/cloudfoundry/multiapps/controller/persistence/db/changelog/db-changelog-1.192.0-persistence.xml"/>
+
+    <include file="/org/cloudfoundry/multiapps/controller/persistence/db/changelog/db-changelog-1.199.0-persistence.xml"/>


Do not forget to align the file with version which will be released

theghost5800 · 2025-10-01T13:00:44Z

...oudfoundry/multiapps/controller/process/configuration/FileUploadThreadPoolConfiguration.java

+                                      // As the threads are only updating a row and waiting it is ok to have more threads
+                                      30,
+                                      TimeUnit.SECONDS,
+                                      new SynchronousQueue<>()); // A synchronous queue is used so deploy from url jobs immediately start


What will happen if all threads are busy and there is already one task in the queue waiting for execution? What will happen with all other threads which tries to submit a new task?

This queue does not store any jobs, if there is a free thread (up to 50) it will immediately start the job, if there are no available threads (50 are used) it will reject the job. According to statistics there are maximum of around ~320 started operations on eu10-004 per hour (most use normal file upload), so 450 threads seems more than enough.

If backend reject file upload task, does the cli will retry on another instance or it will retry on the same one?

will check in the plugin

no, usually hit the same instance because of sticky sessions, except the last retry...
described in the backlog for the session handling

...eb/src/main/java/org/cloudfoundry/multiapps/controller/web/api/impl/FilesApiServiceImpl.java

...b/src/main/java/org/cloudfoundry/multiapps/controller/web/upload/AsyncUploadJobExecutor.java

...c/test/java/org/cloudfoundry/multiapps/controller/web/upload/AsyncUploadJobExecutorTest.java

...oudfoundry/multiapps/controller/process/configuration/FileUploadThreadPoolConfiguration.java

.../src/main/java/org/cloudfoundry/multiapps/controller/process/stream/CountingInputStream.java

.../src/main/java/org/cloudfoundry/multiapps/controller/core/util/ApplicationConfiguration.java

...e/src/main/java/org/cloudfoundry/multiapps/controller/persistence/dto/AsyncUploadJobDto.java

...c/main/java/org/cloudfoundry/multiapps/controller/persistence/model/PersistenceMetadata.java

theghost5800 · 2025-10-09T14:40:34Z

...c/main/java/org/cloudfoundry/multiapps/controller/web/upload/AsyncUploadJobOrchestrator.java

+            } finally {
+                lock.unlock();
+            }
+            MiscUtil.sleep(WAIT_TIME_BETWEEN_ASYNC_JOB_UPDATES_IN_MILLIS);


Can we try to use another approach by avoiding putting thread into sleep by using ScheduledExecutorService for monitoring and db update tasks? ScheduledExecutorService has methods to setup execution intervals and method execution delays. This is only idea, if performance is worse, then we can keep it as it is now.

It is possible to introduce such an executor but I see a couple of issues with this approach.
We will have 3 executors. Currently these 2 asyncFileUploadExecutor and deployFromUrlExecutor have the same amount of maximum load they can handle:
asyncFileUploadExecutor - 6 threads max + 20 waiting jobs (6th tread is started after the 20th waiting job)
deployFromUrlExecutor - max of 26 threads so it handles the same amount of jobs, without leading to unused threads or failing jobs because the other executor is full.

If we introduce a new executor only to remove the thread sleep it also will have to be synchronized with the above mentioned. Another possible problem is, there will be always one more thread allocated to start the monitoring until the thread is freed from deployFromUrlExecutor. The bigger problem is the synchronization, the deployFromUrlExecutor should start checking the queue of the new scheduled executor so it can decide whether to handle or reject the job. In my opinion seems like unnecessary complication of the code

theghost5800 · 2025-10-13T13:04:50Z

Evaluate security hotpost before merge

LMCROSSITXSADEPLOY-3285

sonarqubecloud · 2025-10-15T10:04:18Z

Quality Gate failed

Failed conditions
1 Security Hotspot

See analysis details on SonarQube Cloud

IvanBorislavovDimitrov force-pushed the deploy-from-url-split branch from ed41fe6 to 8754361 Compare September 29, 2025 11:46

IvanBorislavovDimitrov mentioned this pull request Sep 30, 2025

Remove x-cf-app-instance header usage cloudfoundry/multiapps-cli-plugin#245

Open

theghost5800 reviewed Oct 2, 2025

View reviewed changes

s-yonkov-yonkov reviewed Oct 6, 2025

View reviewed changes

...oudfoundry/multiapps/controller/process/configuration/FileUploadThreadPoolConfiguration.java Show resolved Hide resolved

IvanBorislavovDimitrov force-pushed the deploy-from-url-split branch 4 times, most recently from d9b5cda to 403c257 Compare October 7, 2025 11:45

s-yonkov-yonkov reviewed Oct 7, 2025

View reviewed changes

.../src/main/java/org/cloudfoundry/multiapps/controller/process/stream/CountingInputStream.java Show resolved Hide resolved

IvanBorislavovDimitrov force-pushed the deploy-from-url-split branch from 29a1b9e to c734c27 Compare October 7, 2025 13:23

s-yonkov-yonkov reviewed Oct 7, 2025

View reviewed changes

.../src/main/java/org/cloudfoundry/multiapps/controller/core/util/ApplicationConfiguration.java Show resolved Hide resolved

s-yonkov-yonkov reviewed Oct 8, 2025

View reviewed changes

...e/src/main/java/org/cloudfoundry/multiapps/controller/persistence/dto/AsyncUploadJobDto.java Outdated Show resolved Hide resolved

s-yonkov-yonkov reviewed Oct 8, 2025

View reviewed changes

...c/main/java/org/cloudfoundry/multiapps/controller/persistence/model/PersistenceMetadata.java Outdated Show resolved Hide resolved

IvanBorislavovDimitrov force-pushed the deploy-from-url-split branch from a56c5e5 to d2d4196 Compare October 8, 2025 10:01

theghost5800 reviewed Oct 9, 2025

View reviewed changes

s-yonkov-yonkov previously approved these changes Oct 9, 2025

View reviewed changes

IvanBorislavovDimitrov dismissed s-yonkov-yonkov’s stale review via e54cc45 October 13, 2025 09:32

IvanBorislavovDimitrov requested review from s-yonkov-yonkov and theghost5800 October 13, 2025 10:05

theghost5800 previously approved these changes Oct 13, 2025

View reviewed changes

s-yonkov-yonkov previously approved these changes Oct 14, 2025

View reviewed changes

IvanBorislavovDimitrov added 2 commits October 15, 2025 12:58

Remove ecplipse link shared cache

7989526

LMCROSSITXSADEPLOY-3285

Refactor deploy from URL API so it does not depend on app instance

0ae139b

LMCROSSITXSADEPLOY-3285

IvanBorislavovDimitrov dismissed stale reviews from s-yonkov-yonkov and theghost5800 via 0ae139b October 15, 2025 09:59

IvanBorislavovDimitrov force-pushed the deploy-from-url-split branch from 1f58831 to 0ae139b Compare October 15, 2025 09:59

theghost5800 approved these changes Oct 15, 2025

View reviewed changes

IvanBorislavovDimitrov merged commit f50a745 into master Oct 15, 2025
7 of 8 checks passed

IvanBorislavovDimitrov deleted the deploy-from-url-split branch October 15, 2025 10:07


		<include file="/org/cloudfoundry/multiapps/controller/persistence/db/changelog/db-changelog-1.192.0-persistence.xml"/>

		<include file="/org/cloudfoundry/multiapps/controller/persistence/db/changelog/db-changelog-1.199.0-persistence.xml"/>

Deploy from url split #1715

Deploy from url split #1715

Uh oh!

Conversation

IvanBorislavovDimitrov commented Sep 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

theghost5800 commented Oct 13, 2025

Uh oh!

sonarqubecloud bot commented Oct 15, 2025

Quality Gate failed

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

IvanBorislavovDimitrov commented Sep 29, 2025 •

edited

Loading