From edb4f5b46806a2281dff8a2e89e0e378e070e9f8 Mon Sep 17 00:00:00 2001 From: Timon de Groot Date: Tue, 14 Jan 2025 17:35:51 +0100 Subject: [PATCH 01/10] docs: Add Magento 2 Remote Storage documentation --- ...onfigure-remote-storage-for-magento-2-x.md | 72 +++++++++++++++++++ 1 file changed, 72 insertions(+) create mode 100644 docs/ecommerce-applications/magento-2/how-to-configure-remote-storage-for-magento-2-x.md diff --git a/docs/ecommerce-applications/magento-2/how-to-configure-remote-storage-for-magento-2-x.md b/docs/ecommerce-applications/magento-2/how-to-configure-remote-storage-for-magento-2-x.md new file mode 100644 index 00000000..ccde0a94 --- /dev/null +++ b/docs/ecommerce-applications/magento-2/how-to-configure-remote-storage-for-magento-2-x.md @@ -0,0 +1,72 @@ +--- +myst: + html_meta: + description: Configure remote storage for Magento 2.x. Learn how to configure + Magento 2 to start storing files in your bucket using a single command. + title: How to Configure Remote Storage for Magento 2.x | Hypernode +--- + +# How to Configure Remote Storage for Magento 2.x + +Magento 2.x supports remote storage for media files, import/export files, and other files. +This feature allows you to store files in a remote storage location, such as an Amazon S3 bucket, instead of storing them on the server itself. + +This can be useful for many reasons, such as: +- Offloading storage from your server, reducing the load on your server, and improving performance. +- Allows you to make use of [horizontal scaling](../../hypernode-platform/autoscaling/how-does-horizontal-autoscaling-work), as you can easily add more servers without having to worry about syncing files between them. +- Allows for effortless storage capacity scaling, as you can easily increase the storage capacity of your remote storage location. +- Serving assets from a CDN, which can improve the performance of your website. + +## Configuring the application + +Configuring Magento 2 to start storing files in your bucket is done using a single command. +If you're using a different provider than AWS S3, you need to specify the `--remote-storage-endpoint` option. + +**AWS S3** + +```bash +bin/magento setup:config:set \ + --remote-storage-driver="aws-s3" \ + --remote-storage-bucket="my_bucket_name" \ + --remote-storage-region="my-aws-region" \ + --remote-storage-key="abcd1234" \ + --remote-storage-secret="abcd1234" +``` + +**Other S3 compatible providers** + +```bash +bin/magento setup:config:set \ + --remote-storage-driver="aws-s3" \ + --remote-storage-bucket="my_bucket_name" \ + --remote-storage-region="my-aws-region" \ + --remote-storage-key="abcd1234" \ + --remote-storage-secret="abcd1234" \ + --remote-storage-endpoint="https://my-s3-compatible.endpoint.com" +``` + +## Syncing the files + +Instead of running (which is Magento's official way to do this): + +```bash +bin/magento remote-storage:sync +``` + +One can run the following instead to really speed up the process: + +```bash +aws s3 sync pub/media/ s3://my_bucket_name/media/ +aws s3 sync var/import_export s3://my_bucket_name/import_export +``` + +This is much faster than Magento's built-in sync, because `aws s3 sync` uploads files concurrently. + +## The storage flag file in the bucket + +Magento's S3 implementation creates a test file called `storage.flag`, which is basically created to test if the connection works. So this is not a magic file to mark anything ([source](https://github.com/magento/magento2/blob/6f4805f82bb7511f72935daa493d48ebda3d9039/app/code/Magento/AwsS3/Driver/AwsS3.php#L104)). + +## Magento remote storage documentation + +- [Configure Remote Storage](https://experienceleague.adobe.com/en/docs/commerce-operations/configuration-guide/storage/remote-storage/remote-storage) +- [Configure AWS S3 bucket for remote storage](https://experienceleague.adobe.com/en/docs/commerce-operations/configuration-guide/storage/remote-storage/remote-storage-aws-s3) From 48f7dbce5256fa2d36c8563782e0482b4c42ffab Mon Sep 17 00:00:00 2001 From: Timon de Groot Date: Tue, 14 Jan 2025 17:36:23 +0100 Subject: [PATCH 02/10] docs/autoscaling: Link to remote storage --- .../autoscaling/how-does-horizontal-autoscaling-work.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/docs/hypernode-platform/autoscaling/how-does-horizontal-autoscaling-work.md b/docs/hypernode-platform/autoscaling/how-does-horizontal-autoscaling-work.md index 880e95c3..63fa7fcc 100644 --- a/docs/hypernode-platform/autoscaling/how-does-horizontal-autoscaling-work.md +++ b/docs/hypernode-platform/autoscaling/how-does-horizontal-autoscaling-work.md @@ -191,9 +191,12 @@ But please make sure to configure RabbitMQ without the default guest user. Please make sure to enable remote storage for your application and configure it correctly as only AWS-s3 remote storage drivers are supported. -Make sure that the `remote_storage` key is present in the Magento2 configuration file at `/app/etc/env.php` with the correct config. +#### Make sure database storage is disabled & remote storage is enabled and configured -More information about [S3 Remote Storage with Magento2](https://experienceleague.adobe.com/en/docs/commerce-operations/configuration-guide/storage/remote-storage/remote-storage-aws-s3) +Remote storage is a requirement for Horizontal autoscaling. This is because the media files should be stored on a remote storage location instead of the local filesystem. +This way the media files are available across multiple Hypernodes when the app is scaled up, and after scaling down the media files are still available. + +Follow our documentation on [remote storage for Magento 2](../../ecommerce-applications/magento-2/how-to-configure-remote-storage-for-magento-2-x) to configure remote storage. ## Enabling Horizontal Autoscaling From f7d5123aa4b7525f4077f8bcb1e09f1b51a2952b Mon Sep 17 00:00:00 2001 From: Timon de Groot Date: Tue, 14 Jan 2025 17:38:33 +0100 Subject: [PATCH 03/10] docs/autoscaling: Polish up little details --- .../how-does-horizontal-autoscaling-work.md | 92 ++++++++++--------- 1 file changed, 49 insertions(+), 43 deletions(-) diff --git a/docs/hypernode-platform/autoscaling/how-does-horizontal-autoscaling-work.md b/docs/hypernode-platform/autoscaling/how-does-horizontal-autoscaling-work.md index 63fa7fcc..3702aa57 100644 --- a/docs/hypernode-platform/autoscaling/how-does-horizontal-autoscaling-work.md +++ b/docs/hypernode-platform/autoscaling/how-does-horizontal-autoscaling-work.md @@ -63,7 +63,9 @@ We will divide them between Hypernode-specific and Application-specific requirem #### Operating system -- The operating system of the Hypernode should be Debian Bookworm. If you would like to upgrade the os of your Hypernode, feel free to contact our support team for help. https://www.hypernode.com/en/contact/ +Horizontal autoscaling is only available on Hypernodes with the Debian Bookworm operating system. + +If you would like to upgrade the operating system of your Hypernode, feel free to [contact our support team](https://www.hypernode.com/en/contact/) for help. #### Make sure the Hypernode is a production plan @@ -75,32 +77,20 @@ To make use of Horizontal autoscaling, Varnish should be enabled and configured You can check if Varnish is enabled on your Hypernode by running ```console -hypernode-systemctl settings varnish_enabled -``` - -Example output if Varnish is enabled: - -```console +app@abcdef-hnclusterweb1-magweb-tbbm:~$ hypernode-systemctl settings varnish_enabled varnish_enabled is set to value True ``` If Varnish is not enabled, you can [enable Varnish](../varnish/how-to-enable-varnish-on-hypernode.md) by following the documentation -If varnish is enabled on your Hypernode, your Magento store should also be configured to make use of varnish. -You can verify if the Varnish host is configured correctly by running the following command from the Magento root: +If Varnish is enabled on your Hypernode, your Magento store should also be configured to make use of varnish. -```console -php bin/magento config:show system/full_page_cache/varnish/backend_host -``` +Verify that your VCL has the health probe disabled. While we do support the health probe, we do not recommend to use it in a basic setup. +The health probe can be removed by removing the `.probe` from your `backend default` section in the VCL file. +If you do, however, want to use the health probe, make sure that you set the backend host to `127.0.0.1` instead of `localhost`. -The output should show `varnish` as backend. If it is configured as something else (like `localhost` or `127.0.0.1`), you can update it by running which sets the backend host to `varnish` instead. +Additionally, make sure the IP range `10.0.0.0/24` is allowed under the `acl purge` section in the Varnish VCL file. The `acl purge` section should look like the following: -```console -php bin/magento config:set system/full_page_cache/varnish/backend_host varnish -``` - -Additionally make sure the IP range `10.0.0.0/24` is set to the `acl_purge` section in the Varnish VCL file. The `acl_purge` section should look something similar:loaded Varnish VCL. - -```console +```vcl acl purge { "localhost"; "10.0.0.0/24"; @@ -112,12 +102,7 @@ acl purge { The configured MySQL version should be 5.7 or above. You can check the enabled MySQL version by running the following command. ```console -hypernode-systemctl settings mysql_version -``` - -Example output if MySQL version is 8.0: - -```console +app@abcdef-hnclusterweb1-magweb-tbbm:~$ hypernode-systemctl settings mysql_version mysql_version is set to value 8.0 ``` @@ -141,14 +126,22 @@ The database, cache, session and queue of the application must be configured wit Make sure the env variables (db, cache, session, queue) are not using localhost. -### Application Specific Requirements - Magento2 +### Application Specific Requirements - Magento 2 #### Supported CMS Horizontal autoscaling is available for Magento 2.4.7 and higher. To make use of Horizontal autoscaling, there are a couple of other requirements the application should meet. -#### Enable and configure Redis Persistent +#### Make sure Redis cache is configured properly + +Please make sure Redis cache is configured properly in the Magento 2 configuration file at `/app/etc/env.php`. + +For the Redis hostname, use `redismaster` instead of `localhost` or `127.0.0.1`. + +More information about [Redis on Hypernode](../../ecommerce-applications/magento-2/how-to-configure-redis-for-magento-2.md). + +#### Enable and configure Redis sessions Redis persistent is another requirement before you can make use of Horizontal autoscaling. The persistent instance will be used to store the sessions so we can access the same sessions from the Horizontal autoscale Hypernodes. @@ -156,19 +149,14 @@ The persistent instance will be used to store the sessions so we can access the You can check if Redis Persistent is enabled on your Hypernode by running ```console -hypernode-systemctl settings redis_persistent_instance -``` - -Example output if Redis Persistent is enabled: - -```console +app@abcdef-hnclusterweb1-magweb-tbbm:~$ hypernode-systemctl settings redis_persistent_instance redis_persistent_instance is set to value True ``` If Redis Persistent instance is not enabled, you can enable the second Redis instance for sessions you run the command: ```console -hypernode-systemctl settings redis_persistent_instance --value True +app@abcdef-hnclusterweb1-magweb-tbbm:~$ hypernode-systemctl settings redis_persistent_instance True ``` Make sure Redis session is configured as [described](../../ecommerce-applications/magento-2/how-to-configure-redis-for-magento-2.md#configure-magento-2-to-use-redis-as-the-session-store) in our docs @@ -176,20 +164,38 @@ Please notice the Redis host in the setup documentation. The Redis host should b #### Make sure Elasticsearch/Opensearch is configured properly -Please make sure Elasticsearch or Opensearch host is set to `elasticsearchmaster` in the Magento2 configuration file at `/app/etc/env.php` -More information about [Elasticsearch on Hypernode](../../hypernode-platform/tools/how-to-use-elasticsearch-on-hypernode.md) +Please make sure Elasticsearch or Opensearch host is set to `elasticsearchmaster` your Magento 2 configuration. + +To set the hostname, you can run the following command for OpenSearch: + +```console +app@abcdef-example-magweb-cmbl:~$ bin/magento config:set catalog/search/opensearch_server_hostname elasticsearchmaster +``` + +For ElasticSearch: + +```console +app@abcdef-example-magweb-cmbl:~$ bin/magento config:set catalog/search/elasticsearch7_server_hostname elasticsearchmaster +``` + +More information about [Elasticsearch on Hypernode](../../hypernode-platform/tools/how-to-use-elasticsearch-on-hypernode.md). #### Make sure RabbitMQ configured properly -Please make sure RabbitMQ host is set to `rabbitmqmaster` in the Magento2 configuration file at `/app/etc/env.php` -More information about [RabbitMQ o Hypernode](../../best-practices/database/how-to-run-rabbitmq-on-hypernode.md) +Please make sure RabbitMQ host is set to `rabbitmqmaster` in the Magento 2 configuration file at `/app/etc/env.php` +More information about [RabbitMQ on Hypernode](../../best-practices/database/how-to-run-rabbitmq-on-hypernode.md) -There is a rabbitmq user provisioned by Hypernode called hypernode-admin as a non-default user. But you can also configure RabbitMQ with a new different user of your own. -But please make sure to configure RabbitMQ without the default guest user. +There is a rabbitmq user provisioned by Hypernode called `hypernode-admin` as a non-default user, but you can also configure RabbitMQ with a new different user of your own. +Please make sure to configure RabbitMQ without the default guest user. -#### Make sure Database storage is disabled & Remote storage is enabled and configured. +For example, you can run the following command to change your RabbitMQ config in Magento 2: -Please make sure to enable remote storage for your application and configure it correctly as only AWS-s3 remote storage drivers are supported. +```bash +bin/magento setup:config:set \ + --amqp-host="rabbitmqmaster" \ + --amqp-user="my_rabbitmq_user" \ + --amqp-password="my_rabbitmq_password" +``` #### Make sure database storage is disabled & remote storage is enabled and configured From d3817d51de226d98ce2aed4ee652800e1fcdab74 Mon Sep 17 00:00:00 2001 From: Timon de Groot Date: Tue, 14 Jan 2025 17:39:02 +0100 Subject: [PATCH 04/10] docs/rabbitmq: Add nodes on creating users better magento2 config --- .../how-to-run-rabbitmq-on-hypernode.md | 41 +++++++++++-------- 1 file changed, 24 insertions(+), 17 deletions(-) diff --git a/docs/best-practices/database/how-to-run-rabbitmq-on-hypernode.md b/docs/best-practices/database/how-to-run-rabbitmq-on-hypernode.md index 3e145071..3d5c2890 100644 --- a/docs/best-practices/database/how-to-run-rabbitmq-on-hypernode.md +++ b/docs/best-practices/database/how-to-run-rabbitmq-on-hypernode.md @@ -55,7 +55,7 @@ You can run this command to restart RabbitMQ: hypernode-servicectl restart rabbitmq-server ``` -## Accessing RabbitMQ +## Accessing the RabbitMQ admin interface - RabbitMQ only binds on localhost - The default admin account is username `guest` and password `guest`. You can change and add users via the admin interface. @@ -67,22 +67,29 @@ Use your browser to go to `localhost:55672` and logon using guest/guest. Another way to access the admin interface is via [hypernode-vpn](https://changelog.hypernode.com/changelog/release-6064-rabbitmq-can-be-accessed-via-the-hypernode-vpn/) -## RabbitMQ and Magento - -You need to make some changes in Magento in order to use RabbitMQ. For example in `/data/web/magento2/app/etc/env.php`: - -```php -'queue' => - array ( - 'amqp' => - array ( - 'host' => 'rabbitmqmaster', - 'port' => '5672', - 'user' => 'guest', - 'password' => 'guest', - 'virtualhost' => '/', - ), - ), +## Creating RabbitMQ users + +You can also create your own RabbitMQ users. This can be done by in the RabbitMQ admin interface, as follows: + +1. Go to the RabbitMQ admin interface +2. Click on the `Admin` tab +3. Click on `Add a user` +4. Fill in the username and password +5. Click on `Add user` +6. Click on the user you just created +7. Click on `Set permission` + +## RabbitMQ and Magento 2 + +To configure RabbitMQ in Magento 2, you can run the following command: + +```bash +bin/magento setup:config:set \ + --amqp-host="rabbitmqmaster" \ + --amqp-port="5672" \ + --amqp-user="guest" \ + --amqp-password="guest" \ + --amqp-virtualhost="/" ``` Note: Hypernode provisions a non-default user called `hypernode-admin` but you are free to create new users. From 2cda8c0d40cf3566dce9e99ca207f6f8730122bf Mon Sep 17 00:00:00 2001 From: Timon de Groot Date: Tue, 14 Jan 2025 17:39:56 +0100 Subject: [PATCH 05/10] bin/watch: Use cross-platform watchdog --- bin/watch | 82 ++++++++++++++++++------------------ requirements/development.txt | 2 +- 2 files changed, 42 insertions(+), 42 deletions(-) diff --git a/bin/watch b/bin/watch index 9934f83e..0e8f6ef4 100755 --- a/bin/watch +++ b/bin/watch @@ -1,8 +1,9 @@ #!/usr/bin/env python3 -import datetime +import os import subprocess - -from inotify.adapters import InotifyTree +from datetime import datetime +from pathlib import Path +from time import sleep def build_docs(): @@ -15,54 +16,53 @@ def refresh_assets(): ) -def watch_for_changes(debug=False): - i = InotifyTree("docs") +def watch_for_changes(): + """Watch for changes using watchdog.""" + from watchdog.events import FileSystemEvent, FileSystemEventHandler + from watchdog.observers import Observer - IGNORED_EVENTS = ["IN_CLOSE_WRITE", "IN_CLOSE_NOWRITE", "IN_ACCESS", "IN_OPEN"] - IGNORED_PATHS = ["docs/_build"] + class MyHandler(FileSystemEventHandler): + def __init__(self): + super().__init__() + self.last_build = datetime.now() - last_build = None + def on_modified(self, event: FileSystemEvent): + cwd = os.getcwd() + file_path = Path(event.src_path) - for event in i.event_gen(yield_nones=False): - (_, type_names, path, filename) = event + IGNORED_PATHS = ["docs/_build"] - ignore = False - for ignored_event_type in IGNORED_EVENTS: - if ignored_event_type in type_names: - ignore = True - break + if event.is_directory: + return - for ignored_path in IGNORED_PATHS: - if path.startswith(ignored_path): - ignore = True - break + if file_path.name.isdigit(): + return - if ( - filename.endswith(".swx") - or filename.endswith(".swp") - or filename.endswith("~") - ): - ignore = True + if ( + file_path.name.endswith(".swx") + or file_path.name.endswith(".swp") + or file_path.name.endswith("~") + ): + return - if ignore: - continue + for ignored_path in IGNORED_PATHS: + ignored_path = os.path.join(cwd, ignored_path) + if str(file_path).startswith(ignored_path): + return - if last_build: - # If build has been triggered in the last 50 ms, skip - delta = (datetime.datetime.now() - last_build).microseconds / 1000 - if delta < 50: - continue + build_docs() + refresh_assets() - build_docs() - refresh_assets() - last_build = datetime.datetime.now() + event_handler = MyHandler() + observer = Observer() + observer.schedule(event_handler, path="docs", recursive=True) + observer.start() - if debug: - print( - "PATH=[{}] FILENAME=[{}] EVENT_TYPES={}".format( - path, filename, type_names - ) - ) + try: + while True: + sleep(1) + except KeyboardInterrupt: + observer.stop() if __name__ == "__main__": diff --git a/requirements/development.txt b/requirements/development.txt index 5ee18e77..d47687d4 100644 --- a/requirements/development.txt +++ b/requirements/development.txt @@ -8,4 +8,4 @@ pytest-xdist==2.5.0 mypy==1.0.0 flake8==3.9.2 tox==3.25.0 -inotify==0.2.10 +watchdog From 7d216b6b7faaedac52a98f8df267957d73760854 Mon Sep 17 00:00:00 2001 From: Timon de Groot Date: Tue, 14 Jan 2025 17:47:55 +0100 Subject: [PATCH 06/10] chore: mdformat and black --- .../database/how-to-run-rabbitmq-on-hypernode.md | 12 ++++++------ ...ow-to-configure-remote-storage-for-magento-2-x.md | 1 + 2 files changed, 7 insertions(+), 6 deletions(-) diff --git a/docs/best-practices/database/how-to-run-rabbitmq-on-hypernode.md b/docs/best-practices/database/how-to-run-rabbitmq-on-hypernode.md index 3d5c2890..014cf022 100644 --- a/docs/best-practices/database/how-to-run-rabbitmq-on-hypernode.md +++ b/docs/best-practices/database/how-to-run-rabbitmq-on-hypernode.md @@ -72,12 +72,12 @@ Another way to access the admin interface is via [hypernode-vpn](https://changel You can also create your own RabbitMQ users. This can be done by in the RabbitMQ admin interface, as follows: 1. Go to the RabbitMQ admin interface -2. Click on the `Admin` tab -3. Click on `Add a user` -4. Fill in the username and password -5. Click on `Add user` -6. Click on the user you just created -7. Click on `Set permission` +1. Click on the `Admin` tab +1. Click on `Add a user` +1. Fill in the username and password +1. Click on `Add user` +1. Click on the user you just created +1. Click on `Set permission` ## RabbitMQ and Magento 2 diff --git a/docs/ecommerce-applications/magento-2/how-to-configure-remote-storage-for-magento-2-x.md b/docs/ecommerce-applications/magento-2/how-to-configure-remote-storage-for-magento-2-x.md index ccde0a94..1939e50b 100644 --- a/docs/ecommerce-applications/magento-2/how-to-configure-remote-storage-for-magento-2-x.md +++ b/docs/ecommerce-applications/magento-2/how-to-configure-remote-storage-for-magento-2-x.md @@ -12,6 +12,7 @@ Magento 2.x supports remote storage for media files, import/export files, and ot This feature allows you to store files in a remote storage location, such as an Amazon S3 bucket, instead of storing them on the server itself. This can be useful for many reasons, such as: + - Offloading storage from your server, reducing the load on your server, and improving performance. - Allows you to make use of [horizontal scaling](../../hypernode-platform/autoscaling/how-does-horizontal-autoscaling-work), as you can easily add more servers without having to worry about syncing files between them. - Allows for effortless storage capacity scaling, as you can easily increase the storage capacity of your remote storage location. From 631afe8cadcb79fb01313a7301f54687991c1334 Mon Sep 17 00:00:00 2001 From: Timon de Groot Date: Tue, 14 Jan 2025 17:49:50 +0100 Subject: [PATCH 07/10] docs/autoscaling: Fix incorrect console prompt hostnames --- .../autoscaling/how-does-horizontal-autoscaling-work.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/hypernode-platform/autoscaling/how-does-horizontal-autoscaling-work.md b/docs/hypernode-platform/autoscaling/how-does-horizontal-autoscaling-work.md index 3702aa57..23e966cf 100644 --- a/docs/hypernode-platform/autoscaling/how-does-horizontal-autoscaling-work.md +++ b/docs/hypernode-platform/autoscaling/how-does-horizontal-autoscaling-work.md @@ -77,7 +77,7 @@ To make use of Horizontal autoscaling, Varnish should be enabled and configured You can check if Varnish is enabled on your Hypernode by running ```console -app@abcdef-hnclusterweb1-magweb-tbbm:~$ hypernode-systemctl settings varnish_enabled +app@abcdef-example-magweb-cmbl:~$ hypernode-systemctl settings varnish_enabled varnish_enabled is set to value True ``` @@ -102,7 +102,7 @@ acl purge { The configured MySQL version should be 5.7 or above. You can check the enabled MySQL version by running the following command. ```console -app@abcdef-hnclusterweb1-magweb-tbbm:~$ hypernode-systemctl settings mysql_version +app@abcdef-example-magweb-cmbl:~$ hypernode-systemctl settings mysql_version mysql_version is set to value 8.0 ``` @@ -149,14 +149,14 @@ The persistent instance will be used to store the sessions so we can access the You can check if Redis Persistent is enabled on your Hypernode by running ```console -app@abcdef-hnclusterweb1-magweb-tbbm:~$ hypernode-systemctl settings redis_persistent_instance +app@abcdef-example-magweb-cmbl:~$ hypernode-systemctl settings redis_persistent_instance redis_persistent_instance is set to value True ``` If Redis Persistent instance is not enabled, you can enable the second Redis instance for sessions you run the command: ```console -app@abcdef-hnclusterweb1-magweb-tbbm:~$ hypernode-systemctl settings redis_persistent_instance True +app@abcdef-example-magweb-cmbl:~$ hypernode-systemctl settings redis_persistent_instance True ``` Make sure Redis session is configured as [described](../../ecommerce-applications/magento-2/how-to-configure-redis-for-magento-2.md#configure-magento-2-to-use-redis-as-the-session-store) in our docs From 2bddc8628046a2743fa1f0586274687dcc0ec08c Mon Sep 17 00:00:00 2001 From: Timon de Groot Date: Wed, 15 Jan 2025 10:51:06 +0100 Subject: [PATCH 08/10] bin/watch: Add debounce for batched events --- bin/watch | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/bin/watch b/bin/watch index 0e8f6ef4..dda345b6 100755 --- a/bin/watch +++ b/bin/watch @@ -1,7 +1,7 @@ #!/usr/bin/env python3 import os import subprocess -from datetime import datetime +import time from pathlib import Path from time import sleep @@ -24,7 +24,7 @@ def watch_for_changes(): class MyHandler(FileSystemEventHandler): def __init__(self): super().__init__() - self.last_build = datetime.now() + self.last_build = time.time() def on_modified(self, event: FileSystemEvent): cwd = os.getcwd() @@ -50,8 +50,12 @@ def watch_for_changes(): if str(file_path).startswith(ignored_path): return + if self.last_build and time.time() - self.last_build < 0.5: + return + build_docs() refresh_assets() + self.last_build = time.time() event_handler = MyHandler() observer = Observer() From 68347032e74c5a30546a0d5a351fbb4b2febe27f Mon Sep 17 00:00:00 2001 From: Timon de Groot Date: Wed, 15 Jan 2025 10:51:48 +0100 Subject: [PATCH 09/10] docs/magento2: Add serving assets example --- ...onfigure-remote-storage-for-magento-2-x.md | 46 ++++++++++++++++++- 1 file changed, 44 insertions(+), 2 deletions(-) diff --git a/docs/ecommerce-applications/magento-2/how-to-configure-remote-storage-for-magento-2-x.md b/docs/ecommerce-applications/magento-2/how-to-configure-remote-storage-for-magento-2-x.md index 1939e50b..fc1bfa0e 100644 --- a/docs/ecommerce-applications/magento-2/how-to-configure-remote-storage-for-magento-2-x.md +++ b/docs/ecommerce-applications/magento-2/how-to-configure-remote-storage-for-magento-2-x.md @@ -21,7 +21,6 @@ This can be useful for many reasons, such as: ## Configuring the application Configuring Magento 2 to start storing files in your bucket is done using a single command. -If you're using a different provider than AWS S3, you need to specify the `--remote-storage-endpoint` option. **AWS S3** @@ -36,11 +35,13 @@ bin/magento setup:config:set \ **Other S3 compatible providers** +If you're using a different provider than AWS S3, you need to specify the `--remote-storage-endpoint` option. + ```bash bin/magento setup:config:set \ --remote-storage-driver="aws-s3" \ --remote-storage-bucket="my_bucket_name" \ - --remote-storage-region="my-aws-region" \ + --remote-storage-region="provider-region" \ --remote-storage-key="abcd1234" \ --remote-storage-secret="abcd1234" \ --remote-storage-endpoint="https://my-s3-compatible.endpoint.com" @@ -67,6 +68,47 @@ This is much faster than Magento's built-in sync, because `aws s3 sync` uploads Magento's S3 implementation creates a test file called `storage.flag`, which is basically created to test if the connection works. So this is not a magic file to mark anything ([source](https://github.com/magento/magento2/blob/6f4805f82bb7511f72935daa493d48ebda3d9039/app/code/Magento/AwsS3/Driver/AwsS3.php#L104)). +## Serving assets from your S3 bucket + +To start serving media assets from your S3 bucket, you need to make some adjustments to your nginx configuration. + +```nginx +location /media { + # ... + location ~* \.(ico|jpg|jpeg|png|gif|svg|js|css|swf|eot|ttf|otf|woff|woff2)$ { + resolver 8.8.8.8; + set $bucket ""; + proxy_pass https://s3.amazonaws.com/$bucket$uri; + proxy_pass_request_body off; + proxy_pass_request_headers off; + proxy_intercept_errors on; + proxy_hide_header "x-amz-id-2"; + proxy_hide_header "x-amz-request-id"; + proxy_hide_header "x-amz-storage-class"; + proxy_hide_header "Set-Cookie"; + proxy_ignore_headers "Set-Cookie"; + } + # ... +} +``` + +Also make sure your S3 bucket policies are configured correctly, so that only `/media` is publicly readable. For example: + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "AllowPublicReadOnlyForMedia", + "Effect": "Allow", + "Principal": "*", + "Action": "s3:GetObject", + "Resource": "arn:aws:s3:::/media/*" + } + ] +} +``` + ## Magento remote storage documentation - [Configure Remote Storage](https://experienceleague.adobe.com/en/docs/commerce-operations/configuration-guide/storage/remote-storage/remote-storage) From a03b61e35d73d1b609cea652291c318a4c072c65 Mon Sep 17 00:00:00 2001 From: Timon de Groot Date: Wed, 15 Jan 2025 10:52:15 +0100 Subject: [PATCH 10/10] docs/autoscaling: Add more clarification on requirements --- .../how-does-horizontal-autoscaling-work.md | 24 +++++++++++++++---- 1 file changed, 20 insertions(+), 4 deletions(-) diff --git a/docs/hypernode-platform/autoscaling/how-does-horizontal-autoscaling-work.md b/docs/hypernode-platform/autoscaling/how-does-horizontal-autoscaling-work.md index 23e966cf..e97bf4a5 100644 --- a/docs/hypernode-platform/autoscaling/how-does-horizontal-autoscaling-work.md +++ b/docs/hypernode-platform/autoscaling/how-does-horizontal-autoscaling-work.md @@ -73,7 +73,14 @@ For now, we don't support horizontal autoscaling for development plans. #### Enable and configure Varnish -To make use of Horizontal autoscaling, Varnish should be enabled and configured on the Hypernode. +Varnish is a requirement, because it's essential to deliver cached pages from the load balancer instead of the application servers when scaled up. +The benefits are: + +- Pages in the cache are served faster, because it is one less network hop +- Saves network bandwidth +- Saves CPU usage for both load balancer and application servers +- Page cache is not affected by scaling up or down + You can check if Varnish is enabled on your Hypernode by running ```console @@ -99,7 +106,9 @@ acl purge { #### Make sure to use MySQL 5.7 or higher -The configured MySQL version should be 5.7 or above. You can check the enabled MySQL version by running the following command. +MySQL 5.7 or higher is a requirement, because those versions support auto negotiation of TLS. TLS is an essential part for horizontal autoscaling, because the application servers will connect to the MySQL instance over the internet instead of wireguard. + +You can check the enabled MySQL version by running the following command: ```console app@abcdef-example-magweb-cmbl:~$ hypernode-systemctl settings mysql_version @@ -116,10 +125,14 @@ You should see something similar to `'host' => 'mysqlmaster',`. If this is not t Make sure supervisor is disabled and that there are no supervisor services configured. +For now, we don't support having supervisor services, because we can't guarantee everything works when scaling up. If you do think your use case should be supported, please let us know! + #### Disable Podman services Make sure podman is disabled and that there are no podman services running. +For now, we don't support having podman containers/services, because we can't guarantee everything works when scaling up. If you do think your use case should be supported, please let us know! + #### Configure hostnames correctly The database, cache, session and queue of the application must be configured with correct hostnames instead of `localhost` or `127.0.0.1`. This way the services will be available across multiple hypernodes when the app is autoscaled. @@ -133,7 +146,9 @@ Make sure the env variables (db, cache, session, queue) are not using localhost. Horizontal autoscaling is available for Magento 2.4.7 and higher. To make use of Horizontal autoscaling, there are a couple of other requirements the application should meet. -#### Make sure Redis cache is configured properly +#### Make sure Redis cache is configured + +Having a centralized cache database is essential for a consistent experience for your application. Please make sure Redis cache is configured properly in the Magento 2 configuration file at `/app/etc/env.php`. @@ -143,7 +158,8 @@ More information about [Redis on Hypernode](../../ecommerce-applications/magento #### Enable and configure Redis sessions -Redis persistent is another requirement before you can make use of Horizontal autoscaling. +Redis sessions on the persistent instance is also required to make use of Horizontal autoscaling. + The persistent instance will be used to store the sessions so we can access the same sessions from the Horizontal autoscale Hypernodes. You can check if Redis Persistent is enabled on your Hypernode by running