Updates from PyHSS #23
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The config is already getting parsed a few lines above, don't do it twice.
The config is not used here, so remove code for loading it. Remove commented out alternative logging initializing code while at it too.
Rename config to alembic_config, so we can use config as the pyhss config (as it is used in most other files).
Move the config file loading to a new lib/pyhss_config.py file, instead
of having it duplicated in all files that load the config. This allows
replacing code like:
import yaml
try:
with open("../config.yaml", 'r') as stream:
config = (yaml.safe_load(stream))
except:
with open("config.yaml", 'r') as stream:
config = (yaml.safe_load(stream))
with the following in all files:
from pyhss_config import config
Replace code that loaded the config to self.config in various classes
with just using the global config variable. Adjust the few scripts that
had named it yaml_config instead of config to just config, so it is
consistent.
Let the new code not only load the config from the top dir of the git
repository, but also from a PYHSS_CONFIG env var, /etc/pyhss/config.yaml
and /usr/share/pyhss/config.yaml. This is useful for being able to still
load configs after installing PyHSS into a different location, for
example with pip or apt. It also allows running tests against PyHSS with
a config file in a separate directory (e.g. in osmo-ttcn3-hacks).
This script doesn't import anything from lib, so this is not needed.
To run services outside of the source tree, the lib path must be added correctly to sys.path. Fix this by determining it relative to the current script's path instead of assuming that it is in ../lib (relative to the current work dir).
Add main() functions, so pyproject.toml can create launcher scripts for the services.
Add a pyproject.toml file, so it becomes possible to build python whl packages with: $ python3 -m build With this change, Linux distributions could package PyHSS similar to other python projects, it should be possible to install it via pip (from the git repo), and the whl can be used to directly install PyHSS into a venv.
Found by running the Osmocom TTCN-3 tests against pyHSS.
…led-requests GSUP: fix responding to unhandled requests
PyHSS follows a microservice model and spreads it's usage over different services. In a containerized world, it is considered good practice to have one container per service instead of having a giant container that does everthing. While a monolithic container that does it all is convenient for an enduser, a similar thing can be achived by leveraging the `compose` feature of Docker and similar container runtimes. Environment variables are substituted with a go envsubst variant. Using that seemed like the easiest thing to do since pyHss itself does not support env vars for config. Everything can be configured via environment, but has defaults.
This prevents the docker daemon from invalidating build caches when files change that have nothing to do with it. It also prevents them from being copied into the image and bloat it.
Add a simple explanation on how to get started with docker (compose) when running the project.
Github workflows must live under .github/workflows to be effective. This seems to be a stale file that should no longer be here.
For the new docker image it is important to set the correct build context, which must be the project root, not the docker folder. The old way only worked because the image effectively did a git checkout during build. The new image does not do that anymore.
This changes the `.dockerignore` file so that the directories themselves are ignored. They were previously added to the image but were empty. While it is not critical, this was not intentional. The `docker` directory cannot be ignored completly because it contains files needed during certain copy operations. That's why I delete it afterwards from the file system. It is not ideal since the files will be in the docker image layers. COPY --exclude=docker would be the solution but that is not supported on the CI runners. Also I've added the `.idea` directory, which is used by JetBrains IDEs like PyCharm, which is what I use so I want it ignored.
The application still wants to create logfiles so we need to make sure it is able to do so. The previous copy operation will create a log directory but it will belong to root. I've also marked it as a VOLUME because it contains variable data and added it to the compose file. To do this, I had to move the USER directive to a later stage. To me, putting it at the end seems to make the most sense.
Remove the Destination Name IE from the following messages, as it does not belong there. * Insert Subscriber Data Request: https://gitea.osmocom.org/cellular-infrastructure/osmo-gsm-manuals/src/commit/d24f7186f6dff58940ee57b97f9d278cc07f245e/common/chapters/gsup.adoc#L534 * Update Location Result / Error: https://gitea.osmocom.org/cellular-infrastructure/osmo-gsm-manuals/src/commit/d24f7186f6dff58940ee57b97f9d278cc07f245e/common/chapters/gsup.adoc#L442 https://gitea.osmocom.org/cellular-infrastructure/osmo-gsm-manuals/src/commit/d24f7186f6dff58940ee57b97f9d278cc07f245e/common/chapters/gsup.adoc#L429 * Location Cancellation Request: https://gitea.osmocom.org/cellular-infrastructure/osmo-gsm-manuals/src/commit/d24f7186f6dff58940ee57b97f9d278cc07f245e/common/chapters/gsup.adoc#L460
…name-ie GSUP: Update Location Request: no destination name
…pose Improve Docker image + compose
…onfig-loading Allow running outside of source tree
It is not common practice to add an extra empty line before the [Service] section of .service files. Remove it to have consistently one empty line between each section.
Move the API service file next to the other service files, in preparation to install it from there in debian packaging.
The API service script had been renamed from PyHSS_API.py to services/apiService.py some time ago. Adjust the command in the documentation.
This change makes it so that one can more easily configure docker Python interpreters in IDEs like PyCharm. * Rationale for moving the venv directory: You'll most likely want to mount your source root to /opt/pyhss as a volume to use the container runtime while also having up-to-date sources without rebuilding. But that shadows the venv unless it is outside the source tree. * Rationale for exec $@: IDEs will try to inject their own launch CMDs to facilitate things like remote debugging the container. By removing the CONTAINER_ROLE requirement, and defaulting to `exec $@`, that can be achieved.
Ensure we keep this consistent in the future.
Fix that the ChargingRule_Tests can fail on the last_modified timestamp as seen here: https://github.com/nickvsnetworking/pyhss/actions/runs/19851164742/job/56878098409 FAILED tests/test_API.py::ChargingRule_Tests::test_D_Patch_TFT - AssertionError: {'dir[86 chars]5', 'tft_group_id': 1, 'last_modified': '2025-12-02T07:51:13Z'} != {'dir[86 chars]5', 'tft_group_id': 1, 'last_modified': '2025-12-02T07:51:14Z'} {'direction': 2, - 'last_modified': '2025-12-02T07:51:13Z', ? ^ + 'last_modified': '2025-12-02T07:51:14Z', ? ^ 'tft_group_id': 1, 'tft_id': 2, 'tft_string': 'permit out ip from 10.98.0.20 80 to any 1-65535'} : JSON body should match input
…rging-rule-last-modified tests/test_API: ChargingRule: remove last_modified
If the RAT type IE is missing, check both 2G and 3G and block if either one is forbidden. See: nickvsnetworking#283 (comment)
…ctions Implement RAT Restrictions for 2G/3G and 4G
Clean up the database section of the config by removing these two options. They are relevant for the database service, which seems to be a work-in-progress (nickvsnetworking#264). Furthermore the options themselves seem not very useful currently: * readCacheEnabled: services/databaseService.py checks if this option is set to True or False in the config (default: True). When this option is set to False, the script exits in its main loop with a log message. Instead of relying on that mechanism, users can simply not start the databaseService.py script if they don't want to use it. That makes more sense than starting it just to have it exit immediately. The related systemd service is also not mentioned in pyhss.service as Wants= and could be disabled or masked if the user doesn't want to start the service instead of having it exit immediately on startup. This also prevents a problem that right now the service has Restart=always configured, which means that if it is disabled through the config, it would restart a lot of times until systemd decides that it restarts too often in short time and stops it for good. * readCacheInterval: this option is unused in the code. The services/databaseService.py script reads a similarly named cacheReadInterval option from the config instead (default 60). But given that services/databaseService.py is still WIP, I think it makes more sense to just remove the wrong option from the config than correcting it.
The SQLite database is a more suitable default, as it works fine for small scale and development scenarios while it does not need additional installation and configuration of database software. For big deployments, users will need to adjust their config anyway. With new commented out blocks for mysql and postgresql, switching to those is trivial. Existing users are not affected by this change, as they practically need to have their own version of the config with their own database information already.
These options are unused when using the sqlite db_type. Remove them to make the file easier to understand.
Removing these from the test config was forgotten in 7a2fac9 ("config: remove unused redis.sentinel").
According to class Database() in lib/database.py, "only" MySQL,
PostgreSQL and SQLite are supported:
if db_type == 'postgresql':
db_string = 'postgresql+psycopg2://' + …
elif db_type == 'mysql':
db_string = 'mysql://' + …
elif db_type == 'sqlite':
db_string = "sqlite:///" + …
else:
raise RuntimeError(f'Invalid database.db_type set "{db_type}"')
Describe that the default database type is now sqlite (from this PR) and other recent changes.
Add hss.db to gitignore, as this is now the default sqlite database name in config.yaml.
config: database: default to sqlite
…headers Use consistent SPDX license headers
README: link to matrix room
Use the new format to be consistent with the other files and to have the test passing.
…test-fix lib/rat: update license header
Use sqlite by default in docker/config.yaml and docker/.env to follow the change in the main config.yaml. This also speeds up the setup for docker-compose as no mysql container needs to be downloaded.
Without this patch the time before services get "healthy" is 10s: the healthcheck directly runs once and fails, and then waits the interval (10s) before checking again. Fix this by setting a start period of 3s in which we check with a 0.1s interval. Before: docker compose up -d 0,28s user 0,18s system 1% cpu 32,484 total After: docker compose up -d 0,09s user 0,07s system 4% cpu 4,028 total
…er-sqlite docker: use sqlite
Add the hashbang line to each service script and make them executable, so these can be executed directly from the terminal.
…cutables services: add #!/usr/bin/env python3
Add SQN resync to REST api
fix typo on log init message
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.