All notable changes to this project will be documented in this file.
The format is based on Keep a Changelog and this project adheres to Semantic Versioning.
- Python 3.9 support. Minimum required Python version is now 3.10.
- Changed some dependencies:
| Dependency | From | To |
|---|---|---|
| pyopenssl | ~=25.3.0 | ~=26.0.0 |
| pytz | ~=2025.2 | ~=2026.1 |
| pebble | ~=5.1.3 | ~=5.2.0 |
| pytest | ~=8.4.2 | ~=9.0.0 |
| pytest-cov | ~=5.0.0 | ~=6.0.0 |
| responses | ~=0.25.8 | ~=0.26.0 |
- Changed some dependencies:
| Dependency | From | To |
|---|---|---|
| click | ==8.3.1 | ~=8.3.1 |
| pem | ~=21.2.0 | ~=23.1.0 |
| cryptography | ~=44.0.3 | Removed (transitive via pyOpenSSL) |
| certifi | ~=2026.1.4 | Removed (transitive via requests) |
- API Lookup developer guide reference in legacy lookup documentation
- API Lookup developer guide in documentation
- Deprecation message in lookup upload functionality modified
- Deprecation message in lookup upload functionality modified
- Right now the upload of lookups is based on the
my.lookup.dataandmy.lookup.controltables. This method will soon be deprecated on the Devo backend. As an alternative, you can use the Lookups API: Lookups API Documentation. The Python SDK will support the Lookups API in future versions.
- Added new --escapenewline or -enl option in lookup creation, edition and deletion. It will escape with double backslash any new line char found inside a csv field.
- Added Python 3.13 support
- Change logger name in
common/sender/data.pyso it logs todevo.datainstead to ROOT
- Bump security libraries to updated versions
test_api_clicorrected as error code changed nomenclature
- Supported Python versions extended to 10, 11 and 12
- Added time zones in date operations
- Jobs API reviewed and fixed. Jobs searching by type and friendlyName discontinued as it is not supported by API. Jobs API unit test checked and enabled
- Added timeout to unit tests of API queries. They may run forever when faulty
- Keep-alive mechanism not working for queries with
destination. Forcing NO_KEEP_ALIVE in queries withdestination. - SSL wrapping of the TCP connection when no certificates are used improved
- Fix auxiliary Echo serving for unit testing in order to run with new async paradigm
- Documentation fixes. Some parameters missing or non-existent in docstring
- Fix for a unit test when using concurrency (from library
stopittopebble)
- Python 3.8 support discontinued
- Python 3.8 not supported anymore
- Jobs searching by type and friendlyName discontinued in Jobs API. Only search by job id is supported.
- Date requires time zone
- Query with
destinationare forced to NO_KEEP_ALIVE mode for Keep-alive mechanism (instead of DEFAULT_KEEPALIVE_TOKEN)
cryptographydependency upgraded fromcryptography~=42.0.5tocryptography~=43.0.1
responsesdependency upgraded fromresponses~=0.25.0toresponses~=0.25.3certifidependency upgraded fromcertifi~=2024.6.2tocertifi~=2024.7.4
- Force flush in Sender buffer to send incomplete buffers after a timeout
mockdependency added to testing requirements
requestsdependency upgraded fromrequests~=2.31torequests~=2.32certifidependency upgraded fromcertifi~=2024.2.2tocertifi~=2024.6.2cryptographydependency upgraded fromcryptography~=42.0.5tocryptography~=42.0.8pipdeptreedependency upgraded frompipdeptree~=2.17.0topipdeptree~=2.23.0pytestdependency upgraded frompytest~=8.1.1topytest~=8.2.2
- Add ip-as-string flag to the API client to return the IP address as a string instead of an integer.
cryptographydependency upgraded fromcryptography~=42.0.2tocryptography~=42.0.5msgpackdependency upgraded frommsgpack~=1.0.7tomsgpack~=1.0.8pipdeptreedependency upgraded frompipdeptree~=2.14.0topipdeptree~=2.17.0pyopenssldependency upgraded frompyopenssl~=24.0topyopenssl~=24.1.0pytest-covdependency upgraded frompytest-cov~=4.1.0topytest-cov~=5.0.0pytestdependency upgraded frompytest~=7.4.4topytest~=8.1.1responsesdependency upgraded fromresponses~=0.24toresponses~=0.25.0
- Change tests to use
pytestinstead of the custom script that usesunittest. certifidependency upgraded fromcertifi~=2023.11.17tocertifi~=2024.2.2cryptographydependency upgraded fromcryptography~=41.0.7tocryptography~=42.0.2pyopenssldependency upgraded frompyopenssl~=23.3topyopenssl~=24.0pytzdependency upgraded frompytz~=2023.3topytz~=2024.1pipdeptreedependency upgraded frompipdeptree~=2.13.1topipdeptree~=2.14.0
- Tests for encoding of messages in Sender.
- UFT-8 encoding with "replace" is now used by default in Sender.
- Added Code of Conduct.
- Restrict insecure TLS versions in Sender and API clients to TLSv1.2 and TLSv1.3.
- Remove disclosing information in messages about keepalive token.
- Fix incorences in documentation about using
urlandaddressinterchangeably. certifidependency upgraded fromcertifi~=2023.7.22tocertifi~=2023.11.17cryptographydependency upgraded fromcryptography~=41.0.5tocryptography~=41.0.7pipdeptreedependency upgraded frompipdeptree~=2.13.0topipdeptree~=2.13.1pyopenssldependency upgraded frompyopenssl~=23.2topyopenssl~=23.3responsesdependency upgraded fromresponses~=0.22.3toresponses~=0.24
urllib3dependency removed,requestsalready solves it.cryptographydepemdency upgraded fromcryptography~=41.0.3tocryptography~=41.0.5msgpack~=1.0.4testing dependency upgraded frommsgpack~=1.0.4tomsgpack~=1.0.7pipdeptree~=2.5.0testing dependency upgraded frompipdeptree~=2.5.0topipdeptree~=2.13.0- Removed support for Python 3.7.
pyopenssldependency upgraded frompyopenssl~=23.0topyopenssl~=23.2pytzdependency upgraded frompytz~=2019.3topytz~=2023.3responsesdependency upgraded fromresponses~=0.22.0toresponses~=0.23.3urllib3dependency upgraded fromurllib3~=1.26.5tourllib3~=2.0.6- Log level demoted to warning while closing socket in sender data.
clickdependency upgraded fromclick==8.1.3toclick==8.1.7pemdependency open frompem==21.2.0topem~=21.2.0pyyamldependency open frompyyaml==6.0.1topyyaml~=6.0.1- Code reformatted with
yapf,black,isortandflake8tools.
certifidependency open fromcertifi==2021.10.8tocertifi~=2023.7.22cryptographydependency upgraded fromcryptography==41.0.1tocryptography~=41.0.3
requestsdependency updated fromrequests@2.27torequests@2.31cryptographydependency updated fromcryptography@39.0.1tocryptography@41.0.1
pyopenssldependency open frompyopenssl==23.0.*topyopenssl>=23.0- Markdown documents reformated.
- Fix minor documentation error in
docs/sender/data.md
- Sender: NonBlocking Socket support added
responsesdependency open from ==0.22.0 to >=0.22.0
- Sender: Closing handshake protocol improved to avoid losing of events. Upstream channel is closed and then client waits for the closing of downstream channel
- Sender: Monitoring of the EOF signal in downstream channel to detect whether server endpoint have closed the session
- Fixed message file management in CLI
- Fixed wrong import
msgpackdependency open from ==1.0.4 to >=1.0.4
- Fixed exception handling issue in
devo/sender/data.py - Fixed message shown when configuration file (JSON or YAML) is not correct
- Client side exception management refactoring for sending and querying data
- Snyk integration for checking dependencies and static code security
pipdeptreedependency open from ==2.5.0 to >=2.5.0
- Update version range support for
requests
- Documentation fixes in
README.mdanddocs/common.md - Documentation related to testing updated at
README.md
- Added
verify-certificatesparameter support in event sending and lookups management clients
- Fix vulnerability found in dependency
cryptography@38.0.4inherited frompyopenssl@22.1.0
- Github workflow
Package testcan be triggered manually now - Dependabot configuration added for security and versions updates
- Fix documentation (implementation remains the same) for keep-alive mechanism in queries for
xlsformat - Bump version to avoid collision with tag belonging to former release candidate version
- Ingestion endpoint has an inactivity timeout that when reached closes the connection.
devo-sdkis aware of such a timeout and restart connection before is reached. New parameterinactivity_timeoutin classSenderto set up it. Its default value is 30 seconds. - Syntax error when calling
Path.is_file() - Documentation related to parameter
keyremoval atdevo.sender.lookup.Lookup.send_data_linein version 5.0.0
pyopenssldependency bumped
- Error messages added to enum
- Error when processing some exceptions
DevoClientExceptionandDevoSenderExceptionrefactored for better details extraction- In query error detection and feedback through detailed
DevoClientException - New test selection in
run_tests.pytool (included and excluded parameter)
- Parameter
keyremoved fromdevo.sender.lookup.Lookup.send_data_line. Thekeyparameter pointed to the value in thefieldslist that was the key of the lookup item. When the value appeared several times infields, the one expected to be the key cannot be identified. This parameter was set as deprecated since version 3.4.0 (2020-08-06)
- API query timeout by default set to 300 seconds, instead of 30
- Retries timeout following Exponential backoff algorithm. Default retry delay base set to 5 seconds
- Retry delay configurable through
retry_delayinstead oftimeoutparameter
- Added a constraint in client parameters so binary formats like,
msgpackandxls, cannot be invoked withoutoutputparameter
- Add a new command-line option for escaping double quotes with
-eqor--escape_quotes - Add a check and shows a warning if the file contains double quotes and the option
-eqis not used - Add a new command-line option in tests for testing just one module with
-m <module_name>or--module <module_name>
- Create GitHub action to publish package in PyPI
- Create GitHub action for running tests after a pull request
- Added CSV tests
- Default retries now are 0. Therefore, no retry mechanism is enabled by default in query API
- Retries parameter is set to default 0 as there are no retries by default. If, after error, one retry is needed, the parameter should be set to 1
msgpackandxlsresponse mode know return bytes type in query API
- KeepAlive functionality provided by server now is supported by query API
- Stream modes only supported for
csv,tsv,json/simple,json/simple/compactmodes in query API
- Fixed bug when processing keep alive empty tokens
- Dependencies updated for
clickandPyYAML - Some small fixes in documentation
urllib3dependencies forced for CVE-2021-33503- Query in REST API test to be configurable by environment variable
- Some events are not sent completely due to the use of
socket.send()instead ofsocket.sendall()[#141]
- Dependencies added to
setup.py - Some documentation fixes, mostly related to dependencies and compatibilities with Python versions
- Sender: certificate files can now be verified with
verify_config=Trueor"verify_config": truefrom the config file. - Internal support for HTTP unsecure API REST endpoint.
- Sender: bad error management when
socker.shutdownis called and the connection was not established. - test
test_get_common_namesnot running. - Some environment vars for testing were wrong in the sample file.
pemmodule added to depedencies
- Double quotes on lookups can be escaped by adding
"escape_quotes": trueto the config file.
- Fix and skip some non-working tests.
Lookup.send_headersmethod not working as expected with key_index parameter.- Add
type_of_keyparameter toLookup.send_headersmethod. - Avoid lookups' key element to be deleted when
Lookup.send_data_lineis invoked. - api-query: return
Nonewhen the query response has no results. - Remove
testsfolder from distribution package.
- Update pyyaml version dependency
- Info about use custom CA for verify certificates in client
- Client problems with default "From" key for queries
- Socket closes are more gently now, fixed problems with loss events
- Updated message when overwrite sec_level to show only when create Sender
- Updated test for bad credentials. Now api returns error in signature validation
- Support to use in lookup fields lists: ints, booleans and floats. Not necessary send all with str type.
- More documentation in lookup readme
- Problem in list_to_headers when pass key but not key_index
- Count of sended events when zip=True
- Problems with key instead of key_index in lookups
- Fixed problem with row types in Lookup creation
- Fixed (Again) problems with verify flag in Api creation
- Fixed problems with Query Eternal queries
- Fixed problems with some flags in Api creation
- env vars JWT and TOKEN for API shell client
- verify certs in API creation and cli calls
- Documentation about SSLConfigSender
- Unnecesary setters
- Added examples
- Fixed shebangs
- Fixed typo in README
- Remove one blank in Sender
- Support for Python 3.8
- Testing in Travis for python 3.6, 3.7 and 3.8
- Support in API to timeZone flag for Devo API
- Support in API for new devo custom formats
- Documentation for new date formats
- Functions to change buffer size and compression_level of Sender
- Support for zip, buffer and compression_level flags in Sender CLI
- SSL Server support to adapt it from python 3.5 to python 3.8
- SSL Send data tests
- Requirements, updated.
- unnecessary elifs in code following PEP8 recomendations
- Added new security flags to Sender SSL Sender
- Changed API processors to be able to add custom processors
- Documentation
- Sender CLI missing options
- wrongly assigned timeout from configuration
- "timeout" and "retries" parameters are able to be assigned from environment (DEVO_API_TIMEOUT, DEVO_API_RETRIES)
- Changed version info in CLI for show only when asked
- Support to Incremental lookups in CLI
- Support to types in lookups
- Auto-detect types for lookups
- Action field for lookups
- The documentation of the devo-sender is now separated into data and lookups
- list_to_fields function in lookups its deprecated and not in use by the internal code. To be deleted in v4 of Devo-sdk
- Support to CA_MD_TOO_WEAK problems with a new flag "sec_level"
- New flag documentation
- SSL Socket in Sender creation with custom context
- Tests Sender certificates generation/files are updated
- Problems in documentation with API information
- Documentation with examples of version 2.0
- level flag to handlers in logging and updated documentation
- flag/capacity of Sender to use pfx certificates
- level for global logging when get_log. We maintain general level until version 4
- Fixed reconnection to socket in every send instruction of Sender
- Code style changes
- Added pep8speaks file for Github
- API dates when int, verify if len its correct before add 3 zero
- Problem overriding tag when use Sender for logging
- Fixed always debug on reconnection
- Support to Python 2.7
- Configuration class "key_exist()" and "keys()" functions
- Api and Sender "from_config" are now removed and included in Class init flag
- for_logging option in Sender, added more values
- Logging streamhandler
- Tests for CLI
- Have Client query method accept Unix timestamps for to and from dates
- Make Client parameters retries, timeout, and sleep configurable
- CLI of devo-sdk now has --version flag.
- Errors exceptions in API and Sender are now full controlled and homogenized
- logging in Common now are created more flexible
- Sender init class, SSLSender and TCPSender too.
- API init class
- New documentation
- Now devo Configuration object inherits from the dict class
- Configuration set and get are now more dict homogenized
- vars/flags "url" are now like python: "address" and are now tuples
- New "auth" flag to encapsulate key, secret, jwt and token
- Env var "DEVO_AUTH_TOKEN" to "DEVO_API_TOKEN"
- Env var "DEVO_API_URL" to "DEVO_API_ADDRESS"
- Fixed when add key chain to configuration
- Fixed problems with API CLI Config objects
- Fixed problems with SENDER CLI Config objects
- API Proccessors
- Problems with multiline and zip when both are activated
- Fixed multiline sending introduced in version 1.3.0
- Fixed bug in Sender CLI introduced in version 2.2.2
- Sender for logging (Handler) get Severity correctly when use custom Level info
- Lookup shell client now read config correctly
- Lookup shell client now apply default port if empty
- API response in stream mode now return response object if response.code its not 200
- Inits args for Sender(), and from_config(), to make it more flexible to config objects and flags
- Requirements file
- .pyup.yaml file for security dependencies bot
- Logging when send data in multiline or zipped
- Error of for_logging() function in Sender when applicate level
- More documentation info and examples
- All new endpoints references for sender and api
- Problems with proc_json_compact_simple_to_jobj when None
- Problems with recursion depth when zipped send has problems with Socket.
- Devo.common get_log() has more flags for better customization
- Sender for_logging() creation now verify where is tag flag
- Functions for configuration (save, len, etc)
- Added verify flag for API, for not verify TLS certs
- devo.common logging handler
- Errors response in devo.api
- Changed all logic in API calls for stream and non stream calls, improving speed in streaming x3
- API responses are now iterator
- Rollback to request sockets instead of custom sockets in API
- Rollback to one class in Client, instead of Base and Client
- Behavior of the processors, optimizing their performance and only allowing default processors/flags, within the API
- Buffer files
- chain_dict files
- Buffer item
- Old custom processors functions in API calls
- Broken urls in documentations
- Inconsistences in api.base class
- PEP8/Pylint styles in api and common packages
- Documentation about API "destination" parameter
- Task management with API Client: added functions, documentation
- Travis CI configuration
- Refactoring of API, separated now in two files
- Reorganized API documentation
- Refactoring of devo.common.dates from objects to simple functions
- Now API return bytes in Python 3 and str in Python 2 by default
- Problems with non ascii/utf-8 characters in API
- Travis CI autodeploy to pypi
- Travis CI upload wheel to release on github
- Mild refactoring of Sender class
- Refactoring API Response processing
- Typos in docs
- API Socket recv size
- API responses blank lines and splitted lines
- Problems with API CLI and automatic shutdowns
- devo.common buffer now receive data when CSV response
- devo.common buffer.is_empty
- Travis CI deploy automatic options
- Modify Configuration loads functions
- Now Sender send, send_raw, flush_buffer and fill_buffer return the number of lines sent
- Problems when loads configuration from default sites
- Problems with local_server
- Allow the parameters user, application name and comment to queries
- Add tests (test_pragmas and test_pragmas test_pragmas_not_comment_free) in the query.py file.
- Problems when API CAll query has not "to", now added be default "now()" if not to in no stream mode
- Somes revert changes in pull request error are back again
- Use of Token auth and JWT in API, added docs and tests
- Local servers for tests
- YAML file for Travis CI tests
- ConnectionError in python 2
- tests of Sender
- Close order of API/Buffer
- Modify Client init class
- License file
- Dockerfile
- YAML configuration files, like JSON files.
- YAML configuration file information in all docs.
- EXTRA_REQUIRES option for YAML support
- Default Yaml file load in Common
- Removed the octec count for performance, and verify if "\n" at the end of each msg, if applicable.
- Modify classes to privates
- Public functions to private in API
- Update documentation of Sender and API
- Unused class "Props" on Sender, included in past versions
- Bad class debug for Sender, unused too
- Memoize class from Common namespase
- Memoize references in Common readme.
- Int conversion for SenderSSL and SenderTCP classes in port value
- is_empty, set_timeout and close functions for Buffer class
- Modified how API flow work
- Modified Buffer class to avoid failures by timeouts and emptys that block the threads
- Updated documentation
- Modified Sender init class for kwargs arguments
- Sender class inherits from logging.handlers
- Long description value in setup for pypi
- Fixed problems when zip msgs dont have "\n" in each line msg
- Added flush buffer in send data when socket is closed
- Modified documentation for new cert_reqs flag changed
- Tests and requirements for Sender
- Problems with double "\n" in messages
- Sender CLI problems when no certs required
- More error prevention in API
- More debug information when exceptions occur in API
- Strings joins for test files in Sender and Lookup tests
- Name of internal vars in sender tests
- API docs examples
- API urls in docs, examples and code
- Documentation of .devo.json file
- Contributors lists
- test function for setup.py
- API response when code are not 404 or 200 give more info now
- Fixed API EU default url in code and documentation
- Dockerfile now rm some folders for to avoid problems when there are several builds
- More Sender examples in docs/sender.md
- More info about config.json file in sender, api and common doc
- config.example.json in docs/common for give more info about config files for easy use
- Sender.from_config con_type default is None, and change how to manage default var into the function
- Changed how to manage certs_req value into function Sender.from_config
- Sender.from_config now use the correct param "certs_req", not wrong/old "certreq"
- Docker file for testing purposes
- Example vars in environment.env example file
- Updated option of not launch TCP tests for no-devo developers
- .gitignore file
- Zip flag in Devo Sender
- Problems with QueryId and normal query in payload
- On the ninth month of the year, a Devo programmer finally decided to publish a part of the SDK and rest in peace.