Splits python representation of metadata and its parser

The current implementation of python representation of metadata and metadata parser was tightly interconnected. Support for other versions of OData was not possible as in each version elements are added, removed or modified. Therefore, we decided to split metadata representation and its parser. With this approach, we can easily define supported elements and its parsing functions in a single class. This "configuration" class is stateless and has to be a child of ODATAVersion.

Additional changes including updating directory structure and refactoring old code to accommodate for incoming ODATA V4 support.

New module model:
- builder -> MetadataBuilder was moved here to make code easier to read
elements -> All EDM elements were moved here, to make python representation of elements version independent. All parsable elements have to inherit from "from_etree_mixin".

- from_etree_callbacks -> All  from_etree static methods were moved into separated function. This is a naive approach as its premise is that all from_etree implementations will be reusable in version V4.

- types_traits -> "types traits" were moved here to make code cleaner and easier to read.

Module V2:
- __init__ -> includes OData2 definition.

- service -> function-wise nothing has been changed.

"Main" module:
- config -> class Config was moved here to make it version and model-independent. In case we will ever need a config class also for service. Also ODataVersion class lives here.

- policies -> All policies were moved here as well as ParserError enum. Again to make policies version and model-independent.

Tests were only updated to incorporate new API.

Author: mamiksik

CHANGELOG.md

@@ -9,6 +9,7 @@ and this project adheres to [Semantic Versioning](http://semver.org/).
 ### Added
 - Client can be created from local metadata - Jakub Filak
 - support all standard EDM schema versions - Jakub Filak
+- Splits python representation of metadata and metadata parsing - Martin Miksik
 
 ### Fixed
 - make sure configured error policies are applied for Annotations referencing

docs/usage/README.md

@@ -1,4 +1,4 @@
-The User Guide
+versionThe User Guide
 --------------
 
 * [Initialization](initialization.rst)

docs/usage/initialization.rst

@@ -121,7 +121,9 @@ For parser to use your custom configuration, it needs to be passed as an argumen
 .. code-block:: python
 
     import pyodata
-    from pyodata.v2.model import PolicyFatal, PolicyWarning, PolicyIgnore, ParserError, Config
+    from pyodata.v2 import ODataV2
+    from pyodata.policies import PolicyFatal, PolicyWarning, PolicyIgnore, ParserError
+    from pyodata.config import Config
     import requests
 
     SERVICE_URL = 'http://services.odata.org/V2/Northwind/Northwind.svc/'
@@ -132,6 +134,7 @@ For parser to use your custom configuration, it needs to be passed as an argumen
     }
 
     custom_config = Config(
+        ODataV2,
         xml_namespaces=namespaces,
         default_error_policy=PolicyFatal(),
         custom_error_policies={

pyodata/client.py

@@ -3,9 +3,11 @@
 import logging
 import warnings
 
-import pyodata.v2.model
-import pyodata.v2.service
+from pyodata.config import Config
+from pyodata.model.builder import MetadataBuilder
 from pyodata.exceptions import PyODataException, HttpError
+from pyodata.v2.service import Service
+from pyodata.v2 import ODataV2
 
 
 def _fetch_metadata(connection, url, logger):
@@ -34,43 +36,37 @@ class Client:
     """OData service client"""
 
     # pylint: disable=too-few-public-methods
-
-    ODATA_VERSION_2 = 2
-
-    def __new__(cls, url, connection, odata_version=ODATA_VERSION_2, namespaces=None,
-                config: pyodata.v2.model.Config = None, metadata: str = None):
+    def __new__(cls, url, connection, namespaces=None,
+                config: Config = None, metadata: str = None):
         """Create instance of the OData Client for given URL"""
 
         logger = logging.getLogger('pyodata.client')
 
-        if odata_version == Client.ODATA_VERSION_2:
-
-            # sanitize url
-            url = url.rstrip('/') + '/'
-
-            if metadata is None:
-                metadata = _fetch_metadata(connection, url, logger)
-            else:
-                logger.info('Using static metadata')
+        # sanitize url
+        url = url.rstrip('/') + '/'
 
-            if config is not None and namespaces is not None:
-                raise PyODataException('You cannot pass namespaces and config at the same time')
+        if metadata is None:
+            metadata = _fetch_metadata(connection, url, logger)
+        else:
+            logger.info('Using static metadata')
 
-            if config is None:
-                config = pyodata.v2.model.Config()
+        if config is not None and namespaces is not None:
+            raise PyODataException('You cannot pass namespaces and config at the same time')
 
-            if namespaces is not None:
-                warnings.warn("Passing namespaces directly is deprecated. Use class Config instead", DeprecationWarning)
-                config.namespaces = namespaces
+        if config is None:
+            logger.info('No OData version has been provided. Client defaulted to OData v2')
+            config = Config(ODataV2)
 
-            # create model instance from received metadata
-            logger.info('Creating OData Schema (version: %d)', odata_version)
-            schema = pyodata.v2.model.MetadataBuilder(metadata, config=config).build()
+        if namespaces is not None:
+            warnings.warn("Passing namespaces directly is deprecated. Use class Config instead", DeprecationWarning)
+            config.namespaces = namespaces
 
-            # create service instance based on model we have
-            logger.info('Creating OData Service (version: %d)', odata_version)
-            service = pyodata.v2.service.Service(url, schema, connection)
+        # create model instance from received metadata
+        logger.info('Creating OData Schema (version: %d)', config.odata_version)
+        schema = MetadataBuilder(metadata, config=config).build()
 
-            return service
+        # create service instance based on model we have
+        logger.info('Creating OData Service (version: %d)', config.odata_version)
+        service = Service(url, schema, connection)
 
-        raise PyODataException('No implementation for selected odata version {}'.format(odata_version))
+        return service

pyodata/config.py

@@ -0,0 +1,117 @@
+""" Contains definition of configuration class for PyOData and for ODATA versions. """
+
+from abc import ABC, abstractmethod
+from typing import Type, List, Dict, Callable
+
+from pyodata.policies import PolicyFatal, ParserError, ErrorPolicy
+
+
+class ODATAVersion(ABC):
+    """ This is base class for different OData releases. In it we define what are supported types, elements and so on.
+        Furthermore, we specify how individual elements are parsed or represented by python objects.
+    """
+
+    def __init__(self):
+        raise RuntimeError('ODATAVersion and its children are intentionally stateless, '
+                           'therefore you can not create instance of them')
+
+    @staticmethod
+    @abstractmethod
+    def supported_primitive_types() -> List[str]:
+        """ Here we define which primitive types are supported and what is their python representation"""
+
+    @staticmethod
+    @abstractmethod
+    def from_etree_callbacks() -> Dict[object, Callable]:
+        """ Here we define which elements are supported and what is their python representation"""
+
+    @classmethod
+    def is_primitive_type_supported(cls, type_name):
+        """ Convenience method which decides whatever given type is supported."""
+        return type_name in cls.supported_primitive_types()
+
+
+class Config:
+    # pylint: disable=too-many-instance-attributes,missing-docstring
+    # All attributes have purpose and are used for configuration
+    # Having docstring for properties is not necessary as we do have type hints
+
+    """ This is configuration class for PyOData. All session dependent settings should be stored here. """
+
+    def __init__(self,
+                 odata_version: Type[ODATAVersion],
+                 custom_error_policies=None,
+                 default_error_policy=None,
+                 xml_namespaces=None
+                 ):
+
+        """
+        :param custom_error_policies: {ParserError: ErrorPolicy} (default None)
+                                      Used to specified individual policies for XML tags. See documentation for more
+                                      details.
+
+        :param default_error_policy: ErrorPolicy (default PolicyFatal)
+                                     If custom policy is not specified for the tag, the default policy will be used.
+
+        :param xml_namespaces: {str: str} (default None)
+        """
+
+        self._custom_error_policy = custom_error_policies
+
+        if default_error_policy is None:
+            default_error_policy = PolicyFatal()
+
+        self._default_error_policy = default_error_policy
+
+        if xml_namespaces is None:
+            xml_namespaces = {}
+
+        self._namespaces = xml_namespaces
+
+        self._odata_version = odata_version
+
+        self._sap_value_helper_directions = None
+        self._sap_annotation_value_list = None
+        self._annotation_namespaces = None
+
+    def err_policy(self, error: ParserError) -> ErrorPolicy:
+        """ Returns error policy for given error. If custom error policy fo error is set, then returns that."""
+        if self._custom_error_policy is None:
+            return self._default_error_policy
+
+        return self._custom_error_policy.get(error, self._default_error_policy)
+
+    def set_default_error_policy(self, policy: ErrorPolicy):
+        """ Sets default error policy as well as resets custom error policies"""
+        self._custom_error_policy = None
+        self._default_error_policy = policy
+
+    def set_custom_error_policy(self, policies: Dict[ParserError, Type[ErrorPolicy]]):
+        """ Sets custom error policy. It should be called only after setting default error policy, otherwise
+            it has no effect. See implementation of "set_default_error_policy" for more details.
+        """
+        self._custom_error_policy = policies
+
+    @property
+    def namespaces(self) -> str:
+        return self._namespaces
+
+    @namespaces.setter
+    def namespaces(self, value: Dict[str, str]):
+        self._namespaces = value
+
+    @property
+    def odata_version(self) -> Type[ODATAVersion]:
+        return self._odata_version
+
+    @property
+    def sap_value_helper_directions(self):
+        return self._sap_value_helper_directions
+
+    @property
+    def sap_annotation_value_list(self):
+        return self._sap_annotation_value_list
+
+    @property
+    def annotation_namespace(self):
+        return self._annotation_namespaces

pyodata/model/__init__.py

@@ -0,0 +1,151 @@
+"""Metadata Builder Implementation"""
+
+import collections
+import io
+from lxml import etree
+
+from pyodata.config import Config
+from pyodata.exceptions import PyODataParserError
+from pyodata.model.elements import ValueHelperParameter, Schema
+import pyodata.v2 as v2
+
+
+ANNOTATION_NAMESPACES = {
+    'edm': 'http://docs.oasis-open.org/odata/ns/edm',
+    'edmx': 'http://docs.oasis-open.org/odata/ns/edmx'
+}
+
+SAP_VALUE_HELPER_DIRECTIONS = {
+    'com.sap.vocabularies.Common.v1.ValueListParameterIn': ValueHelperParameter.Direction.In,
+    'com.sap.vocabularies.Common.v1.ValueListParameterInOut': ValueHelperParameter.Direction.InOut,
+    'com.sap.vocabularies.Common.v1.ValueListParameterOut': ValueHelperParameter.Direction.Out,
+    'com.sap.vocabularies.Common.v1.ValueListParameterDisplayOnly': ValueHelperParameter.Direction.DisplayOnly,
+    'com.sap.vocabularies.Common.v1.ValueListParameterFilterOnly': ValueHelperParameter.Direction.FilterOnly
+}
+
+
+SAP_ANNOTATION_VALUE_LIST = ['com.sap.vocabularies.Common.v1.ValueList']
+
+
+# pylint: disable=protected-access
+class MetadataBuilder:
+    """Metadata builder"""
+
+    EDMX_WHITELIST = [
+        'http://schemas.microsoft.com/ado/2007/06/edmx',
+        'http://docs.oasis-open.org/odata/ns/edmx',
+    ]
+
+    EDM_WHITELIST = [
+        'http://schemas.microsoft.com/ado/2006/04/edm',
+        'http://schemas.microsoft.com/ado/2007/05/edm',
+        'http://schemas.microsoft.com/ado/2008/09/edm',
+        'http://schemas.microsoft.com/ado/2009/11/edm',
+        'http://docs.oasis-open.org/odata/ns/edm'
+    ]
+
+    def __init__(self, xml, config=None):
+        self._xml = xml
+
+        if config is None:
+            config = Config(v2.ODataV2)
+        self._config = config
+
+    # pylint: disable=missing-docstring
+    @property
+    def config(self) -> Config:
+        return self._config
+
+    def build(self):
+        """ Build model from the XML metadata"""
+
+        if isinstance(self._xml, str):
+            mdf = io.StringIO(self._xml)
+        elif isinstance(self._xml, bytes):
+            mdf = io.BytesIO(self._xml)
+        else:
+            raise TypeError('Expected bytes or str type on metadata_xml, got : {0}'.format(type(self._xml)))
+
+        namespaces = self._config.namespaces
+        xml = etree.parse(mdf)
+        edmx = xml.getroot()
+
+        try:
+            dataservices = next((child for child in edmx if etree.QName(child.tag).localname == 'DataServices'))
+        except StopIteration:
+            raise PyODataParserError('Metadata document is missing the element DataServices')
+
+        try:
+            schema = next((child for child in dataservices if etree.QName(child.tag).localname == 'Schema'))
+        except StopIteration:
+            raise PyODataParserError('Metadata document is missing the element Schema')
+
+        if 'edmx' not in self._config.namespaces:
+            namespace = etree.QName(edmx.tag).namespace
+
+            if namespace not in self.EDMX_WHITELIST:
+                raise PyODataParserError(f'Unsupported Edmx namespace - {namespace}')
+
+            namespaces['edmx'] = namespace
+
+        if 'edm' not in self._config.namespaces:
+            namespace = etree.QName(schema.tag).namespace
+
+            if namespace not in self.EDM_WHITELIST:
+                raise PyODataParserError(f'Unsupported Schema namespace - {namespace}')
+
+            namespaces['edm'] = namespace
+
+        self._config.namespaces = namespaces
+
+        self._config._sap_value_helper_directions = SAP_VALUE_HELPER_DIRECTIONS
+        self._config._sap_annotation_value_list = SAP_ANNOTATION_VALUE_LIST
+        self._config._annotation_namespaces = ANNOTATION_NAMESPACES
+
+        self.update_alias(self.get_aliases(xml, self._config), self._config)
+
+        edm_schemas = xml.xpath('/edmx:Edmx/edmx:DataServices/edm:Schema', namespaces=self._config.namespaces)
+        return Schema.from_etree(edm_schemas, self._config)
+
+    @staticmethod
+    def get_aliases(edmx, config: Config):
+        """Get all aliases"""
+
+        aliases = collections.defaultdict(set)
+        edm_root = edmx.xpath('/edmx:Edmx', namespaces=config.namespaces)
+        if edm_root:
+            edm_ref_includes = edm_root[0].xpath('edmx:Reference/edmx:Include', namespaces=config.annotation_namespace)
+            for ref_incl in edm_ref_includes:
+                namespace = ref_incl.get('Namespace')
+                alias = ref_incl.get('Alias')
+                if namespace is not None and alias is not None:
+                    aliases[namespace].add(alias)
+
+        return aliases
+
+    @staticmethod
+    def update_alias(aliases, config: Config):
+        """Update config with aliases"""
+
+        namespace, suffix = config.sap_annotation_value_list[0].rsplit('.', 1)
+        config._sap_annotation_value_list.extend([alias + '.' + suffix for alias in aliases[namespace]])
+
+        helper_direction_keys = list(config.sap_value_helper_directions.keys())
+        for direction_key in helper_direction_keys:
+            namespace, suffix = direction_key.rsplit('.', 1)
+            for alias in aliases[namespace]:
+                config._sap_value_helper_directions[alias + '.' + suffix] = \
+                    config.sap_value_helper_directions[direction_key]
+
+
+def schema_from_xml(metadata_xml, namespaces=None):
+    """Parses XML data and returns Schema representing OData Metadata"""
+
+    meta = MetadataBuilder(
+        metadata_xml,
+        config=Config(
+            v2.ODataV2,
+            xml_namespaces=namespaces,
+        ))
+
+    return meta.build()