Merge branch 'streaming_expiration'

Wasin Waeosri · Wasin Waeosri · commit 33896d033246 · 2019-01-14T17:44:12.000+07:00
1. Add trkd_wsstreaming.py for TRKD Streaming example
2. Add LICENSE.md file
diff --git a/.gitignore b/.gitignore
@@ -1,3 +1,7 @@
 chart.png
 images/
-docs/TRKD_REST_with_Python_master.doc
+docs/TRKD_REST_with_Python_master.doc
+note.txt
+test.py
+market_price_edpgw_authentication.py
+.vscode/
diff --git a/LICENSE.md b/LICENSE.md
@@ -0,0 +1,13 @@
+Copyright 2018 Refinitiv
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
diff --git a/README.md b/README.md
@@ -4,7 +4,7 @@ The [Thomson Reuters Knowledge Direct (TRKD) API](https://developers.thomsonreut
 
 TRKD offers a wide range of Thomson Reuters' information and services delivered in a request-response scenario via web services using today's industry standard protocols (SOAP/XML and REST/JSON). Connectivity can be via HTTP and HTTPS, over the Internet or Delivery Direct. All data are snapshot (non-streaming) data.
 
-This is an example project that shows how to implement TRKD HTTP JSON Client with python. This project contains the following example scripts for each TRKD services
+This is an example project that shows how to implement TRKD HTTP JSON client and TRKD Streaming client with Python programming lanugage. This project contains the following example scripts for each TRKD services
 - trkd_authen.py: An example application that shows how to authenticate with TRKD service
 - trkd_quote.py: An example application that shows how to subscribe (all fields and specific fields) the Quote data from TRKD service
 - trkd_newsheadline.py: An example application that shows how to subscribe the News Headline data from TRKD service
@@ -13,13 +13,18 @@ This is an example project that shows how to implement TRKD HTTP JSON Client wit
 - trkd_interday.py: An example application that shows how to subscribe the Interday Time-series data from TRKD service
 - trkd_onlinereport.py: An example application that shows how to subscribe the Online Report data from TRKD service
 - trkd_chart.py: An example application that shows how to subscribe and download the Chart image data from TRKD service
+- trkd_wsstreaming.py: An example application that show how to subscribe the Quote data from TRKD Streming service via a WebSocket connection
+- requestments.txt: A requirement file contains a list of required libraries for HTTP JSON and WebSocket connections. 
 - docs\TRKD_REST_with_Python.docx: A document that describes the trkd_authen.py and trkd_quote.py applications
 
+All source code and scripts are provided under the Apache 2.0 license. Thye are provided AS IS with no warranty or guarantee of fit for purpose. See the project's LICENSE.md for details. 
 
 ## Prerequisite
 The following softwares are required to use this script
 - Python 3 
 - The [requests](http://docs.python-requests.org/en/master/) library
+- The [websocket-client](https://pypi.org/project/websocket-client/) library (*version 0.49 or greater*, for trkd_wsstreaming.py application only)
+- The [python-dateutil](https://pypi.org/project/python-dateutil/) library (for trkd_wsstreaming.py application only)
 
 All scripts support Python 3 and not compatible with Python 2.
 
@@ -34,46 +39,57 @@ The best way is via the pip package management tool
 1. export <Python_folder>\Scripts to your OS PATH environment
 2. call pip command to install requests
 	```
-	$>pip install requests
+	$>pip install -r requestments.txt
 	```
 3. If you are behind proxy, set the proxy first
 	```
 	export https_proxy="http://<proxy.server>:<port>"
-	$>pip install requests
+	$>pip install -r requestments.txt
 	```
 
+*Note*: If you aim to use only TRKD HTTP JSON services, you can just install requests library via a ```pip install requests``` command.
+
+## References
+For further details, please check out the following resources:
+* [Thomson Reuters Knowledge Direct API page](https://developers.thomsonreuters.com/thomson-reuters-knowledge-direct-trkd) on the [Thomson Reuters Developer Community](https://developers.thomsonreuters.com/) web site.
+* [Thomson Reuters Knowledge Direct API Catalog](https://www.trkd.thomsonreuters.com/SupportSite/RequestBuilder/requestbuilder.aspx) web site.
+* [Elektron WebSocket API](https://developers.thomsonreuters.com/websocket-api) page on the [Thomson Reuters Developer Community](https://developers.thomsonreuters.com/) web site.
+
 ## Release Note
 - Version 1: 6 Sep 2016
-    - trkd_authen.py
-	- trkd_quote.py
+    - trkd_authen.py.
+	- trkd_quote.py.
 - Version 1.0.1: 7 Sep 2016
-	- trkd_newsheadline.py
+	- trkd_newsheadline.py.
 	- changed code structure to separate call http request
 - Version 1.0.2: 19 Sep 2016
-	- trkd_newsstory.py
+	- trkd_newsstory.py.
 - version 1.0.3: 22 Sep 2016
-	- trkd_intraday.py
-	- trkd_interday.py
-	- trkd_onlinereport.py
-	- trkd_chart.py
+	- trkd_intraday.py.
+	- trkd_interday.py.
+	- trkd_onlinereport.py.
+	- trkd_chart.py.
 - version 1.0.4: 28 Oct 2016
-	- docs\TRKD_REST_with_Python.docx
-	- revise some code
+	- docs\TRKD_REST_with_Python.docx.
+	- revise some code.
 - version 1.0.5: 27 Apr 2017
-	- revies README.md to support markdown
+	- revies README.md to support markdown.
 - version 1.0.6: 3 May 2017
-	- revies README.md
-	- modify trkd_authen.py
-	- modify trkd_quote.py
+	- revies README.md.
+	- modify trkd_authen.py.
+	- modify trkd_quote.py.
 - version 1.0.7: 9 May 2017
-	- revise README.md
-	- modify the rest of application files
+	- revise README.md.
+	- modify the rest of application files.
 - version 1.0.7: 31 Aug 2017
 	- revise README.md
 - version 1.0.8: 04 Sep 2017
-	- Port all scripts to support Python 3
+	- Port all scripts to support Python 3.
 	- Fix the issue that some scripts still send request message to the old REST endpoint.
 - version 1.0.9: 26 Jan 2018
-	- Add debug log for checking outgoing message (disabled by default)
+	- Add debug log for checking outgoing message (disabled by default).
 - version 1.0.10: 9 Aug 2018
-	- remove all ```is not None``` statements and make them a bit more **Pythonic**
+	- remove all ```is not None``` statements and make them a bit more **Pythonic**.
+- version 1.0.11: January 2019
+	- Add trkd_wsstreaming.py application for TRKD Streaming service.
+	- Add License.md file
diff --git a/requirements.txt b/requirements.txt
@@ -0,0 +1,4 @@
+###### Requirements Libraries ######
+requests
+websocket-client>0.49
+python-dateutil
diff --git a/trkd_wsstreaming.py b/trkd_wsstreaming.py
@@ -0,0 +1,242 @@
+'''
+The TRKD API sample code is provided for informational purposes only 
+and without knowledge or assumptions of the end users development environment. 
+We offer this code to provide developers practical and useful guidance while developing their own code. 
+However, we do not offer support and troubleshooting of issues that are related to the use of this code 
+in a particular environment; it is offered solely as sample code for guidance. 
+Please see the Thomson Reuters Knowledge Direct product page at http://customers.thomsonreuters.com 
+for additional information regarding the TRKD API.'''
+
+import os
+import sys
+import requests
+import json
+import getpass
+import time
+import getopt
+import socket
+import websocket
+import threading
+from threading import Thread, Event
+
+from datetime import datetime, timezone, timedelta
+import dateutil.parser
+
+
+# Global Default Variables
+ws_address = 'wss://streaming.trkd.thomsonreuters.com/WebSocket/'
+trkd_authen_address = 'https://api.trkd.thomsonreuters.com/api/TokenManagement/TokenManagement.svc/REST/Anonymous/TokenManagement_1/CreateServiceToken_1'
+ws_protocol = 'tr_json2'
+position = socket.gethostbyname(socket.gethostname())
+
+# Global Variables
+web_socket_app = None
+web_socket_open = False
+username = None
+password = None
+appid = None
+token = None
+expiration = None
+logged_in = False
+
+ric_name = 'EUR='
+
+expire_time = 0
+time_to_relogin = 15 * 60 # 15 Minutes to Seconds
+
+## ------------------------------------------ TRKD HTTP REST functions ------------------------------------------ ##
+
+# Send HTTP request for all services
+def doSendRequest(url, requestMsg, headers):
+    result = None
+    try:
+        ##send request
+        result = requests.post(url, data=json.dumps(requestMsg), headers=headers)
+        # print('outgoing message is %s'%(json.dumps(requestMsg)))
+        ## handle error
+        if result.status_code is not 200:
+            print('Request fail')
+            print('response status %s'%(result.status_code))
+            if result.status_code == 500: ## if username or password or appid is wrong
+                print('Error: %s'%(result.json()))
+            result.raise_for_status()
+    except requests.exceptions.RequestException as e:
+        print('Exception!!!')
+        print(e)
+        sys.exit(1)
+    return result
+
+## Perform authentication
+def CreateAuthorization(username, password, appid):
+    token = None
+    expiration = None
+    ##create authentication request URL, message and header
+    authenMsg = {'CreateServiceToken_Request_1': { 'ApplicationID':appid, 'Username':username,'Password':password }}
+    headers = {'content-type': 'application/json;charset=utf-8'}
+    print('############### Sending Authentication request message to TRKD ###############')
+    authenResult = doSendRequest(trkd_authen_address, authenMsg, headers)
+    if authenResult and authenResult.status_code == 200:
+        print('Authentication success')
+        print('response status %s'%(authenResult.status_code))
+        print('Authentication response %s'%json.dumps(authenResult.json(), sort_keys=True, indent=2, separators=(',', ':')))
+        ##get Token
+        token = authenResult.json()['CreateServiceToken_Response_1']['Token']
+        expiration = authenResult.json()['CreateServiceToken_Response_1']['Expiration'] # Expiration time of this session in UTC 
+
+        ## Calcuate Expiration time
+        expire_datetime_utc = dateutil.parser.parse(expiration) ## Parse incoming Expiration to Python datetime object (UTC)
+        now_datetime_utc = datetime.now(timezone.utc) ## Get current machine datetime object in UTC 
+
+        expire_time = expire_datetime_utc - now_datetime_utc ## Get time different between now and expiration time value 
+        expire_time = int(round(expire_time / timedelta(seconds=1))) ## convert it to second as a round int
+
+    return token, expiration, expire_time
+
+## ------------------------------------------ TRKD WebSocket functions ------------------------------------------ ##
+
+# Process incoming messages from TRKD Elektron WebSocket Server
+def process_message(message_json):
+    """ Parse at high level and output JSON of message """
+    message_type = message_json['Type']
+
+    if message_type == "Refresh":
+        if 'Domain' in message_json:
+            message_domain = message_json['Domain']
+            if message_domain == "Login":
+                process_login_response(message_json)
+    # Ping and Pong messages are exchanged between endpoints of a connection to verify that the remote endpoint is still alive.
+    elif message_type == "Ping": 
+        pong_json = { 'Type':'Pong' } # when either endpoint receives a Ping message, it should send a Pong message in response.
+        web_socket_app.send(json.dumps(pong_json))
+        print("SENT:")
+        print(json.dumps(pong_json, sort_keys=True, indent=2, separators=(',', ':')))
+
+# Process incoming Login Refresh message from TRKD Elektron WebSocket Server
+def process_login_response(message_json):
+    """ Send item request """
+    global logged_in
+
+    logged_in = True
+    send_market_price_request(ric_name)
+
+# Send JSON OMM Market Price Request message to TRKD Elektron WebSocket Server
+def send_market_price_request(ric_name):
+    """ Create and send simple Market Price request """
+    mp_req_json = {
+        'ID': 2,
+        'Key': {
+            'Name': ric_name,
+        },
+    }
+    web_socket_app.send(json.dumps(mp_req_json))
+    print("SENT:")
+    print(json.dumps(mp_req_json, sort_keys=True, indent=2, separators=(',', ':')))
+
+# Send JSON OMM Login Request message to TRKD Elektron WebSocket Server to initiate the OMM connection
+def send_login_request(is_refresh_token=False):
+    """ Generate a login request from command line data (or defaults) and send """
+    login_json = {
+        'ID': 1,
+        'Domain': 'Login',
+        'Key': {
+            'Name': '',
+            'NameType': 'AuthnToken',
+            'Elements': {
+                'ApplicationId': '',
+                'Position': '',
+                'AuthenticationToken': ''
+            }
+        }
+    }
+
+    login_json['Key']['Name'] = username
+    login_json['Key']['Elements']['ApplicationId'] = appid
+    login_json['Key']['Elements']['Position'] = position
+    login_json['Key']['Elements']['AuthenticationToken'] = token
+
+    # If the token is a refresh token, this is not our first login attempt.
+    if is_refresh_token:
+        login_json['Refresh'] = False
+
+    web_socket_app.send(json.dumps(login_json))
+    print("SENT:")
+    print(json.dumps(login_json, sort_keys=True, indent=2, separators=(',', ':')))
+
+# Receive every messages from TRKD Elektron WebSocket Server
+def on_message(_, message):
+    """ Called when message received, parse message into JSON for processing """
+    print("RECEIVED: ")
+    message_json = json.loads(message)
+    print(json.dumps(message_json, sort_keys=True, indent=2, separators=(',', ':')))
+
+    for singleMsg in message_json:
+        process_message(singleMsg)
+
+
+def on_error(__file__, error):
+    """ Called when websocket error has occurred """
+    print(error)
+
+
+def on_close(_):
+    """ Called when websocket is closed """
+    global web_socket_open
+    print("WebSocket Closed")
+    web_socket_open = False
+
+# Establish a WebSocket connection success
+def on_open(_):
+    """ Called when handshake is complete and websocket is open, send login """
+
+    print("WebSocket successfully connected!")
+    global web_socket_open
+    web_socket_open = True
+    send_login_request(is_refresh_token=False)
+
+## ------------------------------------------ Main App ------------------------------------------ ##
+
+if __name__ == '__main__':
+    ## Get username, password and applicationid
+    username = input('Please input username: ')
+    ## use getpass.getpass to hide user inputted password
+    password = getpass.getpass(prompt='Please input password: ')
+    appid = input('Please input appid: ')
+
+    token, expiration, expire_time = CreateAuthorization(username,password,appid)
+    print('Token = %s'%(token))
+    print('Expiration  = %s'%(expiration))
+    print('Expiration in next = %d seconds'%(expire_time))
+    ## if authentiacation success, continue subscribing Quote
+    if token and expiration:
+        print('Do WS here')
+        # doWebSocketConnection(username, appid, token)
+        print("Connecting to WebSocket " + ws_address + " ...")
+        web_socket_app = websocket.WebSocketApp(ws_address, header=['User-Agent: Python'],
+            on_message=on_message,
+            on_error=on_error,
+            on_close=on_close,
+            subprotocols=[ws_protocol])
+        web_socket_app.on_open = on_open
+
+        # Event loop
+        wst = threading.Thread(target=web_socket_app.run_forever)
+        wst.start()
+
+        try:
+            while True:
+                if (expire_time > time_to_relogin):
+                    time.sleep(expire_time - time_to_relogin) # Sleep Thread until 15 minutes before expiration
+                else:
+                    # failt the refresh sine value too small
+                    sys.exit(1)
+                
+                # Re-issue login request before token expiration to keep session open. 
+                token, expiration, expire_time = CreateAuthorization(username,password,appid)
+                if not token:
+                    sys.exit(1)
+                if logged_in:
+                    print('############### Re-new Authentication to TRKD ###############')
+                    # Resend JSON OMM Login Request message with Refresh: false to TRKD Elektron WebSocket Server
+                    send_login_request(is_refresh_token=True) 
+        except KeyboardInterrupt:
+            web_socket_app.close()
