Skip to content

Commit da14b80

Browse files
wasin-waeosriwasin-waeosri
committed
add readme for python
1 parent df17c5c commit da14b80

File tree

7 files changed

+265
-3
lines changed

7 files changed

+265
-3
lines changed

‎.gitignore‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
Python/.ipynb_checkpoints
22
Python/.env
3-
Python/esg_data.json
3+
44

55
Java/
66
Java/.idea

‎Python/.dockerignore‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
.env
2+
.env.example

‎Python/.env.example‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
# RDP Core Credentials
2+
RDP_USER=
3+
RDP_PASSWORD=
4+
RDP_APP_KEY=
5+
6+
# RDP Core Endpoints
7+
RDP_BASE_URL= https://api.refinitiv.com
8+
RDP_AUTH_URL=/auth/oauth2/v1/token
9+
RDP_ESG_URL=/data/environmental-social-governance/v2/views/scores-full

‎Python/Dockerfile‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
#Build stage, using slim based-image because alpine cannot use Pandas and Matplotlib
2+
FROM python:3.8.11-slim AS builder
3+
4+
#Copy requirements.txt
5+
COPY requirements.txt .
6+
7+
# install dependencies to the local user directory (eg. /root/.local)
8+
RUN pip install --user -r requirements.txt
9+
10+
# Run stage, using slim based-image because alpine cannot use Pandas and Matplotlib
11+
FROM python:3.8.11-slim
12+
WORKDIR /app
13+
14+
# Set Python buffer to make Docker print every messages instantly.
15+
ENV PYTHONUNBUFFERED=1
16+
17+
# copy only the dependencies installation from the 1st stage image
18+
COPY --from=builder /root/.local /root/.local
19+
COPY rdp_apis_console.py .
20+
21+
# update PATH environment variable
22+
ENV PATH=/root/.local:$PATH
23+
ENV USERNAME=DOCKER_CONTAINER
24+
25+
#Run Python
26+
ENTRYPOINT [ "python", "./rdp_apis_console.py" ]

‎Python/rdp_apis_console.py‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,8 @@
1010
# Load Environment Variables
1111
load_dotenv()
1212

13+
print('User: ', os.getenv('USERNAME'))
14+
1315
# Build and Parse Command line arguments for item/universe, which always change.
1416
my_parser = argparse.ArgumentParser(description='Interested Symbol')
1517
my_parser.add_argument('-i','--item', type = str, default= 'LSEG.L')
@@ -25,7 +27,7 @@
2527
base_URL = os.getenv('RDP_BASE_URL')
2628
auth_endpoint = base_URL + os.getenv('RDP_AUTH_URL')
2729

28-
# Get RDP Credentials information from Environment Variables
30+
# Get RDP Credentials information from Environment Variables
2931
username = os.getenv('RDP_USER')
3032
password = os.getenv('RDP_PASSWORD')
3133
app_key = os.getenv('RDP_APP_KEY')
@@ -93,4 +95,4 @@
9395
df=pd.DataFrame(data=dataArray,columns=titles)
9496

9597
if df.empty is False:
96-
print(df)
98+
print('Top 10 rows data is \n',df.head(10))

‎README.md‎

Lines changed: 223 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,223 @@
1+
# How to separate your credentials, secrets, and configurations from your source code with environment variables
2+
- version: 1.0
3+
- Last update: July 2021
4+
- Environment: Windows
5+
- Prerequisite: [Access to RDP credentials](#prerequisite)
6+
7+
## <a id="intro"></a>Introduction
8+
9+
As a modern application, your application always deal with credentials, secrets and configurations to connect to other services like Authentication service, Database, Cloud services, Microservice, ect. It is not a good idea to keep your username, password and other credentials hard code in your source code as your credentials may leak when you share or publish the application. You need to delete or remark those credentials before you share the code which adds extra work for you. And eventually, you may forgot to do it.
10+
11+
The services configurations such as API endpoint, Database URL should not be embedded in the source code too. The reason is every time you change or update the configurations you need to modify the code which may lead to more errors.
12+
13+
How should we solve this issue?
14+
15+
## <a id=""></a>Store config in the environment
16+
17+
The [Twelve-Factor App methodology](https://12factor.net/) which is one of the most influential pattern to designing scalable software-as-a-service application. The methodology [3rd factor](https://12factor.net/config) (aka Config principle) states that configuration information should be kept as environment as environment variables and injected into the application on runtime as the following quotes:
18+
19+
>An app’s config is everything that is likely to vary between deploys (staging, production, developer environments, etc). This includes:
20+
>- Resource handles to the database, Memcached, and other backing services
21+
>- Credentials to external services such as Amazon S3 or Twitter
22+
>- Per-deploy values such as the canonical hostname for the deploy
23+
>
24+
>Apps sometimes store config as constants in the code. This is a violation of twelve-factor, which requires strict separation of config from code. Config varies substantially across deploys, code does not.
25+
26+
>The twelve-factor app stores config in environment variables (often shortened to env vars or env). Env vars are easy to change between deploys without changing any code; unlike config files, there is little chance of them being checked into the code repo accidentally; and unlike custom config files, or other config mechanisms such as Java System Properties, they are a language- and OS-agnostic standard.
27+
28+
### What is Environment Variables?
29+
30+
An environment variable is a dynamic-named value that set through the Operating System, not the program. The variables are impact the process the OS and running process. In Widows, you can access the environment variables to view or modify them through This PC --> Properties --> Advanced system settings --> Environment Variables.. menu.
31+
32+
![Figure-1](images/01_windows_envs.png "Windows Environment Variables")
33+
34+
The benefits of storing credentials and configurations in environment variables are the following:
35+
1. The credentials and configurations are separated from the code. Project team can change the credentials and configurations based on scenario and environment (Dev, Test, Product, etc) without touching the application source code.
36+
2. The sensitive information (username, password, token, etc) are kept and maintain locally. The team can share the code among peers without be worried about information leak.
37+
3. Reduce the possibility of messing up between environments such as configure the Production server address in the Test environment.
38+
39+
However, the environment variable has OS dependency. Each OS requires different way to access and modify the variables. It is not always practical to set environment variables on development machines (as the variables may keeps growing) or continuous integration servers where multiple projects are run.
40+
41+
These drawbacks leads to the *dotenv* method.
42+
43+
## Introduction to .ENV file and dotenv
44+
45+
The dotenv method lets the application loads variables from a ```.env``` file into environment/running process the same way as the application load variables from environment variables. The application can load or modify the environment variables from the OS and ```.env``` file with a simple function call.
46+
47+
[dotenv](https://github.com/bkeepers/dotenv) is a library that originates from [Ruby](https://www.ruby-lang.org/en/) developers (especially the [Ruby on Rails](https://rubyonrails.org/) framework) and has been widely adopted and ported to many programming languages such as [python-dotenv](https://github.com/theskumar/python-dotenv), [dotenv-java](https://github.com/cdimascio/dotenv-java), [Node.js](https://github.com/motdotla/dotenv), etc.
48+
49+
The ```.env``` file is a simple text file locates at the root of the project with a key-value pair setting as the following:
50+
51+
```
52+
# DB
53+
DB_USER=User
54+
DB_PASSWORD=MyPassword
55+
# Cloud
56+
CLOUD_URL=192.168.1.1
57+
```
58+
59+
Please note that you *do not* need the ```""``` or ```''``` characters for a string value.
60+
61+
### Caution
62+
63+
You *should not* share this ```.env``` file to your peers or commit/push it to the version control. You should add the file to the ```.gitignore``` file to avoid adding it to a version control or public repo accidentally.
64+
65+
You can create a ```.env.example``` file as a template for environment variables and ```.env``` file sharing. The file has the same parameters' keys as a ```.env``` file but without values as the following example:
66+
67+
```
68+
# DB
69+
DB_USER=
70+
DB_PASSWORD=
71+
# Cloud
72+
CLOUD_URL=
73+
```
74+
75+
Then you can push this ```.env.example``` file to the repository. Developers who got your source code project can create their own ```.env``` file from this template ```.env.example``` file.
76+
77+
Please note that if the configuration is not a sensitive information (such as a public API endpoint URL), you can include it to a ```.env.example``` file.
78+
79+
## dotenv with Python
80+
81+
Let's demonstrate with the [python-dotenv](https://github.com/theskumar/python-dotenv) library first. The example console application uses python-dotenv library to store the [Refinitiv Data Platform (RDP) APIs](https://developers.refinitiv.com/en/api-catalog/refinitiv-data-platform/refinitiv-data-platform-apis) credentials and configurations.
82+
83+
### <a id="whatis_rdp"></a>What is Refinitiv Data Platform (RDP) APIs?
84+
85+
The [Refinitiv Data Platform (RDP) APIs](https://developers.refinitiv.com/en/api-catalog/refinitiv-data-platform/refinitiv-data-platform-apis) provide various Refinitiv data and content for developers via easy to use Web based API.
86+
87+
RDP APIs give developers seamless and holistic access to all of the Refinitiv content such as Historical Pricing, Environmental Social and Governance (ESG), News, Research, etc and commingled with their content, enriching, integrating, and distributing the data through a single interface, delivered wherever they need it. The RDP APIs delivery mechanisms are the following:
88+
* Request - Response: RESTful web service (HTTP GET, POST, PUT or DELETE)
89+
* Alert: delivery is a mechanism to receive asynchronous updates (alerts) to a subscription.
90+
* Bulks: deliver substantial payloads, like the end of day pricing data for the whole venue.
91+
* Streaming: deliver real-time delivery of messages.
92+
93+
This example project is focusing on the Request-Response: RESTful web service delivery method only.
94+
95+
For more detail regarding Refinitiv Data Platform, please see the following APIs resources:
96+
- [Quick Start](https://developers.refinitiv.com/en/api-catalog/refinitiv-data-platform/refinitiv-data-platform-apis/quick-start) page.
97+
- [Tutorials](https://developers.refinitiv.com/en/api-catalog/refinitiv-data-platform/refinitiv-data-platform-apis/tutorials) page.
98+
- [RDP APIs: Introduction to the Request-Response API](https://developers.refinitiv.com/en/api-catalog/refinitiv-data-platform/refinitiv-data-platform-apis/tutorials#introduction-to-the-request-response-api) page.
99+
- [RDP APIs: Authorization - All about tokens](https://developers.refinitiv.com/en/api-catalog/refinitiv-data-platform/refinitiv-data-platform-apis/tutorials#authorization-all-about-tokens) page.
100+
101+
## python-dotenv and .env file set up
102+
103+
You can install the python-dotenv library via the following pip command:
104+
105+
```
106+
pip install python-dotenv
107+
```
108+
109+
The create a ```.env``` file at the root of the project with the following content
110+
111+
```
112+
# RDP Core Credentials
113+
RDP_USER=<Your RDP username>
114+
RDP_PASSWORD=<Your RDP password>
115+
RDP_APP_KEY=<Your RDP appkey>
116+
117+
# RDP Core Endpoints
118+
RDP_BASE_URL = https://api.refinitiv.com
119+
RDP_AUTH_URL=/auth/oauth2/v1/token
120+
RDP_ESG_URL=/data/environmental-social-governance/v2/views/scores-full
121+
```
122+
123+
## Using python-dotenv library
124+
125+
To use the python-dotenv library, you just import the library and call ```load_dotenv()``` statement. Then you can access both System environment variables and ```.env```'s configurations from the Python ```os.environ``` or ```os.getenv``` statement.
126+
127+
Please note that the OS/System's environment variables always override ```.env``` configurations by default as the following example.
128+
129+
```
130+
import os
131+
from dotenv import load_dotenv
132+
133+
load_dotenv() # take environment variables from .env.
134+
135+
print('User: ', os.getenv('USERNAME')) # Return your System USERNAME configuration.
136+
```
137+
138+
The following example shows how to use get configurations from a ```.env``` file to get the RDP APIs Auth service endpoint and user's RDP credential.
139+
140+
```
141+
# Get RDP Token service information from Environment Variables
142+
base_URL = os.getenv('RDP_BASE_URL')
143+
auth_endpoint = base_URL + os.getenv('RDP_AUTH_URL')
144+
145+
# Get RDP Credentials information from Environment Variables
146+
username = os.getenv('RDP_USER')
147+
password = os.getenv('RDP_PASSWORD')
148+
app_key = os.getenv('RDP_APP_KEY')
149+
```
150+
151+
Next, the application creates the RDP Auth service request message and sends the HTTP Post request message to the RDP APIs endpoint.
152+
153+
```
154+
import requests
155+
156+
# -- Init and Authenticate Session
157+
auth_request_msg = {
158+
'username': username ,
159+
'password': password ,
160+
'grant_type': "password",
161+
'scope': scope,
162+
'takeExclusiveSignOnControl': "true"
163+
}
164+
165+
# Authentication with RDP Auth Service
166+
try:
167+
response = requests.post(auth_endpoint, headers = {'Accept':'application/json'}, data = auth_request_msg, auth = (app_key, client_secret))
168+
except Exception as exp:
169+
print('Caught exception: %s' % str(exp))
170+
171+
if response.status_code == 200: # HTTP Status 'OK'
172+
print('Authentication success')
173+
auth_obj = response.json()
174+
else:
175+
print('RDP authentication result failure: %s %s' % (response.status_code, response.reason))
176+
print('Text: %s' % (response.text))
177+
```
178+
The next step is requesting ESG (Environmental, Social, and Governance) data from RDP. We use the ESG scores-full API endpoint which provides full coverage of Refinitiv's proprietary ESG Scores with full history for consumers as an example API.
179+
180+
The ESG scores-full API requires the item name (aka universe) information which is most likely to be different for each individual run , so we get this information via a command line argument.
181+
182+
```
183+
import argparse
184+
185+
my_parser = argparse.ArgumentParser(description='Interested Symbol')
186+
my_parser.add_argument('-i','--item', type = str, default= 'LSEG.L')
187+
args = my_parser.parse_args()
188+
189+
universe = args.item
190+
```
191+
192+
We get the RDP ESG Service API endpoint from a ```.env``` file.
193+
194+
```
195+
# Get RDP Token service information from Environment Variables
196+
esg_url = base_URL + os.getenv('RDP_ESG_URL')
197+
198+
payload = {'universe': universe}
199+
esg_object = None
200+
201+
# Request data for ESG Score Full Service
202+
try:
203+
response = requests.get(esg_url, headers={'Authorization': 'Bearer {}'.format(auth_obj['access_token'])}, params = payload)
204+
except Exception as exp:
205+
print('Caught exception: %s' % str(exp))
206+
```
207+
Then we can get the ESG data from the ```response.json()``` statement for further data processing.
208+
209+
The above code shows that you do not need to change the code if the RDP credentials or service endpoint is changed (example update API version). We can just update the configurations in a ```.env``` file and re-run the application.
210+
211+
## <a id="references"></a>References
212+
For further details, please check out the following resources:
213+
* [Refinitiv Data Platform APIs page](https://developers.refinitiv.com/en/api-catalog/refinitiv-data-platform/refinitiv-data-platform-apis) on the [Refinitiv Developer Community](https://developers.refinitiv.com/) website.
214+
* [Refinitiv Data Platform APIs Playground page](https://api.refinitiv.com).
215+
* [Refinitiv Data Platform APIs: Introduction to the Request-Response API](https://developers.refinitiv.com/en/api-catalog/refinitiv-data-platform/refinitiv-data-platform-apis/tutorials#introduction-to-the-request-response-api).
216+
* [Refinitiv Data Platform APIs: Authorization - All about tokens](https://developers.refinitiv.com/en/api-catalog/refinitiv-data-platform/refinitiv-data-platform-apis/tutorials#authorization-all-about-tokens).
217+
* [Using RDP API to request ESG data on Jupyter Notebook](https://developers.refinitiv.com/en/article-catalog/article/using-rdp-api-request-esg-data-jupyter-notebook)
218+
* [python-dotenv GitHub page](https://github.com/theskumar/python-dotenv)
219+
* [How to NOT embedded credential in Jupyter notebook](https://yuthakarn.medium.com/how-to-not-show-credential-in-jupyter-notebook-c349f9278466)
220+
https://www.reddit.com/r/node/comments/6cz4jw/having_trouble_understanding_the_benefits_and/
221+
222+
223+
For any questions related to Refinitiv Data Platform, please use the Developers Community [Q&A Forum](https://community.developers.refinitiv.com/spaces/231/index.html).

‎images/01_windows_envs.png‎

32.4 KB
Loading

0 commit comments

Comments
 (0)