Skip to content

Commit f7113cf

Browse files
author
Clark Perkins
committed
Initial port of stackdio-tooling into this repo
1 parent 7391b63 commit f7113cf

File tree

19 files changed

+2164
-2
lines changed

19 files changed

+2164
-2
lines changed

‎.gitignore‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -55,3 +55,5 @@ target/
5555

5656
# PyCharm
5757
.idea/
58+
59+
.vagrant/

‎MANIFEST.in‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
include README.md

‎README.md‎

Lines changed: 104 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,107 @@
11
stackdio-python-client
22
======================
33

4-
The canonical Python client for the stackd.io API
4+
The canonical Python client and cli for the stackd.io API
5+
6+
7+
## Overview
8+
This is a small set of tools for internal use of stackd.io. After cloning
9+
this repo, you should be able to quickly get up and running with your own
10+
stacks.
11+
12+
Advanced usage like creating custom blueprints or writing your own formulas is
13+
beyond the scope of this.
14+
15+
## Installation
16+
We recommend using virtualenv via [virtualenvwrapper] to install this in a
17+
virtualenv. If you consider yourself a knowledgeable Pythonista, feel free to
18+
install this however you'd like, but this document will assume that you are
19+
using virtualenvwrapper. See the full [virtualenvwrapper] docs for details,
20+
but in short you can install it on most systems like:
21+
22+
pip install virtualenvwrapper
23+
24+
Once you've got it, installing this tool goes something like:
25+
26+
mkvirtualenv stackdio-tooling
27+
28+
# assuming you are in whatever dir you cloned this repo to:
29+
pip install --process-dependency-links .
30+
31+
** The --process-dependency-links flag is only needed in pip 1.5.6 **
32+
You'll see a few things scrolling by, but should be set after this. To use
33+
this later, you'll need to re-activate the virtualenv like:
34+
35+
workon stackdio-tooling
36+
37+
Whenever it's activated, `stackdio-cli` should be on your path.
38+
39+
## First Use
40+
The first time that you fire up `stackdio-cli`, you'll need to run the
41+
`initial-setup` command. This will prompt you for your LDAP username and
42+
password, and store them securely in your OS keychain for later use. It will
43+
import some standard formula, and create a few commonly used blueprints.
44+
45+
$ stackdio-cli
46+
None @ https://stackd.corp.digitalreasoning.com/api/
47+
> initial_setup
48+
# YOU WILL BE WALKED THROUGH A SIMPLE SET OF QUESTIONS
49+
50+
## Stack Operations
51+
All of the following assume that you have run `initial-setup` successfully. To
52+
launch the cli, simply type:
53+
54+
$ stackdio-cli
55+
56+
You can run `help` at any point to see available commands. For details on a
57+
specific command you can run `help COMMAND`, e.g. `help stacks`. The rest of
58+
these commands assume you have the cli running.
59+
60+
### Launching Stacks
61+
Stacks are launched from blueprints. To launch the 3 node HBase stack that's
62+
included with this you do:
63+
64+
> stacks launch cdh450-ipa-3 MYSTACKNAME
65+
66+
**NOTE:** To avoid DNS namespace collisions, the stack name needs to be unique.
67+
An easy way to ensure this is to include your name in the stack name.
68+
69+
### Deleting Stacks
70+
When you are done with a stack you can delete it. This is destructive and
71+
cannot be recovered from, so think carefully before deleting your stack!
72+
73+
> stacks delete STACK_NAME
74+
75+
Alternatively you can `terminate` a stack which will terminate all instances,
76+
but leave the stack definition in place.
77+
78+
### Provisioning Stacks
79+
Occassionally something will go wrong when launching your stack, e.g. network
80+
connections may flake out causing some package installations to fail. If this
81+
happens you can manually provision your stack, causing everything to be brought
82+
back up to date:
83+
84+
> stacks provision STACK_NAME
85+
86+
### Stack Info
87+
Once you have launched a stack, you can then monitor the status of it like:
88+
89+
> stacks history STACK_NAME
90+
91+
This displays the top level information for a stack. You can supply additional
92+
arguments to pull back additional info about a stack. For example, to get a
93+
list of FQDNs (aka hostnames) for a stack:
94+
95+
> stacks hostnames STACK_NAME
96+
97+
There are various logs available that you can access with the `stacks logs`
98+
command.
99+
100+
## What's Next?
101+
For anything not covered by this tool, you'll need to use the [web UI] or
102+
[API] directly. See someone on the [pi team] with specific questions.
103+
104+
[virtualenvwrapper]: https://pypi.python.org/pypi/virtualenvwrapper
105+
[web UI]: https://stackd.corp.digitalreasoning.com
106+
[API]: https://stackd.corp.digitalreasoning.com/api
107+
[pi team]: mailto:pi@digitalreasoning.com?subject=stackd.io%20questions

‎Vagrantfile‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
# -*- mode: ruby -*-
2+
# vi: set ft=ruby :
3+
4+
# Vagrantfile API/syntax version. Don't touch unless you know what you're doing!
5+
VAGRANTFILE_API_VERSION = "2"
6+
7+
Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
8+
config.vm.box = "precise64"
9+
10+
config.vm.provision "shell", path: "vbox_setup.sh"
11+
12+
config.vm.synced_folder "~/Workspace/pi/stackdio-blueprints", "/home/vagrant/.stackdio-blueprints", create: true
13+
end

‎blueprints/cdh4-3node.json‎

Lines changed: 191 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,191 @@
1+
{
2+
"public": false,
3+
"properties": {
4+
"java": {
5+
"enable_jce": false,
6+
"oracle": {
7+
"cookies": "gpw_e24=http%3A%2F%2Fwww.oracle.com%2F; oraclelicense=accept-securebackup-cookie",
8+
"staging": "/tmp/.java_staging",
9+
"jdk7": {
10+
"rpm": "jdk-1.7.0_45",
11+
"uri": "http://download.oracle.com/otn-pub/java/jdk/7u45-b18/jdk-7u45-linux-x64.rpm"
12+
}
13+
}
14+
},
15+
"cdh4": {
16+
"landing_page": true,
17+
"hue": {
18+
"secret_key": "CHANGE_THIS",
19+
"start_service": true
20+
},
21+
"mapred": {
22+
"system_dir": "/hadoop/system/mapred",
23+
"map_tasks_max": 5,
24+
"reduce_tasks_max": 3,
25+
"local_dir": "/mnt/hadoop/mapred/local",
26+
"child_java_opts": "-Xmx2000m",
27+
"reduce_tasks": 6,
28+
"child_ulimit": 8000000
29+
},
30+
"zookeeper": {
31+
"data_dir": "/mnt/zk/data",
32+
"start_service": true
33+
},
34+
"max_log_index": 1,
35+
"dfs": {
36+
"name_dir": "/mnt/hadoop/hdfs/nn",
37+
"block_size": 268435456,
38+
"data_dir": "/mnt/hadoop/hdfs/data",
39+
"permissions": true
40+
},
41+
"version": 4,
42+
"datanode": {
43+
"start_service": true
44+
},
45+
"hive": {
46+
"home": "/usr/lib/hive",
47+
"metastore_password": "CHANGE_THIS",
48+
"start_service": true,
49+
"user": "hive"
50+
},
51+
"io": {
52+
"sort_mb": 250,
53+
"sort_factor": 25
54+
},
55+
"oozie": {
56+
"start_service": true
57+
},
58+
"namenode": {
59+
"start_service": true
60+
},
61+
"hbase": {
62+
"region_initial_heap": "1024m",
63+
"region_max_heap": "1024m",
64+
"tmp_dir": "/mnt/hbase/tmp",
65+
"start_service": true,
66+
"master_young_gen": "256m",
67+
"replication": 3,
68+
"master_initial_heap": "1024m",
69+
"manage_zk": true,
70+
"log_dir": "/mnt/hbase/logs",
71+
"region_young_gen": "256m",
72+
"master_max_heap": "1024m",
73+
"jute_maxbuffer": 1000000
74+
},
75+
"impala": {
76+
"version": "1.2.3",
77+
"start_service": true
78+
}
79+
}
80+
},
81+
"hosts": [
82+
{
83+
"count": 1,
84+
"description": "NameNode and Master",
85+
"zone": "us-east-1d",
86+
"title": "NameNode and Master",
87+
"hostname_template": "{namespace}-nn",
88+
"cloud_profile": "TO_BE_CHANGED",
89+
"formula_components": [
90+
{
91+
"id": [
92+
"OpenJDK and Oracle Java",
93+
"Oracle Java 7 JDK"
94+
],
95+
"order": 0
96+
},
97+
{
98+
"id": [
99+
"CDH4 Salt Formula",
100+
"NameNode"
101+
],
102+
"order": 1
103+
},
104+
{
105+
"id": [
106+
"CDH4 Salt Formula",
107+
"HBase Master"
108+
],
109+
"order": 3
110+
},
111+
{
112+
"id": [
113+
"CDH4 Salt Formula",
114+
"Oozie"
115+
],
116+
"order": 5
117+
},
118+
{
119+
"id": [
120+
"CDH4 Salt Formula",
121+
"Pig"
122+
],
123+
"order": 6
124+
},
125+
{
126+
"id": [
127+
"CDH4 Salt Formula",
128+
"Hive"
129+
],
130+
"order": 7
131+
},
132+
{
133+
"id": [
134+
"CDH4 Salt Formula",
135+
"Impala State Store"
136+
],
137+
"order": 8
138+
},
139+
{
140+
"id": [
141+
"CDH4 Salt Formula",
142+
"Hue"
143+
],
144+
"order": 10
145+
}
146+
],
147+
"size": "m2.2xlarge"
148+
},
149+
{
150+
"count": 2,
151+
"description": "DataNodes and Regionservers.",
152+
"zone": "us-east-1d",
153+
"title": "DataNodes and RegionServers",
154+
"hostname_template": "{namespace}-dn-{index}",
155+
"cloud_profile": "TO_BE_CHANGED",
156+
"formula_components": [
157+
{
158+
"id": [
159+
"OpenJDK and Oracle Java",
160+
"Oracle Java 7 JDK"
161+
],
162+
"order": 0
163+
},
164+
{
165+
"id": [
166+
"CDH4 Salt Formula",
167+
"DataNode"
168+
],
169+
"order": 2
170+
},
171+
{
172+
"id": [
173+
"CDH4 Salt Formula",
174+
"HBase RegionServer"
175+
],
176+
"order": 4
177+
},
178+
{
179+
"id": [
180+
"CDH4 Salt Formula",
181+
"Impala Server"
182+
],
183+
"order": 9
184+
}
185+
],
186+
"size": "m2.2xlarge"
187+
}
188+
],
189+
"description": "3 node stack with CDH 4",
190+
"title": "cdh4-3node"
191+
}

0 commit comments

Comments
 (0)