Skip to content
This repository was archived by the owner on Mar 13, 2025. It is now read-only.

Commit fc5adf0

Browse files
jmgasperjmgasper
committed
Dockerization docs and scripts
1 parent d446e1f commit fc5adf0

9 files changed

+665
-0
lines changed

‎dockerize/README.md

Lines changed: 95 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,95 @@
1+
Note this doc is informational only, it provides guides about how to build docker images. For validation refer to [VALIDATION.md](VALIDATION.md) for details.
2+
3+
4+
5+
## How To Build Docker Images
6+
7+
In each app, there is a `build.sh` file which uses a `Dockerfile` file to build that app.
8+
9+
The `build.sh` expects 2 arguments:
10+
11+
- BUILD_TYPE: The build type, like "local", "dev", "test", "prod". Required.
12+
13+
- ENV_FOLDER: The folder containing env files. Optional.
14+
15+
If ENV_FOLDER is not present, then system environment values will be used; otherwise the `<app-name>.env.default` and `<app-name>.env.<BUILD_TYPE>` files within given ENV_FOLDER will be used.
16+
17+
18+
19+
For example, build topcoder-x-ui with system environment values:
20+
21+
```bash
22+
./build.sh dev
23+
# This will lookup the env values prefixed with "DEFAULT_" and "DEV_", then merge and inject them into docker image. The docker image built will have tag "topcoder-x-ui:dev"
24+
25+
./build.sh prod
26+
# This will lookup the env values prefixed with "DEFAULT_" and "PROD_", then merge and inject them into docker image. The docker image built will have tag "topcoder-x-ui:prod"
27+
28+
29+
# Assuming following system environment values:
30+
# DEFAULT_VERSION=1.0
31+
# DEFAULT_SECRET=111111
32+
# DEV_PORT=8080
33+
# DEV_SECRET=123456
34+
# PROD_PORT=80
35+
# PROD_SECRET=654321
36+
37+
# Then for dev deployment type the generated .env file will have following content:
38+
# VERSION=1.0
39+
# PORT=8080
40+
# SECRET=123456
41+
42+
# Then for prod deployment type the generated .env file will have following content:
43+
# VERSION=1.0
44+
# PORT=80
45+
# SECRET=654321
46+
```
47+
48+
49+
50+
Another example, build topcoder-x-ui with env files:
51+
52+
```bash
53+
./build.sh dev <path-to-env-files-folder>
54+
# This will get the env values in topcoder-x-ui.env.default and topcoder-x-ui.env.dev in the given folder, then merge and inject them into docker image. The docker image built will have tag "topcoder-x-ui:dev"
55+
56+
./build.sh prod <path-to-env-files-folder>
57+
# This will get the env values in topcoder-x-ui.env.default and topcoder-x-ui.env.prod in the given folder, then merge and inject them into docker image. The docker image built will have tag "topcoder-x-ui:prod"
58+
59+
60+
# Assuming <path-to-env-files-folder>/topcoder-x-ui.env.default has following content:
61+
# VERSION=1.0
62+
# SECRET=111111
63+
# Assuming <path-to-env-files-folder>/topcoder-x-ui.env.dev has following content:
64+
# PORT=8080
65+
# SECRET=123456
66+
# Assuming <path-to-env-files-folder>/topcoder-x-ui.env.prod has following content:
67+
# PORT=80
68+
# SECRET=654321
69+
70+
# Then for dev deployment type the generated .env file will have following content:
71+
# VERSION=1.0
72+
# PORT=8080
73+
# SECRET=123456
74+
75+
# Then for prod deployment type the generated .env file will have following content:
76+
# VERSION=1.0
77+
# PORT=80
78+
# SECRET=654321
79+
```
80+
81+
82+
83+
## Supported ENV
84+
85+
Following lists the env variables supported:
86+
87+
- topcoder-x-ui: refer to `topcoder-x-ui/cofiguration.md` for supported env variables.
88+
- topcoder-x-receiver: refer to `topcoder-x-receiver/cofiguration.md` for supported env variables.
89+
- topcoder-x-processor: refer to `topcoder-x-processor/cofiguration.md` for supported env variables.
90+
91+
92+
93+
## Validation
94+
95+
For a local validation, please refer to [VALIDATION.md](VALIDATION.md) for details.

‎dockerize/VALIDATION.md

Lines changed: 248 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,248 @@
1+
## Prerequisite
2+
3+
Git
4+
5+
Docker & Docker-compose
6+
7+
JDK 1.8 & OpenSSL: used to generate certificates. The OpenSSL most likely already exists in Linux/Mac system, if you don't have it get it here https://wiki.openssl.org/index.php/Binaries
8+
9+
10+
11+
## Clone Repo
12+
13+
Within submission root dir, clone git repos, e.g clone the master branch to deploy:
14+
15+
```bash
16+
# Within submission root dir
17+
18+
# The submission works on commit 96988f9ce80d968fe6be0a0fd31632b73442cbc0
19+
git clone -b master https://github.com/topcoder-platform/topcoder-x-ui
20+
cd topcoder-x-ui && git checkout 96988f9ce80d968fe6be0a0fd31632b73442cbc0 && cd ..
21+
22+
# The submission works on commit 3c5d798d133adc5440ff61e56345a03fca9167cf
23+
git clone -b master https://github.com/topcoder-platform/topcoder-x-receiver
24+
cd topcoder-x-receiver && git checkout 3c5d798d133adc5440ff61e56345a03fca9167cf && cd ..
25+
26+
# The submission works on commit e408d62fd190e9b2412e02004c916a6afa5e70db
27+
git clone -b master https://github.com/topcoder-platform/topcoder-x-processor
28+
cd topcoder-x-processor && git checkout e408d62fd190e9b2412e02004c916a6afa5e70db && cd ..
29+
```
30+
31+
32+
33+
## Apply Patch
34+
35+
```bash
36+
cd topcoder-x-ui
37+
git am --ignore-space-change ../patches/topcoder-x-ui.patch
38+
cd ..
39+
40+
cd topcoder-x-receiver
41+
git am --ignore-space-change ../patches/topcoder-x-receiver.patch
42+
cd ..
43+
44+
cd topcoder-x-processor
45+
git am --ignore-space-change ../patches/topcoder-x-processor.patch
46+
cd ..
47+
```
48+
49+
50+
51+
## Config Docker IP
52+
53+
Normally you can use `127.0.0.1` as the docker ip. If you use docker machine, use `docker-machine ip` to get the docker machine ip.
54+
55+
Then in your hosts file config `x.topcoder-dev.com` to resolve to docker ip:
56+
57+
```
58+
127.0.0.1 x.topcoder-dev.com
59+
```
60+
61+
62+
63+
## Setup GitHub OAuth App
64+
65+
Follow this section to generate your own `GITHUB_CLIENT_ID` and `GITHUB_CLIENT_SECRET` in [local-deploy/env/topcoder-x-ui.env.local](local-deploy/env/topcoder-x-ui.env.local) configuration:
66+
67+
1. Login into github.com
68+
2. Navigate to https://github.com/settings/developers
69+
3. Click `New OAuth App` button
70+
4. Fill in the fields with following values (See http://take.ms/ajdgy):
71+
- Application name: Topcoder-X
72+
- Homepage URL: http://x.topcoder-dev.com
73+
- Authorization callback URL: http://x.topcoder-dev.com
74+
5. After creating the OAuth app, you can see its client id and client secret, set them as `GITHUB_CLIENT_ID` and `GITHUB_CLIENT_SECRET` values in [local-deploy/env/topcoder-x-ui.env.default](local-deploy/env/topcoder-x-ui.env.default) configuration
75+
76+
77+
78+
Also config your own `GITHUB_ACCESS_TOKEN` in [local-deploy/env/topcoder-x-processor.env.test](local-deploy/env/topcoder-x-processor.env.test) , you can create access token from https://github.com/settings/tokens
79+
80+
81+
82+
## Setup GitLab OAuth App
83+
84+
Follow this section to generate your own `GITLAB_CLIENT_ID` and `GITLAB_CLIENT_SECRET` in [local-deploy/env/topcoder-x-ui.env.local](local-deploy/env/topcoder-x-ui.env.local) configuration:
85+
86+
- Login into gitlab.com
87+
- Navigate to https://gitlab.com/profile/applications
88+
- Fill in the fields with following values (See http://take.ms/ERoDs):
89+
- Name: Topcoder-X
90+
- Redirect URI: enter two callback URLs, one callback URL per line, so there are two lines:
91+
http://topcoderx.topcoder-dev.com/api/v1/gitlab/owneruser/callback
92+
http://topcoderx.topcoder-dev.com/api/v1/gitlab/normaluser/callback
93+
- Scopes: check `api` and `read_user`, `api` is for owner user, `read_user` is for normal user
94+
- After creating the OAuth app, you can see its generated Application Id and Secret, set them as `GITLAB_CLIENT_ID` and `GITLAB_CLIENT_SECRET` values in [local-deploy/env/topcoder-x-ui.env.default](local-deploy/env/topcoder-x-ui.env.default) configuration
95+
96+
97+
98+
Also config your own `GITLAB_USERNAME` and `GITLAB_PASSWORD` in [local-deploy/env/topcoder-x-processor.env.test](local-deploy/env/topcoder-x-processor.env.test)
99+
100+
101+
102+
## Setup Ngrok
103+
104+
Download Ngrok from https://ngrok.com/download and install it. Then signup and go to https://dashboard.ngrok.com/auth to find your auth token. run:
105+
106+
```bash
107+
./ngrok authtoken <your-own-ngrok-token>
108+
./ngrok http 3002
109+
# You will see something like following:
110+
111+
# Session Status online
112+
# Session Expired Restart ngrok or upgrade: ngrok.com/upgrade
113+
# Version 2.2.8
114+
# Region United States (us)
115+
# Web Interface http://127.0.0.1:4040
116+
# Forwarding http://a9ee41f3.ngrok.io -> localhost:3002
117+
# Forwarding https://a9ee41f3.ngrok.io -> localhost:3002
118+
```
119+
120+
121+
122+
Set your own forwarding http URL (like `http://a9ee41f3.ngrok.io`) to `HOOK_BASE_URL` value in [local-deploy/env/topcoder-x-ui.env.default](local-deploy/env/topcoder-x-ui.env.default) and [local-deploy/env/topcoder-x-processor.env.test](local-deploy/env/topcoder-x-processor.env.test)
123+
124+
125+
126+
At this point the configuration are all setup.
127+
128+
129+
130+
## Deploy locally
131+
132+
```bash
133+
cd local-deploy
134+
./local-deploy.sh local
135+
136+
# The first time it takes several minutes to build and deploy, be patient
137+
```
138+
139+
140+
141+
When you see something like following, the docker services are all started:
142+
143+
```verilog
144+
Creating network "topcoder-x-docker_default" with the default driver
145+
Creating maildev.local ... done
146+
Creating mongo.local ... done
147+
Creating zookeeper.local ... done
148+
Creating kafka.local ... done
149+
maildev.local is up-to-date
150+
mongo.local is up-to-date
151+
zookeeper.local is up-to-date
152+
kafka.local is up-to-date
153+
Creating topcoder-x-ui.local ... done
154+
Creating topcoder-x-processor.local ... done
155+
Creating topcoder-x-receiver.local ... done
156+
```
157+
158+
159+
160+
## Github Verification
161+
162+
Goto http://x.topcoder-dev.com , login with `mess/appirio123` : http://take.ms/Erprt
163+
164+
Goto http://x.topcoder-dev.com/#/app/settings , setup the OAuth authentication of Github and Gitlab: http://take.ms/d2RqT
165+
166+
167+
168+
Goto http://x.topcoder-dev.com/#/app/projects , add a project with github repo (you can create a public repo to test), enter 7377 for Direct ID: http://take.ms/hn4p0
169+
170+
171+
172+
After project created, click `Edit` button: http://take.ms/hn4p0
173+
174+
Then click `Add Labels` button, then in github labels page (https://github.com/you-user-name/your-repo-name/labels) you should see "tax_XXX" labels created: http://take.ms/4naCe
175+
176+
Then click `Add Webhook` button, then in github webhooks page (https://github.com/your-user-name/your-repo-name/settings/hooks) you should see webhooks created: http://take.ms/Xgexa
177+
178+
Then click `Add Wiki Rules` button, then in github issues page you should see a ticket rules issue created: http://take.ms/sIx0T
179+
180+
181+
182+
Now in Github, create an issue with name `[$100] Some test issue` (the issue name format is important), then wait about one minute, refresh the issue page you should see a challenge is auto created: http://take.ms/zlibK . Then click the challenge url: http://take.ms/bDU5w
183+
184+
185+
186+
At this point we have verified the Github OAuth authentication, the communication between Github/Topcoder-x/Kafka and TC-API works.
187+
188+
189+
190+
## Gitlab Verification
191+
192+
Goto http://x.topcoder-dev.com , login with `mess/appirio123` : http://take.ms/Erprt
193+
194+
Goto http://x.topcoder-dev.com/#/app/settings , setup the OAuth authentication of Github and Gitlab: http://take.ms/d2RqT
195+
196+
197+
198+
Goto http://x.topcoder-dev.com/#/app/projects , add a project with gitlab repo (you can create a public repo to test), enter 7377 for Direct ID: http://take.ms/XDvPB
199+
200+
201+
202+
After project created, click `Edit` button: http://take.ms/mLtlq
203+
204+
Then click `Add Labels` button, then in gitlab labels page (https://gitlab.com/your-user-name/your-repo-name/labels) you should see "tax_XXX" labels created: http://take.ms/gOMMX
205+
206+
Then click `Add Webhook` button, then in gitlab webhooks page (https://gitlab.com/your-user-name/your-repo-name/settings/integrations) you should see webhooks created: http://take.ms/EjHLy
207+
208+
Then click `Add Wiki Rules` button, then in gitlab wiki page (https://gitlab.com/your-user-name/your-repo-name/wikis/Gitlab-ticket-rules) you should see the ticket rules wiki page created: http://take.ms/yrzF3
209+
210+
211+
212+
Now in Gitlab, create an issue with name `[$100] Some test issue` (the issue name format is important), then wait about one minute, refresh the issue page you should see a challenge is auto created: http://take.ms/K8UHS. Then click the challenge url: http://take.ms/U5afZ
213+
214+
215+
216+
At this point we have verified the Gitlab OAuth authentication, the communication between Gitlab/Topcoder-x/Kafka and TC-API works.
217+
218+
219+
220+
## Unit Tests
221+
222+
At first deploy test environment:
223+
224+
```bash
225+
cd local-deploy
226+
./local-deploy.sh test
227+
```
228+
229+
230+
231+
Then:
232+
233+
```bash
234+
# Unit tests for gitlab
235+
docker exec -it topcoder-x-processor.test npm run test:gitlab
236+
237+
# It has code bug and immediately fails
238+
```
239+
240+
241+
242+
```bash
243+
# Unit tests for github
244+
docker exec -it topcoder-x-processor.test npm run test:github
245+
246+
# It runs more than 30 minutes and still not finish
247+
```
248+

0 commit comments

Comments
 (0)