@@ -2,312 +2,4 @@ Title: Setting up private clouds with Simplestreams
22
33# Set up a Private Cloud using Simplestreams
44
5- When Juju bootstraps a cloud, it needs two critical pieces of information:
6-
7- 1 . The uuid of the image to use when starting new compute instances.
8- 2 . The URL from which to download the correct version of a tools tarball.
9-
10- This necessary information is stored in a json metadata format called
11- "simplestreams". For supported public cloud services such as Amazon Web
12- Services, HP Cloud, Azure, etc, no action is required by the end user. However,
13- those setting up a private cloud, or who want to change how things work (eg use
14- a different Ubuntu image), can create their own metadata, after understanding a
15- bit about how it works.
16-
17- The simplestreams format is used to describe related items in a structural
18- fashion.([ See the Launchpad project lp: simplestreams for more details on
19- implementation] ( https://launchpad.net/simplestreams ) ). Below we will discuss how
20- Juju determines which metadata to use, and how to create your own images and
21- tools and have Juju use them instead of the defaults.
22-
23- ## Basic Workflow
24-
25- Whether images or tools, Juju uses a search path to try and find suitable
26- metadata. The path components (in order of lookup) are:
27-
28- 1 . User supplied location (specified by tools-metadata-url or image-metadata-url config settings).
29- 2 . The environment's cloud storage.
30- 3 . Provider specific locations (eg keystone endpoint if on Openstack).
31- 4 . A web location with metadata for supported public clouds
32- (https://streams.canonical.com for tools and http://cloud-images.ubuntu.com for images).
33-
34- Metadata may be inline signed, or unsigned. We indicate a metadata file is
35- signed by using the '.sjson' extension. Each location in the path is first
36- searched for signed metadata, and if none is found, unsigned metadata is
37- attempted before moving onto the next path location.
38-
39- Juju ships with public keys used to validate the integrity of image and tools
40- metadata obtained from http://cloud-images.ubuntu.com and https://streams.canonical.com .
41- So out of the box, Juju will "Just Work" with any supported public cloud, using
42- signed metadata. Setting up metadata for a private (eg Openstack) cloud requires
43- metadata to be generated using tools which ship with Juju.
44-
45- ## Image Metadata Contents
46-
47- Image metadata uses a simplestreams content type of "image-ids". The product id
48- is formed as follows:
49-
50- com.ubuntu.cloud:server:<series_version>:<arch>
51-
52- For Example:
53-
54- com.ubuntu.cloud:server:14.04:amd64
55-
56- Non-released images (eg beta, daily etc) have product ids like:
57-
58- com.ubuntu.cloud.daily:server:13.10:amd64
59-
60- The metadata index and product files are required to be in the following
61- directory tree (relative to the URL associated with each path component):
62-
63- ``` no-highlight
64- <path_url>
65- |-streams
66- |-v1
67- |-index.(s)json
68- |-product-foo.(s)json
69- |-product-bar.(s)json
70- ```
71-
72- The index file must be called "index.(s)json" (sjson for signed). The various
73- product files are named according to the Path values contained in the index
74- file.
75-
76- # Tools Metadata Contents
77-
78- Tools metadata uses a simplestreams content type of "content-download". The
79- product id is formed as follows:
80-
81- "com.ubuntu.juju:<series_version>:<arch>"
82-
83- For example:
84-
85- "com.ubuntu.juju:12.04:amd64"
86-
87- The metadata index and product files are required to be in the following
88- directory tree (relative to the URL associated with each path component). In
89- addition, tools tarballs which Juju needs to download are also expected.
90-
91- ``` no-highlight
92- |-streams
93- | |-v1
94- | |-index.(s)json
95- | |-product-foo.(s)json
96- | |-product-bar.(s)json
97- |
98- |-releases
99- |-tools-abc.tar.gz
100- |-tools-def.tar.gz
101- |-tools-xyz.tar.gz
102- ```
103-
104- The index file must be called "index.(s)json" (sjson for signed). The product
105- file and tools tarball name(s) match whatever is in the index/product files.
106-
107-
108- ## Configuration
109-
110- For supported public clouds, no extra configuration is required; things work
111- out-of-the-box. However, for testing purposes, or for non-supported cloud
112- deployments, Juju needs to know where to find the tools and which image to run.
113- Even for supported public clouds where all required metadata is available, the
114- user can put their own metadata in the search path to override what is provided
115- by the cloud.
116-
117- ### User specified URLs
118-
119- These are initially specified in the environments.yaml file (and then
120- subsequently copied to the jenv file when the environment is bootstrapped). For
121- images, use "image-metadata-url"; for tools, use "tools-metadata-url". The URLs
122- can point to a world readable container/bucket in the cloud, an address served
123- by a http server, or even a shared directory which is accessible by all node
124- instances running in the cloud.
125-
126- Assume an Apache http server with base URL ` https://juju-metadata ` , providing
127- access to information at ` <base>/images ` and ` <base>/tools ` . The Juju
128- environments yaml file could have the following entries (one or both):
129-
130- ``` yaml
131- tools-metadata-url : https://juju-metadata/tools
132- image-metadata-url : https://juju-metadata/images
133- ` ` `
134-
135- The required files in each location is as per the directory layout described
136- earlier. For a shared directory, use a URL of the form "file:///sharedpath".
137-
138- ### Cloud storage
139-
140- If no matching metadata is found in the user specified URL, environment's cloud
141- storage is searched. No user configuration is required here - all Juju
142- environments are set up with cloud storage which is used to store state
143- information, charms etc. Cloud storage setup is provider dependent; for Amazon
144- and Openstack clouds, the storage is defined by the "control-bucket" value, for
145- Azure, the "storage-account-name" value is relevant.
146-
147- The (optional) directory structure inside the cloud storage is as follows:
148-
149- ` ` ` no-highlight
150- |-tools
151- | |-streams
152- | |-v1
153- | |-releases
154- |
155- |-images
156- |-streams
157- |-v1
158- ```
159-
160- Of course, if only custom image metadata is required, the tools directory will
161- not be required, and vice versa.
162-
163- Note that if juju bootstrap is run with the ` --upload-tools ` option, the tools
164- and metadata are placed according to the above structure. That's why the tools
165- are then available for Juju to use.
166-
167-
168- ### Provider specific storage
169-
170- Providers may allow additional locations to search for metadata and tools. For
171- OpenStack, Keystone endpoints may be created by the cloud administrator. These
172- are defined as follows:
173-
174- ** juju-tools**
175- The ` <path_url> ` value as described above in Tools Metadata Contents
176-
177- ** product-streams**
178- The ` <path_url> ` value as described above in Image Metadata Contents
179-
180- Other providers may similarly be able to specify locations, though the
181- implementation will vary.
182-
183- ### Central web location
184- ([ https://streams.canonical.com ] ( https://streams.canonical.com ) )
185-
186- This is the default location used to search for tools metadata and is
187- used if no matches are found earlier in any of the above locations. No user
188- configuration is required.
189-
190- ([ http://cloud-images.ubuntu.com ] ( http://cloud-images.ubuntu.com ) )
191-
192- This is the default location used to search for image metadata and is
193- used if no matches are found earlier in any of the above locations. No user
194- configuration is required.
195-
196-
197- # Deploying Private Clouds
198-
199- There are two main issues when deploying a private cloud:
200-
201- 1 . Image ids will be specific to the cloud.
202- 2 . Often, outside internet access is blocked
203-
204- Issue 1 means that image id metadata needs to be generated and made available.
205-
206- Issue 2 means that tools need to be mirrored locally to make them accessible.
207-
208- Juju tools exist to help with generating and validating image and tools
209- metadata. For tools, it is often easiest to just mirror
210- ` https://streams.canonical.com/juju/tools ` . However image metadata cannot be simply
211- mirrored because the image ids are taken from the cloud storage provider, so
212- this needs to be generated and validated using the commands described below.
213-
214- The available Juju metadata tools can be seen by using the help command:
215-
216- ``` bash
217- juju help metadata
218- ```
219-
220- The overall workflow is:
221-
222- - Generate image metadata
223- - Copy image metadata to somewhere in the metadata search path
224- - Optionally, mirror tools to somewhere in the metadata search path
225- - Optionally, configure tools-metadata-url and/or image-metadata-url
226-
227- ### Image metadata
228-
229- Generate image metadata using
230-
231- ``` bash
232- juju metadata generate-image -d < metadata_dir>
233- ```
234-
235- As a minimum, the above command needs to know the image id to use and a
236- directory in which to write the files.
237-
238- Other required parameters like region, series, architecture etc. are taken from
239- the current Juju environment (or an environment specified with the -e option).
240- These parameters can also be overridden on the command line.
241-
242- The image metadata command can be run multiple times with different regions,
243- series, architecture, and it will keep adding to the metadata files. Once all
244- required image ids have been added, the index and product json files can be
245- uploaded to a location in the Juju metadata search path. As per the
246- Configuration section, this may be somewhere specified by the `image-metadata-
247- url` setting or the cloud's storage etc.
248-
249- Examples:
250-
251- 1 . image-metadata-url
252-
253- - upload contents of to ` http://somelocation `
254- - set image-metadata-url to ` http://somelocation/images `
255-
256- 2 . Cloud storage
257-
258- - Upload contents of directly to environment's cloud storage
259- - Use the validation command to ensure an image id can be discovered for a given scenario (region series, arch):
260-
261- ``` bash
262- juju metadata validate-images
263- ```
264-
265- If run without parameters, the validation command will take all required details
266- from the current Juju environment (or as specified by -e) and output the image
267- id it would use to spin up an instance. Alternatively, series, region,
268- architecture etc. can be specified on the command line to override the values in
269- the environment config.
270-
271-
272- ### Tools metadata
273-
274- Generally, tools and related metadata are mirrored from
275- ` https://streams.canonical.com/juju/tools ` . However, it is possible to manually
276- generate metadata for a custom built tools tarball.
277-
278- First, create a tarball of the relevant tools and place in a directory
279- structured like this:
280-
281- <tools_dir>/tools/releases/
282-
283- Now generate relevant metadata for the tools by running the command:
284-
285- ``` bash
286- juju metadata generate-tools -d < tools_dir>
287- ```
288-
289- Finally, the contents of ` <tools_dir> ` can be uploaded to a location in the Juju metadata
290- search path. As per the Configuration section, this may be somewhere specified
291- by the tools-metadata-url setting or the cloud's storage path settings etc.
292-
293- Examples:
294-
295- 1 . tools-metadata-url
296-
297- - upload contents of ` <tools_dir> ` to ` http://somelocation `
298- - set tools-metadata-url to ` http://somelocation/tools `
299-
300- 2 . Cloud storage
301-
302- - upload contents of ` <tools_dir> ` directly to environment's cloud storage
303-
304- As with image metadata, the validation command is used to ensure tools are
305- available for Juju to use:
306-
307- ``` bash
308- juju metadata validate-tools
309- ```
310-
311- The same comments apply. Run the validation tool without parameters to use
312- details from the Juju environment, or override values as required on the command
313- line. See ` juju help metadata validate-tools ` for more details.
5+ !!! Note: This page is currently being rewritten for 2.0
0 commit comments