+$ python tf_upgrade.py --infile InputFile --outfile OutputFile ++ +For example, the following command converts a 0.n TensorFlow +program named `test.py` to a 1.0 TensorFlow program named `test_1.0.py`: + +
+$ python tf_upgrade.py --infile test.py --outfile test_1.0.py ++ +The `tf_upgrade.py` script also generates a file named `report.txt`, which +details all the changes it performed and makes additional suggestions about +changes you might need to make manually. + +To upgrade a whole directory of 0.n TensorFlow programs to 1.0, +enter a command having the following format: + +
+$ python tf_upgrade.py --intree InputDir --outtree OutputDir ++ +For example, the following command converts all the 0.n TensorFlow programs +in the `/home/user/cool` directory, creating their 1.0 equivalents in +the `/home/user/cool_1.0` directory: + +
+$ python tf_upgrade.py --intree /home/user/cool --outtree /home/user/cool_1.0 ++ +### Limitations + +There are a few things to watch out for. Specifically: + + * You must manually fix any instances of `tf.reverse()`. + The `tf_upgrade.py` script will warn you about `tf.reverse()` in + stdout and in the `report.txt` file. + * On reordered arguments, `tf_upgrade.py` tries to minimally reformat + your code, so it cannot automatically change the actual argument order. + Instead, `tf_upgrade.py` makes your function invocations order-independent + by introducing keyword arguments. + * Constructions like `tf.get_variable_scope().reuse_variables()` + will likely not work. We recommend deleting those lines and replacing + them with lines such as the following: + +
+ with tf.variable_scope(tf.get_variable_scope(), reuse=True): + ... ++ + * Analogously to `tf.pack` and `tf.unpack`, we're renamed + `TensorArray.pack` and `TensorArray.unpack` to + `TensorArray.stack` and `TensorArray.unstack`. However, `TensorArray.pack` + and `TensorArray.unpack` cannot be detected lexically since they are + indirectly related to the `tf` namespace e.g. + `foo = tf.TensorArray(); foo.unpack()` + +## Upgrading your code manually + +Instead of running `tf_upgrade.py`, you may manually upgrade your code. +The remainder of this document provides a comprehensive list of +all backward incompatible changes made in TensorFlow 1.0. + + +### Variables + +Variable functions have been made more consistent and less confusing. + +* `tf.VARIABLES` + * should be renamed to `tf.GLOBAL_VARIABLES` +* `tf.all_variables` + * should be renamed to `tf.global_variables` +* `tf.initialize_all_variables` + * should be renamed to `tf.global_variables_initializer` +* `tf.initialize_local_variables` + * should be renamed to `tf.local_variables_initializer` +* `tf.initialize_variables` + * should be renamed to `tf.variables_initializer` + +### Summary functions + +Summary functions have been consolidated under the `tf.summary` namespace. + +* `tf.audio_summary` + * should be renamed to `tf.summary.audio` +* `tf.contrib.deprecated.histogram_summary` + * should be renamed to `tf.summary.histogram` +* `tf.contrib.deprecated.scalar_summary` + * should be renamed to `tf.summary.scalar` +* `tf.histogram_summary` + * should be renamed to `tf.summary.histogram` +* `tf.image_summary` + * should be renamed to `tf.summary.image` +* `tf.merge_all_summaries` + * should be renamed to `tf.summary.merge_all` +* `tf.merge_summary` + * should be renamed to `tf.summary.merge` +* `tf.scalar_summary` + * should be renamed to `tf.summary.scalar` +* `tf.train.SummaryWriter` + * should be renamed to `tf.summary.FileWriter` + +### Numeric differences + + +Integer division and `tf.floordiv` now uses flooring semantics. This is to +make the results of `np.divide` and `np.mod` consistent with `tf.divide` and +`tf.mod`, respectively. In addition we have changed the rounding algorithm +used by `tf.round` to match NumPy. + + +* `tf.div` + + * The semantics of `tf.divide` division have been changed to match Python +semantics completely. That is, `/` in Python 3 and future division mode in +Python 2 will produce floating point numbers always, `//` will produce floored +division. However, even `tf.div` will produce floored integer division. +To force C-style truncation semantics, you must use `tf.truncatediv`. + + * Consider changing your code to use `tf.divide`, which follows Python semantics for promotion. + +* `tf.mod` + + * The semantics of `tf.mod` have been changed to match Python semantics. In +particular, flooring semantics are used for integers. If you wish to have +C-style truncation mod (remainders), you can use `tf.truncatemod` + + +The old and new behavior of division can be summarized with this table: + +| Expr | TF 0.11 (py2) | TF 0.11 (py3) | TF 1.0 (py2) | TF 1.0 (py3) | +|---------------------|---------------|---------------|--------------|--------------| +| tf.div(3,4) | 0 | 0 | 0 | 0 | +| tf.div(-3,4) | 0 | 0 | -1 | -1 | +| tf.mod(-3,4) | -3 | -3 | 1 | 1 | +| -3/4 | 0 | -0.75 | -1 | -0.75 | +| -3/4tf.divide(-3,4) | N/A | N/A | -0.75 | -1 | + +The old and new behavior of rounding can be summarized with this table: + +| Input | Python | NumPy | C++ round() | TensorFlow 0.11(floor(x+.5)) | TensorFlow 1.0 | +|-------|--------|-------|-------------|------------------------------|----------------| +| -3.5 | -4 | -4 | -4 | -3 | -4 | +| -2.5 | -2 | -2 | -3 | -2 | -2 | +| -1.5 | -2 | -2 | -2 | -1 | -2 | +| -0.5 | 0 | 0 | -1 | 0 | 0 | +| 0.5 | 0 | 0 | 1 | 1 | 0 | +| 1.5 | 2 | 2 | 2 | 2 | 2 | +| 2.5 | 2 | 2 | 3 | 3 | 2 | +| 3.5 | 4 | 4 | 4 | 4 | 4 | + + + +### NumPy matching names + + +Many functions have been renamed to match NumPy. This was done to make the +transition between NumPy and TensorFlow as easy as possible. There are still +numerous cases where functions do not match, so this is far from a hard and +fast rule, but we have removed several commonly noticed inconsistencies. + +* `tf.inv` + * should be renamed to `tf.reciprocal` + * This was done to avoid confusion with NumPy's matrix inverse `np.inv` +* `tf.list_diff` + * should be renamed to `tf.setdiff1d` +* `tf.listdiff` + * should be renamed to `tf.setdiff1d` +* `tf.mul` + * should be renamed to `tf.multiply` +* `tf.neg` + * should be renamed to `tf.negative` +* `tf.select` + * should be renamed to `tf.where` + * `tf.where` now takes 3 arguments or 1 argument, just like `np.where` +* `tf.sub` + * should be renamed to `tf.subtract` + +### NumPy matching arguments + +Arguments for certain TensorFlow 1.0 methods now match arguments in certain +NumPy methods. To achieve this, TensorFlow 1.0 has changed keyword arguments +and reordered some arguments. Notably, TensorFlow 1.0 now uses `axis` rather +than `dimension`. TensorFlow 1.0 aims to keep the tensor argument first on +operations that modify Tensors. (see the `tf.concat` change). + + +* `tf.argmax` + * keyword argument `dimension` should be renamed to `axis` +* `tf.argmin` + * keyword argument `dimension` should be renamed to `axis` +* `tf.concat` + * keyword argument `concat_dim` should be renamed to `axis` + * arguments have been reordered to `tf.concat(values, axis, name='concat')`. +* `tf.count_nonzero` + * keyword argument `reduction_indices` should be renamed to `axis` +* `tf.expand_dims` + * keyword argument `dim` should be renamed to `axis` +* `tf.reduce_all` + * keyword argument `reduction_indices` should be renamed to `axis` +* `tf.reduce_any` + * keyword argument `reduction_indices` should be renamed to `axis` +* `tf.reduce_join` + * keyword argument `reduction_indices` should be renamed to `axis` +* `tf.reduce_logsumexp` + * keyword argument `reduction_indices` should be renamed to `axis` +* `tf.reduce_max` + * keyword argument `reduction_indices` should be renamed to `axis` +* `tf.reduce_mean` + * keyword argument `reduction_indices` should be renamed to `axis` +* `tf.reduce_min` + * keyword argument `reduction_indices` should be renamed to `axis` +* `tf.reduce_prod` + * keyword argument `reduction_indices` should be renamed to `axis` +* `tf.reduce_sum` + * keyword argument `reduction_indices` should be renamed to `axis` +* `tf.reverse` + * `tf.reverse` used to take a 1D `bool` tensor to control which dimensions were reversed. Now we use a Tensor of axis indices. + * For example `tf.reverse(a, [True, False, True])` now must be `tf.reverse(a, [0, 2])` +* `tf.reverse_sequence` + * keyword argument `batch_dim` should be renamed to `batch_axis` + * keyword argument `seq_dim` should be renamed to `seq_axis` +* `tf.sparse_concat` + * keyword argument `concat_dim` should be renamed to `axis` +* `tf.sparse_reduce_sum` + * keyword argument `reduction_axes` should be renamed to `axis` +* `tf.sparse_reduce_sum_sparse` + * keyword argument `reduction_axes` should be renamed to `axis` +* `tf.sparse_split` + * keyword argument `split_dim` should be renamed to `axis` + * arguments have been reordered to `tf.sparse_split(keyword_required=KeywordRequired(), sp_input=None, num_split=None, axis=None, name=None, split_dim=None)`. +* `tf.split` + * keyword argument `split_dim` should be renamed to `axis` + * keyword argument `num_split` should be renamed to `num_or_size_splits` + * arguments have been reordered to `tf.split(value, num_or_size_splits, axis=0, num=None, name='split')`. +* `tf.squeeze` + * keyword argument `squeeze_dims` should be renamed to `axis` +* `tf.svd` + * arguments have been reordered to `tf.svd(tensor, full_matrices=False, compute_uv=True, name=None)`. + +### Simplified math variants + +Batched versions of math operations have been removed. Now the functionality is +contained in the non-batched versions. Similarly,`tf.complex_abs` has had its +functionality moved to `tf.abs` + +* `tf.batch_band_part` + * should be renamed to `tf.band_part` +* `tf.batch_cholesky` + * should be renamed to `tf.cholesky` +* `tf.batch_cholesky_solve` + * should be renamed to `tf.cholesky_solve` +* `tf.batch_fft` + * should be renamed to `tf.fft` +* `tf.batch_fft3d` + * should be renamed to `tf.fft3d` +* `tf.batch_ifft` + * should be renamed to `tf.ifft` +* `tf.batch_ifft2d` + * should be renamed to `tf.ifft2d` +* `tf.batch_ifft3d` + * should be renamed to `tf.ifft3d` +* `tf.batch_matmul` + * should be renamed to `tf.matmul` +* `tf.batch_matrix_determinant` + * should be renamed to `tf.matrix_determinant` +* `tf.batch_matrix_diag` + * should be renamed to `tf.matrix_diag` +* `tf.batch_matrix_inverse` + * should be renamed to `tf.matrix_inverse` +* `tf.batch_matrix_solve` + * should be renamed to `tf.matrix_solve` +* `tf.batch_matrix_solve_ls` + * should be renamed to `tf.matrix_solve_ls` +* `tf.batch_matrix_transpose` + * should be renamed to `tf.matrix_transpose` +* `tf.batch_matrix_triangular_solve` + * should be renamed to `tf.matrix_triangular_solve` +* `tf.batch_self_adjoint_eig` + * should be renamed to `tf.self_adjoint_eig` +* `tf.batch_self_adjoint_eigvals` + * should be renamed to `tf.self_adjoint_eigvals` +* `tf.batch_set_diag` + * should be renamed to `tf.set_diag` +* `tf.batch_svd` + * should be renamed to `tf.svd` +* `tf.complex_abs` + * should be renamed to `tf.abs` + +### Misc Changes + +Several other changes have been made, including the following: + +* `tf.image.per_image_whitening` + * should be renamed to `tf.image.per_image_standardization` +* `tf.nn.sigmoid_cross_entropy_with_logits` + * arguments have been reordered to `tf.nn.sigmoid_cross_entropy_with_logits(_sentinel=None, labels=None, logits=None, name=None)`. +* `tf.nn.softmax_cross_entropy_with_logits` + * arguments have been reordered to `tf.nn.softmax_cross_entropy_with_logits(_sentinel=None, labels=None, logits=None, dim=-1, name=None)`. +* `tf.nn.sparse_softmax_cross_entropy_with_logits` + * arguments have been reordered to `tf.nn.sparse_softmax_cross_entropy_with_logits(_sentinel=None, labels=None, logits=None, name=None)`. +* `tf.ones_initializer` + * should be changed to a function call i.e. `tf.ones_initializer()` +* `tf.pack` + * should be renamed to `tf.stack` +* `tf.round` + * The semantics of `tf.round` now match Banker's rounding. +* `tf.unpack` + * should be renamed to `tf.unstack` +* `tf.zeros_initializer` + * should be changed to a function call i.e. `tf.zeros_initializer()` diff --git a/community/contributing.md b/community/contributing.md new file mode 100644 index 0000000..afbb8bb --- /dev/null +++ b/community/contributing.md @@ -0,0 +1,49 @@ +# Contributing to TensorFlow + +TensorFlow is an open-source project, and we welcome your participation +and contribution. This page describes how to get involved. + +## Repositories + +The code for TensorFlow is hosted in the [TensorFlow GitHub +organization](https://github.com/tensorflow). Multiple projects are located +inside the organization, including: + +* [TensorFlow](https://github.com/tensorflow/tensorflow) +* [Models](https://github.com/tensorflow/models) +* [TensorBoard](https://github.com/tensorflow/tensorboard) +* [TensorFlow.js](https://github.com/tensorflow/tfjs) +* [TensorFlow Serving](https://github.com/tensorflow/serving) +* [TensorFlow Documentation](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/docs_src) + +## Contributor checklist + +* Before contributing to TensorFlow source code, please review the [contribution +guidelines](https://github.com/tensorflow/tensorflow/blob/master/CONTRIBUTING.md). + +* Join the +[developers@tensorflow.org](https://groups.google.com/a/tensorflow.org/d/forum/developers) +mailing list, to coordinate and discuss with others contributing to TensorFlow. + +* For coding style conventions, read the @{$style_guide$TensorFlow Style Guide}. + +* Finally, review @{$documentation$Writing TensorFlow Documentation}, which + explains documentation conventions. + +You may also wish to review our guide to @{$benchmarks$defining and running benchmarks}. + +## Special Interest Groups + +To enable focused collaboration on particular areas of TensorFlow, we host +Special Interest Groups (SIGs). SIGs do their work in public: if you want to +join and contribute, review the work of the group, and get in touch with the +relevant SIG leader. Membership policies vary on a per-SIG basis. + +* **SIG Build** focuses on issues surrounding building, packaging, and + distribution of TensorFlow. [Mailing list](https://groups.google.com/a/tensorflow.org/d/forum/build). + +* **SIG TensorBoard** furthers the development and direction of TensorBoard and its plugins. + [Mailing list](https://groups.google.com/a/tensorflow.org/d/forum/sig-tensorboard). + +* **SIG Rust** collaborates on the development of TensorFlow's Rust bindings. + [Mailing list](https://groups.google.com/a/tensorflow.org/d/forum/rust). diff --git a/community/documentation.md b/community/documentation.md index 003e0a2..d5bc7a5 100644 --- a/community/documentation.md +++ b/community/documentation.md @@ -148,19 +148,7 @@ viewing. Do not include url parameters in the source code URL. Before building the documentation, you must first set up your environment by doing the following: -1. If pip isn't installed on your machine, install it now by issuing the -following command: - - $ sudo easy_install pip - -2. Use pip to install codegen, mock, and pandas by issuing the following - command (Note: If you are using - a [virtualenv](https://virtualenv.pypa.io/en/stable/) to manage your - dependencies, you may not want to use sudo for these installations): - - $ sudo pip install codegen mock pandas - -3. If bazel is not installed on your machine, install it now. If you are on +1. If bazel is not installed on your machine, install it now. If you are on Linux, install bazel by issuing the following command: $ sudo apt-get install bazel # Linux @@ -168,10 +156,10 @@ following command: If you are on Mac OS, find bazel installation instructions on [this page](https://bazel.build/versions/master/docs/install.html#mac-os-x). -4. Change directory to the top-level `tensorflow` directory of the TensorFlow +2. Change directory to the top-level `tensorflow` directory of the TensorFlow source code. -5. Run the `configure` script and answer its prompts appropriately for your +3. Run the `configure` script and answer its prompts appropriately for your system. $ ./configure @@ -477,31 +465,29 @@ should use Markdown in the docstring. Here's a simple example: -```python -def foo(x, y, name="bar"): - """Computes foo. + def foo(x, y, name="bar"): + """Computes foo. - Given two 1-D tensors `x` and `y`, this operation computes the foo. + Given two 1-D tensors `x` and `y`, this operation computes the foo. - Example: + Example: - ``` - # x is [1, 1] - # y is [2, 2] - tf.foo(x, y) ==> [3, 3] - ``` - Args: - x: A `Tensor` of type `int32`. - y: A `Tensor` of type `int32`. - name: A name for the operation (optional). + ``` + # x is [1, 1] + # y is [2, 2] + tf.foo(x, y) ==> [3, 3] + ``` + Args: + x: A `Tensor` of type `int32`. + y: A `Tensor` of type `int32`. + name: A name for the operation (optional). - Returns: - A `Tensor` of type `int32` that is the foo of `x` and `y`. + Returns: + A `Tensor` of type `int32` that is the foo of `x` and `y`. - Raises: - ValueError: If `x` or `y` are not of type `int32`. - """ -``` + Raises: + ValueError: If `x` or `y` are not of type `int32`. + """ ## Description of the docstring sections diff --git a/community/groups.md b/community/groups.md new file mode 100644 index 0000000..d92f577 --- /dev/null +++ b/community/groups.md @@ -0,0 +1,17 @@ +# User Groups + +TensorFlow has communities around the world. + +## Asia + +* [TensorFlow Korea (TF-KR) User Group](https://www.facebook.com/groups/TensorFlowKR/) _(Korean language)_ +* [TensorFlow User Group Tokyo](https://tfug-tokyo.connpass.com/) _(Japanese Language)_ +* [Soleil Data Dojo](https://soleildatadojo.connpass.com/) _(Japanese language)_ +* [TensorFlow User Group Utsunomiya](https://tfug-utsunomiya.connpass.com/) + + +## Europe + +* [TensorFlow Barcelona](https://www.meetup.com/Barcelona-Machine-Learning-Meetup/) +* [TensorFlow Madrid](https://www.meetup.com/TensorFlow-Madrid/) + diff --git a/community/index.md b/community/index.md index 8e67022..eec2e51 100644 --- a/community/index.md +++ b/community/index.md @@ -1,14 +1,85 @@ # Community -This section contains the following documents: - - * @{$welcome$Welcome to the TensorFlow Community}, which explains how - you can get involved, where to report issues, and where to join - like-minded TensorFlow enthusiasts online. - * @{$documentation$Writing TensorFlow Documentation}, which explains - TensorFlow's documentation conventions. If you are modifying - TensorFlow source code or documentation, please read this guide. - * @{$style_guide$TensorFlow Style Guide}, which identifies coding style - conventions that TensorFlow developers and users should follow. - * @{$community/benchmarks$Benchmarks}, Benchmarks, a guide for defining and - running a TensorFlow benchmark. +Welcome to the TensorFlow community! This page explains where to get help, and +different ways to be part of the community. We are committed to fostering an +open and welcoming environment, and request that you review our [code of +conduct](https://github.com/tensorflow/tensorflow/blob/master/CODE_OF_CONDUCT.md). + +## Get Help + +### Technical Questions + +To ask or answer technical questions about TensorFlow, use [Stack +Overflow](https://stackoverflow.com/questions/tagged/tensorflow). For example, +ask or search about a particular error message you encountered during +installation. + +### Bugs and Feature Requests + +To report bugs or make feature requests, file an issue on GitHub. Please choose +the appropriate repository for the project. Major repositories include: + + * [TensorFlow](https://github.com/tensorflow/tensorflow/issues) + * [TensorBoard](https://github.com/tensorflow/tensorboard/issues) + * [TensorFlow models](https://github.com/tensorflow/models/issues) + +### Security + +Before using TensorFlow, please take a look at our security model, list of +recent security announcements, and ways you can report security issues to the +TensorFlow team at the +[Using TensorFlow Securely](https://github.com/tensorflow/tensorflow/blob/master/SECURITY.md) page on GitHub. + +## Stay Informed + +### Announcements Mailing List + +All major releases and important announcements are sent to +[announce@tensorflow.org](https://groups.google.com/a/tensorflow.org/forum/#!forum/announce). +We recommend that you join this list if you depend on TensorFlow in any way. + +### Development Roadmap + +The @{$roadmap$Roadmap} summarizes plans for upcoming additions to TensorFlow. + +### Social Media + +For news and updates from around the universe of TensorFlow projects, follow +[@tensorflow](https://twitter.com/tensorflow) on Twitter. + +### Blog + +We post regularly to the [TensorFlow Blog](http://blog.tensorflow.org/), +with content from the TensorFlow team and the best articles from the community. + +### YouTube + +Our [YouTube Channel](http://youtube.com/tensorflow/) focuses on machine learing +and AI with TensorFlow. On it we have a number of new shows, including: + +- TensorFlow Meets: meet with community contributors to learn and share what they're doing +- Ask TensorFlow: the team answers the best questions tagged #AskTensorFlow from social media +- Coding TensorFlow: short bites with tips for success with TensorFlow + +## Community Support + +### Mailing Lists + +For general discussion about TensorFlow development and direction, please join +the [TensorFlow discuss mailing +list](https://groups.google.com/a/tensorflow.org/d/forum/discuss). + +A number of other mailing lists exist, focused on different project areas, which +can be found at @{$lists$TensorFlow Mailing Lists}. + +### User Groups + +To meet with like-minded people local to you, check out the many +@{$groups$TensorFlow user groups} around the world. + + +## Contributing To TensorFlow + +We welcome contributions and collaboration on TensorFlow. For more information, +please read [Contributing to TensorFlow](contributing.md). + diff --git a/community/leftnav_files b/community/leftnav_files index c1595d3..2bae60d 100644 --- a/community/leftnav_files +++ b/community/leftnav_files @@ -1,5 +1,9 @@ index.md -welcome.md +roadmap.md +contributing.md +lists.md +groups.md documentation.md style_guide.md benchmarks.md +swift.md diff --git a/community/lists.md b/community/lists.md new file mode 100644 index 0000000..7450ab3 --- /dev/null +++ b/community/lists.md @@ -0,0 +1,51 @@ +# Mailing Lists + +As a community, we do much of our collaboration on public mailing lists. +Please note that if you're looking for help using TensorFlow, [Stack +Overflow](https://stackoverflow.com/questions/tagged/tensorflow) and +[GitHub issues](https://github.com/tensorflow/tensorflow/issues) +are the best initial places to look. For more information, +see [how to get help](/community/#get_help). + +## General TensorFlow lists + +* [announce](https://groups.google.com/a/tensorflow.org/d/forum/announce) - Low-volume announcements of new releases. +* [discuss](https://groups.google.com/a/tensorflow.org/d/forum/discuss) - General community discussion around TensorFlow. +* [developers](https://groups.google.com/a/tensorflow.org/d/forum/developers) - Discussion for developers contributing to TensorFlow. + +## Project-specific lists + +These projects inside the TensorFlow GitHub organization have lists dedicated to their communities: + +* [hub](https://groups.google.com/a/tensorflow.org/d/forum/hub) - + Discussion and collaboration around [TensorFlow Hub](https://github.com/tensorflow/hub). +* [magenta-discuss](https://groups.google.com/a/tensorflow.org/d/forum/magenta-discuss) - + General discussion about [Magenta](https://magenta.tensorflow.org/) + development and directions. +* [swift](https://groups.google.com/a/tensorflow.org/d/forum/swift) - + Community and collaboration around Swift for TensorFlow. +* [tensor2tensor](https://groups.google.com/d/forum/tensor2tensor) - Discussion + and peer support for Tensor2Tensor. +* [tfjs-announce](https://groups.google.com/a/tensorflow.org/d/forum/tfjs-announce) - + Announcements of new TensorFlow.js releases. +* [tfjs](https://groups.google.com/a/tensorflow.org/d/forum/tfjs) - Discussion + and peer support for TensorFlow.js. +* [tflite](https://groups.google.com/a/tensorflow.org/d/forum/tflite) - Discussion and + peer support for TensorFlow Lite. +* [tpu-users](https://groups.google.com/a/tensorflow.org/d/forum/tpu-users) - Community discussion + and support for TPU users. + +## Special Interest Groups + +TensorFlow's [Special Interest +Groups](/community/contributing#special_interest_groups) (SIGs) support +community collaboration on particular project focuses. Members of these groups +work together to build and support TensorFlow related projects. While their +archives are public, different SIGs have their own membership policies. + +* [build](https://groups.google.com/a/tensorflow.org/d/forum/build) - + Supporting SIG Build, for build, distribution and packaging of TensorFlow. +* [sig-tensorboard](https://groups.google.com/a/tensorflow.org/d/forum/sig-tensorboard) - + Supporting SIG TensorBoard, for plugin development and other contribution. +* [rust](https://groups.google.com/a/tensorflow.org/d/forum/rust) - + Supporting SIG Rust, for the Rust language bindings. diff --git a/community/roadmap.md b/community/roadmap.md new file mode 100644 index 0000000..a3170a1 --- /dev/null +++ b/community/roadmap.md @@ -0,0 +1,85 @@ +# Roadmap +**Last updated: Feb 15, 2018** + +TensorFlow is a rapidly moving, community supported project. This document is intended +to provide guidance about priorities and focus areas of the core set of TensorFlow +developers and about functionality that can be expected in the upcoming releases of +TensorFlow. Many of these areas are driven by community use cases, and we welcome +further +[contributions](https://github.com/tensorflow/tensorflow/blob/master/CONTRIBUTING.md) +to TensorFlow. + +The features below do not have concrete release dates. However, the majority can be +expected in the next one to two releases. + +### APIs +#### High Level APIs: +* Easy multi-GPU utilization with Estimators +* Easy-to-use high-level pre-made estimators for Gradient Boosted Trees, Time Series, and other models + +#### Eager Execution: +* Efficient utilization of multiple GPUs +* Distributed training (multi-machine) +* Performance improvements +* Simpler export to a GraphDef/SavedModel + +#### Keras API: +* Better integration with tf.data (ability to call `model.fit` with data tensors) +* Full support for Eager Execution (both Eager support for the regular Keras API, and ability +to create Keras models Eager- style via Model subclassing) +* Better distribution/multi-GPU support and TPU support (including a smoother model-to-estimator workflow) + +#### Official Models: +* A set of +[reference models](https://github.com/tensorflow/models/tree/master/official) +across image recognition, speech, object detection, and + translation that demonstrate best practices and serve as a starting point for + high-performance model development. + +#### Contrib: +* Deprecation notices added to parts of tf.contrib where preferred implementations exist outside of tf.contrib. +* As much as possible, large projects inside tf.contrib moved to separate repositories. +* The tf.contrib module will eventually be discontinued in its current form, experimental development will in future happen in other repositories. + + +#### Probabilistic Reasoning and Statistical Analysis: +* Rich set of tools for probabilistic and statistical analysis in tf.distributions + and tf.probability. These include new samplers, layers, optimizers, losses, and structured models +* Statistical tools for hypothesis testing, convergence diagnostics, and sample statistics +* Edward 2.0: High-level API for probabilistic programming + +### Platforms +#### TensorFlow Lite: +* Increased coverage of supported ops in TensorFlow Lite +* Easier conversion of a trained TensorFlow graph for use on TensorFlow Lite +* Support for GPU acceleration in TensorFlow Lite (iOS and Android) +* Support for hardware accelerators via Android NeuralNets API +* Improved CPU performance by quantization and other network optimizations (eg. pruning, distillation) +* Increased support for devices beyond Android and iOS (eg. RPi, Cortex-M) + +### Performance +#### Distributed TensorFlow: +* Multi-GPU support optimized for a variety of GPU topologies +* Improved mechanisms for distributing computations on several machines + +#### Optimizations: +* Mixed precision training support with initial example model and guide +* Native TensorRT support +* Int8 support for SkyLake via MKL +* Dynamic loading of SIMD-optimized kernels + +### Documentation and Usability: +* Updated documentation, tutorials and Getting Started guides +* Process to enable external contributions to tutorials, documentation, and blogs showcasing best practice use-cases of TensorFlow and high-impact applications + +### Community and Partner Engagement +#### Special Interest Groups: +* Mobilizing the community to work together in focused domains +* [tf-distribute](https://groups.google.com/a/tensorflow.org/forum/#!forum/tf-distribute): build and packaging of TensorFlow +* More to be identified and launched + +#### Community: +* Incorporate public feedback on significant design decisions via a Request-for-Comment (RFC) process +* Formalize process for external contributions to land in TensorFlow and associated projects +* Grow global TensorFlow communities and user groups +* Collaborate with partners to co-develop and publish research papers diff --git a/community/security.md b/community/security.md new file mode 100644 index 0000000..8d13c7a --- /dev/null +++ b/community/security.md @@ -0,0 +1,7 @@ +# Using TensorFlow Securely + +Before using TensorFlow, please take a look at our security model, list of +recent security announcements, and ways you can report security issues to the +TensorFlow team at the +[https://github.com/tensorflow/tensorflow/blob/master/SECURITY.md](Using +TensorFlow Securely) page on GitHub. diff --git a/community/swift.md b/community/swift.md new file mode 100644 index 0000000..54d9960 --- /dev/null +++ b/community/swift.md @@ -0,0 +1,35 @@ +# Swift Community + +Welcome to the Swift for TensorFlow development community! + +Swift for TensorFlow is a result of first-principles thinking applied to machine +learning frameworks, and works quite differently than existing TensorFlow +language bindings. Whereas prior solutions are designed within the constraints +of what can be achieved by a (typically Python or Lua) library, Swift for +TensorFlow is based on the belief that machine learning is important enough to +deserve first-class language and compiler support. + +First-class language and compiler support allows us to innovate in areas that +have traditionally been out of bounds for machine learning libraries. Our +results provide the performance of TensorFlow graphs with the ease of use of +define-by-run models, and provides a great user experience - for example, by +catching more mistakes before you run your code. + +## Open Source + +As announced at the TensorFlow Developer Summit, we are planning to launch our +open source project on GitHub in April. In addition to releasing the code, we +will be using an open design model, where design discussions happen in public. + +Between now and then, we are writing some technical white papers that explain in +detail the design approach (e.g., the core compiler partitioning technique that +underlies the whole thing, our approach to automatic differentiation, etc.), +implementation tradeoffs, and the status of this work. We can’t wait to engage +with the broader community, but prefer to start the conversation when these +white papers are ready. + +[Sign up here to join the community Google +group](https://groups.google.com/a/tensorflow.org/d/forum/swift). We will +initially use it for announcements, and then open it for general discussion when +we are ready in April. + diff --git a/community/welcome.md b/community/welcome.md deleted file mode 100644 index d2d3f9e..0000000 --- a/community/welcome.md +++ /dev/null @@ -1,69 +0,0 @@ -# Welcome to the TensorFlow Community - -TensorFlow is an open-source project. This page explains how to contribute, -where to ask questions, and how to help each other. - - -## Development - -The source code for TensorFlow is on -[GitHub](https://github.com/tensorflow/tensorflow). - -Before contributing to TensorFlow source code, please review the -[Contribution guidelines](https://github.com/tensorflow/tensorflow/blob/master/CONTRIBUTING.md). - -### Projects developed by the TensorFlow community - -The TensorFlow community has created many great projects around TensorFlow, including: - -* [Machine Learning with TensorFlow (Book & Code)](http://tensorflowbook.com) -* [@jtoy's awesome "Awesome TensorFlow" list of awesome things](https://github.com/jtoy/awesome-tensorflow) -* [TensorFlow tutorials](https://github.com/pkmital/tensorflow_tutorials) -* [Caffe to TensorFlow model converter](https://github.com/ethereon/caffe-tensorflow) -* [Bitfusion's` GPU-enabled AWS EC2 TensorFlow AMI](https://github.com/bitfusionio/amis/tree/master/awsmrkt-bfboost-ubuntu14-cuda75-tensorflow) ([Launch AMI](https://aws.amazon.com/marketplace/pp/B01EYKBEQ0)) -* [Rust language bindings](https://github.com/google/tensorflow-rust) -* [Operator Vectorization Library](https://github.com/opveclib/opveclib) -* [Swift language bindings](https://github.com/PerfectlySoft/Perfect-TensorFlow) -* [Sublime Tensorflow - A plugin for Sublime Text](https://github.com/baptisteArnaud/Sublime-Tensorflow) -* [Edward - A library for probabilistic modeling, inference, and criticism](http://edwardlib.org) ([Github](https://github.com/blei-lab/edward), [Forum](https://discourse.edwardlib.org)) -* [GPflow - Gaussian processes in TensorFlow](https://github.com/GPflow/GPflow) -* [CS 20SI: Tensorflow for Deep Learning Research](https://web.stanford.edu/class/cs20si/) - Please note, this course was designed with TensorFlow v0.12, so some of the notes may be out of date - but it's still a great resource. - -## TensorFlow Communities Around the World - -Asia: - -* [TensorFlow Korea (TF-KR) User Group](https://www.facebook.com/groups/TensorFlowKR/) _(Korean language)_ -* [TensorFlow User Group Tokyo](https://tfug-tokyo.connpass.com/) _(Japanese Language)_ -* [Soleil Data Dojo](https://soleildatadojo.connpass.com/) _(Japanese language)_ -* [TensorFlow User Group Utsunomiya](https://tfug-utsunomiya.connpass.com/) - - -Europe: - -* [TensorFlow Barcelona](https://www.meetup.com/Barcelona-Machine-Learning-Meetup/) -* [TensorFlow Madrid](https://www.meetup.com/TensorFlow-Madrid/) - - - -## Support - -TensorFlow provides multiple communication paths. To pick the right path, -please read the following list carefully: - - * To ask or answer technical questions about TensorFlow, use - [Stack Overflow](https://stackoverflow.com/questions/tagged/tensorflow). - For example, ask or search Stack Overflow about a particular error message - you encountered during installation. - * To join general discussions about TensorFlow development and directions, - please join the - [TensorFlow discuss mailing list](https://groups.google.com/a/tensorflow.org/d/forum/discuss). - For example, use this mailing list to learn about new features in - upcoming releases of TensorFlow. - * To report bugs or make feature requests, use the - [TensorFlow issues tracker](https://github.com/tensorflow/tensorflow/issues) - on GitHub. For example, use the issue tracker to request a - new operation in TensorFlow. - * To report vulnerabilities, please follow our - [vulnerability disclosure guidelines](https://github.com/tensorflow/tensorflow/blob/master/SECURITY.md). - diff --git a/deploy/deploy_to_js.md b/deploy/deploy_to_js.md new file mode 100644 index 0000000..d7ce3ea --- /dev/null +++ b/deploy/deploy_to_js.md @@ -0,0 +1,4 @@ +# Deploy to JavaScript + +You can find details about deploying JavaScript TensorFlow programs +in the separate [js.tensorflow.org site](https://js.tensorflow.org). diff --git a/deploy/index.md b/deploy/index.md index 5831960..07b1bc9 100644 --- a/deploy/index.md +++ b/deploy/index.md @@ -7,6 +7,8 @@ the following documents: a cluster of TensorFlow servers. * @{$hadoop$How to run TensorFlow on Hadoop}, which has a highly self-explanatory title. + * @{$s3$How to run TensorFlow with the S3 filesystem}, which explains how + to run TensorFlow with the S3 file system. * The entire document set for [TensorFlow serving](/serving), an open-source, flexible, high-performance serving system for machine-learned models designed for production environments. TensorFlow Serving provides diff --git a/deploy/leftnav_files b/deploy/leftnav_files index f8f8d57..93f5bd1 100644 --- a/deploy/leftnav_files +++ b/deploy/leftnav_files @@ -1,3 +1,5 @@ index.md distributed.md hadoop.md +s3.md +deploy_to_js.md diff --git a/deploy/s3.md b/deploy/s3.md new file mode 100644 index 0000000..38f8428 --- /dev/null +++ b/deploy/s3.md @@ -0,0 +1,40 @@ +# How to run TensorFlow on S3 + +This document describes how to run TensorFlow on S3 file system. + +## S3 + +We assume that you are familiar with @{$reading_data$reading data}. + +To use S3 with TensorFlow, change the file paths you use to read and write +data to an S3 path. For example: + +```python +filenames = ["s3://bucketname/path/to/file1.tfrecord", + "s3://bucketname/path/to/file2.tfrecord"] +dataset = tf.data.TFRecordDataset(filenames) +``` + +When reading or writing data on S3 with your TensorFlow program, the behavior +could be controlled by various environmental variables: + +* **AWS_REGION**: By default, regional endpoint is used for S3, with region + controlled by `AWS_REGION`. If `AWS_REGION` is not specified, then + `us-east-1` is used. +* **S3_ENDPOINT**: The endpoint could be overridden explicitly with + `S3_ENDPOINT` specified. +* **S3_USE_HTTPS**: HTTPS is used to access S3 by default, unless + `S3_USE_HTTPS=0`. +* **S3_VERIFY_SSL**: If HTTPS is used, SSL verification could be disabled + with `S3_VERIFY_SSL=0`. + +To read or write objects in a bucket that is no publicly accessible, +AWS credentials must be provided through one of the following methods: + +* Set credentials in the AWS credentials profile file on the local system, + located at: `~/.aws/credentials` on Linux, macOS, or Unix, or + `C:\Users\USERNAME\.aws\credentials` on Windows. +* Set the `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` environment + variables. +* If TensorFlow is deployed on an EC2 instance, specify an IAM role and then + give the EC2 instance access to that role. diff --git a/extend/add_filesys.md b/extend/add_filesys.md index f0591b7..bc0f662 100644 --- a/extend/add_filesys.md +++ b/extend/add_filesys.md @@ -81,6 +81,8 @@ filesystem implementations call their existing libraries. Examples include: plugin](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/core/platform/hadoop/hadoop_file_system.h) * [GCS plugin](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/core/platform/cloud/gcs_file_system.h) +* [S3 + plugin](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/core/platform/s3/s3_file_system.h) #### The File interfaces @@ -223,7 +225,7 @@ it will use the `FooBarFileSystem` implementation. Next, you must build a shared object containing this implementation. An example of doing so using bazel's `cc_binary` rule can be found [here](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/BUILD#L244), -but you may use any build system to do so. See the section on @{$adding_an_op#build-the-op-library$building the op library} for similar +but you may use any build system to do so. See the section on @{$adding_an_op#build_the_op_library$building the op library} for similar instructions. The result of building this target is a `.so` shared object file. diff --git a/extend/index.md b/extend/index.md index bdff60b..1ab0340 100644 --- a/extend/index.md +++ b/extend/index.md @@ -16,9 +16,10 @@ TensorFlow: for your own file and record formats. Python is currently the only language supported by TensorFlow's API stability -promises. However, TensorFlow also provides functionality in C++, Java, and Go, +promises. However, TensorFlow also provides functionality in C++, Go, Java and +[JavaScript](https://js.tensorflow.org), plus community support for [Haskell](https://github.com/tensorflow/haskell) and -[Rust](https://github.com/tensorflow/rust). If you'd like to create or +[Rust](https://github.com/tensorflow/rust). If you'd like to create or develop TensorFlow features in a language other than these languages, read the following guide: diff --git a/extend/new_data_formats.md b/extend/new_data_formats.md index b3cc968..2c33a6b 100644 --- a/extend/new_data_formats.md +++ b/extend/new_data_formats.md @@ -1,4 +1,4 @@ -# Custom Data Readers +# Reading custom file and record formats PREREQUISITES: @@ -9,187 +9,273 @@ PREREQUISITES: We divide the task of supporting a file format into two pieces: -* File formats: We use a *Reader* Op to read a *record* (which can be any - string) from a file. -* Record formats: We use decoder or parsing Ops to turn a string record +* File formats: We use a reader `tf.data.Dataset` to read raw *records* (which + are typically represented by scalar string tensors, but can have more + structure) from a file. +* Record formats: We use decoder or parsing ops to turn a string record into tensors usable by TensorFlow. For example, to read a [CSV file](https://en.wikipedia.org/wiki/Comma-separated_values), we use -@{tf.TextLineReader$a Reader for text files} -followed by -@{tf.decode_csv$an Op that parses CSV data from a line of text}. +@{tf.data.TextLineDataset$a dataset for reading text files line-by-line} +and then @{tf.data.Dataset.map$map} an +@{tf.decode_csv$op} that parses CSV data from each line of text in the dataset. [TOC] -## Writing a Reader for a file format +## Writing a `Dataset` for a file format -A `Reader` is something that reads records from a file. There are some examples -of Reader Ops already built into TensorFlow: +A @{tf.data.Dataset} represents a sequence of *elements*, which can be the +individual records in a file. There are several examples of "reader" datasets +that are already built into TensorFlow: -* @{tf.TFRecordReader} - ([source in `kernels/tf_record_reader_op.cc`](https://www.tensorflow.org/code/tensorflow/core/kernels/tf_record_reader_op.cc)) -* @{tf.FixedLengthRecordReader} - ([source in `kernels/fixed_length_record_reader_op.cc`](https://www.tensorflow.org/code/tensorflow/core/kernels/fixed_length_record_reader_op.cc)) -* @{tf.TextLineReader} - ([source in `kernels/text_line_reader_op.cc`](https://www.tensorflow.org/code/tensorflow/core/kernels/text_line_reader_op.cc)) +* @{tf.data.TFRecordDataset} + ([source in `kernels/data/reader_dataset_ops.cc`](https://www.tensorflow.org/code/tensorflow/core/kernels/data/reader_dataset_ops.cc)) +* @{tf.data.FixedLengthRecordDataset} + ([source in `kernels/data/reader_dataset_ops.cc`](https://www.tensorflow.org/code/tensorflow/core/kernels/data/reader_dataset_ops.cc)) +* @{tf.data.TextLineDataset} + ([source in `kernels/data/reader_dataset_ops.cc`](https://www.tensorflow.org/code/tensorflow/core/kernels/data/reader_dataset_ops.cc)) -You can see these all expose the same interface, the only differences -are in their constructors. The most important method is `read`. -It takes a queue argument, which is where it gets filenames to -read from whenever it needs one (e.g. when the `read` op first runs, or -the previous `read` reads the last record from a file). It produces -two scalar tensors: a string key and a string value. +Each of these implementations comprises three related classes: -To create a new reader called `SomeReader`, you will need to: +* A `tensorflow::DatasetOpKernel` subclass (e.g. `TextLineDatasetOp`), which + tells TensorFlow how to construct a dataset object from the inputs to and + attrs of an op, in its `MakeDataset()` method. -1. In C++, define a subclass of - [`tensorflow::ReaderBase`](https://www.tensorflow.org/code/tensorflow/core/framework/reader_base.h) - called `SomeReader`. -2. In C++, register a new reader op and kernel with the name `"SomeReader"`. -3. In Python, define a subclass of @{tf.ReaderBase} called `SomeReader`. +* A `tensorflow::GraphDatasetBase` subclass (e.g. `TextLineDatasetOp::Dataset`), + which represents the *immutable* definition of the dataset itself, and tells + TensorFlow how to construct an iterator object over that dataset, in its + `MakeIterator()` method. -You can put all the C++ code in a file in -`tensorflow/core/user_ops/some_reader_op.cc`. The code to read a file will live -in a descendant of the C++ `ReaderBase` class, which is defined in -[`tensorflow/core/kernels/reader_base.h`](https://www.tensorflow.org/code/tensorflow/core/framework/reader_base.h). -You will need to implement the following methods: +* A `tensorflow::DatasetIterator
+For a more detailed introduction to neural networks, see the +[Introduction to Neural Nets section of Machine Learning Crash Course](https://developers.google.com/machine-learning/crash-course/introduction-to-neural-networks/anatomy). + To specify a model type, instantiate an [**Estimator**](https://developers.google.com/machine-learning/glossary/#Estimators) class. TensorFlow provides two categories of Estimators: @@ -449,9 +459,9 @@ will become very important. ### Train the model -Instantiating a `tf.Estimator.DNNClassifier` creates a framework for learning -the model. Basically, we've wired a network but haven't yet let data flow -through it. To train the neural network, call the Estimator object's `train` +Instantiating a `tf.Estimator.DNNClassifier` creates a framework for learning +the model. Basically, we've wired a network but haven't yet let data flow +through it. To train the neural network, call the Estimator object's `train` method. For example: ```python @@ -560,15 +570,15 @@ of 0.5. The following suggests a more effective model:
javac -cp libtensorflow-1.6.0.jar HelloTF.java+
javac -cp libtensorflow-1.8.0.jar HelloTF.java### Running @@ -239,11 +239,11 @@ two files are available to the JVM: For example, the following command line executes the `HelloTF` program on Linux and macOS X: -
java -cp libtensorflow-1.6.0.jar:. -Djava.library.path=./jni HelloTF+
java -cp libtensorflow-1.8.0.jar:. -Djava.library.path=./jni HelloTFAnd the following command line executes the `HelloTF` program on Windows: -
java -cp libtensorflow-1.6.0.jar;. -Djava.library.path=jni HelloTF+
java -cp libtensorflow-1.8.0.jar;. -Djava.library.path=jni HelloTFIf the program prints Hello from version, you've successfully installed TensorFlow for Java and are ready to use the API. If the program diff --git a/install/install_linux.md b/install/install_linux.md index be74a0d..16a232e 100644 --- a/install/install_linux.md +++ b/install/install_linux.md @@ -31,23 +31,24 @@ If you are installing TensorFlow with GPU support using one of the mechanisms described in this guide, then the following NVIDIA software must be installed on your system: - * CUDA® Toolkit 9.0. For details, see - [NVIDIA's documentation](http://docs.nvidia.com/cuda/cuda-installation-guide-linux/#axzz4VZnqTJ2A). - Ensure that you append the relevant Cuda pathnames to the + * [CUDA Toolkit 9.0](http://nvidia.com/cuda). For details, see + [NVIDIA's documentation](http://docs.nvidia.com/cuda/cuda-installation-guide-linux/). + Ensure that you append the relevant CUDA pathnames to the `LD_LIBRARY_PATH` environment variable as described in the NVIDIA documentation. - * The NVIDIA drivers associated with CUDA Toolkit 9.0. - * cuDNN v7.0. For details, see - [NVIDIA's documentation](https://developer.nvidia.com/cudnn). + * [cuDNN SDK v7](http://developer.nvidia.com/cudnn). For details, see + [NVIDIA's documentation](http://docs.nvidia.com/deeplearning/sdk/cudnn-install/). Ensure that you create the `CUDA_HOME` environment variable as described in the NVIDIA documentation. * GPU card with CUDA Compute Capability 3.0 or higher for building from source and 3.5 or higher for our binaries. See [NVIDIA documentation](https://developer.nvidia.com/cuda-gpus) for a list of supported GPU cards. + * [GPU drivers](http://nvidia.com/driver) supporting your version of the CUDA + Toolkit. * The libcupti-dev library, which is the NVIDIA CUDA Profile Tools Interface. This library provides advanced profiling support. To install this library, - issue the following command for CUDA Toolkit >= 8.0: + issue the following command for CUDA Toolkit >= 9.0:
$ sudo apt-get install cuda-command-line-tools
@@ -56,7 +57,7 @@ must be installed on your system:
and add its path to your `LD_LIBRARY_PATH` environment variable:
- $ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda/extras/CUPTI/lib64
+ $ export LD_LIBRARY_PATH=${LD_LIBRARY_PATH:+${LD_LIBRARY_PATH}:}/usr/local/cuda/extras/CUPTI/lib64
For CUDA Toolkit <= 7.5 do:
@@ -65,16 +66,41 @@ must be installed on your system:
$ sudo apt-get install libcupti-dev
+ * **[OPTIONAL]** For optimized inferencing performance, you can also install
+ **NVIDIA TensorRT 3.0**. The minimal set of TensorRT runtime components needed
+ for use with the pre-built `tensorflow-gpu` package can be installed as follows:
+
+ + $ wget https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1404/x86_64/nvinfer-runtime-trt-repo-ubuntu1404-3.0.4-ga-cuda9.0_1.0-1_amd64.deb + $ sudo dpkg -i nvinfer-runtime-trt-repo-ubuntu1404-3.0.4-ga-cuda9.0_1.0-1_amd64.deb + $ sudo apt-get update + $ sudo apt-get install -y --allow-downgrades libnvinfer-dev libcudnn7-dev=7.0.5.15-1+cuda9.0 libcudnn7=7.0.5.15-1+cuda9.0 ++ + **IMPORTANT:** For compatibility with the pre-built `tensorflow-gpu` + package, please use the Ubuntu **14.04** package of TensorRT as shown above, + even when installing onto an Ubuntu 16.04 system.
+
+ To build the TensorFlow-TensorRT integration module from source rather than + using pre-built binaries, see the [module documentation](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/tensorrt#using-tensorrt-in-tensorflow). + For detailed TensorRT installation instructions, see [NVIDIA's TensorRT documentation](http://docs.nvidia.com/deeplearning/sdk/tensorrt-install-guide/index.html).
+
+ To avoid cuDNN version conflicts during later system upgrades, you can hold + the cuDNN version at 7.0.5: + +
+ $ sudo apt-mark hold libcudnn7 libcudnn7-dev ++ + To later allow upgrades, you can remove the hold: + +
+ $ sudo apt-mark unhold libcudnn7 libcudnn7-dev ++ If you have an earlier version of the preceding packages, please upgrade to the specified versions. If upgrading is not possible, then you may still run -TensorFlow with GPU support, but only if you do the following: - - * Install TensorFlow from sources as documented in - @{$install_sources$Installing TensorFlow from Sources}. - * Install or upgrade to at least the following NVIDIA versions: - * CUDA toolkit 7.0 or greater - * cuDNN v3 or greater - * GPU card with CUDA Compute Capability 3.0 or higher. +TensorFlow with GPU support, if you @{$install_sources$install TensorFlow from Sources}. ## Determine how to install TensorFlow @@ -149,7 +175,8 @@ Take the following steps to install TensorFlow with Virtualenv: commands:
$ source ~/tensorflow/bin/activate # bash, sh, ksh, or zsh - $ source ~/tensorflow/bin/activate.csh # csh or tcsh+ $ source ~/tensorflow/bin/activate.csh # csh or tcsh + $ . ~/tensorflow/bin/activate.fish # fish The preceding source command should change your prompt to the following: @@ -189,7 +216,7 @@ Take the following steps to install TensorFlow with Virtualenv: Virtualenv environment:
(tensorflow)$ pip3 install --upgrade \ - https://storage.googleapis.com/tensorflow/linux/cpu/tensorflow-1.6.0-cp34-cp34m-linux_x86_64.whl+ https://storage.googleapis.com/tensorflow/linux/cpu/tensorflow-1.8.0-cp34-cp34m-linux_x86_64.whl If you encounter installation problems, see [Common Installation Problems](#common_installation_problems). @@ -294,7 +321,7 @@ take the following steps:
$ sudo pip3 install --upgrade \
- https://storage.googleapis.com/tensorflow/linux/cpu/tensorflow-1.6.0-cp34-cp34m-linux_x86_64.whl
+ https://storage.googleapis.com/tensorflow/linux/cpu/tensorflow-1.8.0-cp34-cp34m-linux_x86_64.whl
If this step fails, see
@@ -357,24 +384,23 @@ where:
to 6006.
* TensorFlowCPUImage is required. It identifies the Docker
container. Specify one of the following values:
- * gcr.io/tensorflow/tensorflow, which is the TensorFlow CPU binary image.
- * gcr.io/tensorflow/tensorflow:latest-devel, which is the latest
+ * tensorflow/tensorflow, which is the TensorFlow CPU binary image.
+ * tensorflow/tensorflow:latest-devel, which is the latest
TensorFlow CPU Binary image plus source code.
- * gcr.io/tensorflow/tensorflow:version, which is the
+ * tensorflow/tensorflow:version, which is the
specified version (for example, 1.1.0rc1) of TensorFlow CPU binary image.
- * gcr.io/tensorflow/tensorflow:version-devel, which is
+ * tensorflow/tensorflow:version-devel, which is
the specified version (for example, 1.1.0rc1) of the TensorFlow GPU
binary image plus source code.
- gcr.io is the Google Container Registry. Note that some
- TensorFlow images are also available at
+ TensorFlow images are available at
[dockerhub](https://hub.docker.com/r/tensorflow/tensorflow/).
For example, the following command launches the latest TensorFlow CPU binary image
in a Docker container from which you can run TensorFlow programs in a shell:
-$ docker run -it gcr.io/tensorflow/tensorflow bash +$ docker run -it tensorflow/tensorflow bashThe following command also launches the latest TensorFlow CPU binary image in a @@ -382,7 +408,7 @@ Docker container. However, in this Docker container, you can run TensorFlow programs in a Jupyter notebook:
-$ docker run -it -p 8888:8888 gcr.io/tensorflow/tensorflow +$ docker run -it -p 8888:8888 tensorflow/tensorflowDocker will download the TensorFlow binary image the first time you launch it. @@ -406,14 +432,14 @@ where: hostPort and
containerPort to `8888`.
* TensorFlowGPUImage specifies the Docker container. You must
specify one of the following values:
- * gcr.io/tensorflow/tensorflow:latest-gpu, which is the latest
+ * tensorflow/tensorflow:latest-gpu, which is the latest
TensorFlow GPU binary image.
- * gcr.io/tensorflow/tensorflow:latest-devel-gpu, which is
+ * tensorflow/tensorflow:latest-devel-gpu, which is
the latest TensorFlow GPU Binary image plus source code.
- * gcr.io/tensorflow/tensorflow:version-gpu, which is the
+ * tensorflow/tensorflow:version-gpu, which is the
specified version (for example, 0.12.1) of the TensorFlow GPU
binary image.
- * gcr.io/tensorflow/tensorflow:version-devel-gpu, which is
+ * tensorflow/tensorflow:version-devel-gpu, which is
the specified version (for example, 0.12.1) of the TensorFlow GPU
binary image plus source code.
@@ -422,7 +448,7 @@ following command launches the latest TensorFlow GPU binary image in a
Docker container from which you can run TensorFlow programs in a shell:
-$ nvidia-docker run -it gcr.io/tensorflow/tensorflow:latest-gpu bash +$ nvidia-docker run -it tensorflow/tensorflow:latest-gpu bashThe following command also launches the latest TensorFlow GPU binary image @@ -430,13 +456,13 @@ in a Docker container. In this Docker container, you can run TensorFlow programs in a Jupyter notebook:
-$ nvidia-docker run -it -p 8888:8888 gcr.io/tensorflow/tensorflow:latest-gpu +$ nvidia-docker run -it -p 8888:8888 tensorflow/tensorflow:latest-gpuThe following command installs an older TensorFlow version (0.12.1):
-$ nvidia-docker run -it -p 8888:8888 gcr.io/tensorflow/tensorflow:0.12.1-gpu +$ nvidia-docker run -it -p 8888:8888 tensorflow/tensorflow:0.12.1-gpuDocker will download the TensorFlow binary image the first time you launch it. @@ -481,8 +507,7 @@ Take the following steps to install TensorFlow in an Anaconda environment:
(tensorflow)$ pip install --ignore-installed --upgrade \
- https://storage.googleapis.com/tensorflow/linux/cpu/tensorflow-1.6.0-cp34-cp34m-linux_x86_64.whl
-
+ https://storage.googleapis.com/tensorflow/linux/cpu/tensorflow-1.8.0-cp34-cp34m-linux_x86_64.whl
## Validate your installation
@@ -507,7 +532,7 @@ If you installed through Docker, start a Docker container
from which you can run bash. For example:
-$ docker run -it gcr.io/tensorflow/tensorflow bash +$ docker run -it tensorflow/tensorflow bash@@ -532,11 +557,18 @@ TensorFlow programs:
Hello, TensorFlow!-If you are new to TensorFlow, see @{$get_started/premade_estimators$Getting Started with TensorFlow}. - If the system outputs an error message instead of a greeting, see [Common installation problems](#common_installation_problems). +If you are new to machine learning, we recommend the following: + +* [Machine Learning Crash Course](https://developers.google.com/machine-learning/crash-course) +* @{$get_started/get_started_for_beginners$Getting Started for ML Beginners} + +If you are experienced with machine learning but new to TensorFlow, see +@{$get_started/premade_estimators$Getting Started with TensorFlow}. + + ## Common installation problems We are relying on Stack Overflow to document TensorFlow installation problems @@ -649,14 +681,14 @@ This section documents the relevant values for Linux installations. CPU only:
-https://storage.googleapis.com/tensorflow/linux/cpu/tensorflow-1.6.0-cp27-none-linux_x86_64.whl +https://storage.googleapis.com/tensorflow/linux/cpu/tensorflow-1.8.0-cp27-none-linux_x86_64.whlGPU support:
-https://storage.googleapis.com/tensorflow/linux/gpu/tensorflow_gpu-1.6.0-cp27-none-linux_x86_64.whl +https://storage.googleapis.com/tensorflow/linux/gpu/tensorflow_gpu-1.8.0-cp27-none-linux_x86_64.whlNote that GPU support requires the NVIDIA hardware and software described in @@ -668,14 +700,14 @@ Note that GPU support requires the NVIDIA hardware and software described in CPU only:
-https://storage.googleapis.com/tensorflow/linux/cpu/tensorflow-1.6.0-cp34-cp34m-linux_x86_64.whl +https://storage.googleapis.com/tensorflow/linux/cpu/tensorflow-1.8.0-cp34-cp34m-linux_x86_64.whlGPU support:
-https://storage.googleapis.com/tensorflow/linux/gpu/tensorflow_gpu-1.6.0-cp34-cp34m-linux_x86_64.whl +https://storage.googleapis.com/tensorflow/linux/gpu/tensorflow_gpu-1.8.0-cp34-cp34m-linux_x86_64.whlNote that GPU support requires the NVIDIA hardware and software described in @@ -687,14 +719,14 @@ Note that GPU support requires the NVIDIA hardware and software described in CPU only:
-https://storage.googleapis.com/tensorflow/linux/cpu/tensorflow-1.6.0-cp35-cp35m-linux_x86_64.whl +https://storage.googleapis.com/tensorflow/linux/cpu/tensorflow-1.8.0-cp35-cp35m-linux_x86_64.whlGPU support:
-https://storage.googleapis.com/tensorflow/linux/gpu/tensorflow_gpu-1.6.0-cp35-cp35m-linux_x86_64.whl +https://storage.googleapis.com/tensorflow/linux/gpu/tensorflow_gpu-1.8.0-cp35-cp35m-linux_x86_64.whl@@ -706,14 +738,14 @@ Note that GPU support requires the NVIDIA hardware and software described in CPU only:
-https://storage.googleapis.com/tensorflow/linux/cpu/tensorflow-1.6.0-cp36-cp36m-linux_x86_64.whl +https://storage.googleapis.com/tensorflow/linux/cpu/tensorflow-1.8.0-cp36-cp36m-linux_x86_64.whlGPU support:
-https://storage.googleapis.com/tensorflow/linux/gpu/tensorflow_gpu-1.6.0-cp36-cp36m-linux_x86_64.whl +https://storage.googleapis.com/tensorflow/linux/gpu/tensorflow_gpu-1.8.0-cp36-cp36m-linux_x86_64.whldiff --git a/install/install_mac.md b/install/install_mac.md index 62f8963..c79075b 100644 --- a/install/install_mac.md +++ b/install/install_mac.md @@ -5,7 +5,11 @@ instructions might also work on other macOS variants, we have only tested (and we only support) these instructions on machines meeting the following requirements: - * macOS X 10.11 (El Capitan) or higher + * macOS 10.12.6 (Sierra) or higher + +Note: There are known, accuracy-affecting numerical issues before macOS 10.12.6 +(Sierra) that are described in +[GitHub#15933](https://github.com/tensorflow/tensorflow/issues/15933#issuecomment-366331383). Note: As of version 1.2, TensorFlow no longer provides GPU support on macOS. @@ -115,7 +119,7 @@ Take the following steps to install TensorFlow with Virtualenv: TensorFlow in the active Virtualenv is as follows:
$ pip3 install --upgrade \ - https://storage.googleapis.com/tensorflow/mac/cpu/tensorflow-1.6.0-py3-none-any.whl+ https://storage.googleapis.com/tensorflow/mac/cpu/tensorflow-1.8.0-py3-none-any.whl If you encounter installation problems, see [Common Installation Problems](#common-installation-problems). @@ -234,11 +238,11 @@ take the following steps: operating system and Python version. Find the appropriate value for tfBinaryURL [here](#the_url_of_the_tensorflow_python_package). For example, if - you are installing TensorFlow for Mac OS and Python 2.7 + you are installing TensorFlow for macOS and Python 2.7 issue the following command:
$ sudo pip3 install --upgrade \ - https://storage.googleapis.com/tensorflow/mac/cpu/tensorflow-1.6.0-py3-none-any.whl+ https://storage.googleapis.com/tensorflow/mac/cpu/tensorflow-1.8.0-py3-none-any.whl If the preceding command fails, see [installation problems](#common-installation-problems). @@ -288,24 +292,23 @@ where: to 6006. * TensorFlowImage is required. It identifies the Docker container. You must specify one of the following values: - *
gcr.io/tensorflow/tensorflow: TensorFlow binary image.
- * gcr.io/tensorflow/tensorflow:latest-devel: TensorFlow
+ * tensorflow/tensorflow: TensorFlow binary image.
+ * tensorflow/tensorflow:latest-devel: TensorFlow
Binary image plus source code.
-gcr.io is the Google Container Registry. Note that some
-TensorFlow images are also available at
+The TensorFlow images are available at
[dockerhub](https://hub.docker.com/r/tensorflow/tensorflow/).
For example, the following command launches a TensorFlow CPU binary image
in a Docker container from which you can run TensorFlow programs in a shell:
-$ docker run -it gcr.io/tensorflow/tensorflow bash+
$ docker run -it tensorflow/tensorflow bashThe following command also launches a TensorFlow CPU binary image in a Docker container. However, in this Docker container, you can run TensorFlow programs in a Jupyter notebook: -
$ docker run -it -p 8888:8888 gcr.io/tensorflow/tensorflow+
$ docker run -it -p 8888:8888 tensorflow/tensorflowDocker will download the TensorFlow binary image the first time you launch it. @@ -347,7 +350,7 @@ Take the following steps to install TensorFlow in an Anaconda environment: TensorFlow for Python 2.7:
(targetDirectory)$ pip install --ignore-installed --upgrade \ - https://storage.googleapis.com/tensorflow/mac/cpu/tensorflow-1.6.0-py2-none-any.whl+ https://storage.googleapis.com/tensorflow/mac/cpu/tensorflow-1.8.0-py2-none-any.whl @@ -372,7 +375,7 @@ do the following: If you installed through Docker, start a Docker container that runs bash. For example: -
$ docker run -it gcr.io/tensorflow/tensorflow bash+
$ docker run -it tensorflow/tensorflow bash@@ -397,12 +400,18 @@ writing TensorFlow programs:
Hello, TensorFlow!-If you are new to TensorFlow, see -@{$get_started/premade_estimators$Getting Started with TensorFlow}. - If the system outputs an error message instead of a greeting, see [Common installation problems](#common_installation_problems). +If you are new to machine learning, we recommend the following: + +* [Machine Learning Crash Course](https://developers.google.com/machine-learning/crash-course) +* @{$get_started/get_started_for_beginners$Getting Started for ML Beginners} + +If you are experienced with machine learning but new to TensorFlow, see +@{$get_started/premade_estimators$Getting Started with TensorFlow}. + + ## Common installation problems We are relying on Stack Overflow to document TensorFlow installation problems @@ -509,18 +518,13 @@ RuntimeError: Broken toolchain: cannot link a simple C program ## The URL of the TensorFlow Python package A few installation mechanisms require the URL of the TensorFlow Python package. -The value you specify depends on three factors: - - * operating system - * Python version - -This section documents the relevant values for Mac OS installations. +The value you specify depends on your Python version. ### Python 2.7
-https://storage.googleapis.com/tensorflow/mac/cpu/tensorflow-1.6.0-py2-none-any.whl +https://storage.googleapis.com/tensorflow/mac/cpu/tensorflow-1.8.0-py2-none-any.whl@@ -528,5 +532,5 @@ https://storage.googleapis.com/tensorflow/mac/cpu/tensorflow-1.6.0-py2-none-any.
-https://storage.googleapis.com/tensorflow/mac/cpu/tensorflow-1.6.0-py3-none-any.whl +https://storage.googleapis.com/tensorflow/mac/cpu/tensorflow-1.8.0-py3-none-any.whldiff --git a/install/install_sources.md b/install/install_sources.md index 638a64c..3d93736 100644 --- a/install/install_sources.md +++ b/install/install_sources.md @@ -133,30 +133,21 @@ The following NVIDIA hardware must be installed on your system: The following NVIDIA software must be installed on your system: - * NVIDIA's Cuda Toolkit (>= 7.0). We recommend version 9.0. + * [CUDA Toolkit](http://nvidia.com/cuda) (>= 7.0). We recommend version 9.0. For details, see - [NVIDIA's documentation](http://docs.nvidia.com/cuda/cuda-installation-guide-linux/#axzz4VZnqTJ2A). - Ensure that you append the relevant Cuda pathnames to the + [NVIDIA's documentation](http://docs.nvidia.com/cuda/cuda-installation-guide-linux/). + Ensure that you append the relevant CUDA pathnames to the `LD_LIBRARY_PATH` environment variable as described in the NVIDIA documentation. - * The NVIDIA drivers associated with NVIDIA's Cuda Toolkit. - * cuDNN (>= v3). We recommend version 6.0. For details, see - [NVIDIA's documentation](https://developer.nvidia.com/cudnn), - particularly the description of appending the appropriate pathname - to your `LD_LIBRARY_PATH` environment variable. - -Finally, you must also install `libcupti` which for Cuda Toolkit >= 8.0 you do via - -
$ sudo apt-get install cuda-command-line-tools- -and add its path to your `LD_LIBRARY_PATH` environment variable: - -
$ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda/extras/CUPTI/lib64- -For Cuda Toolkit <= 7.5, you install `libcupti-dev` by invoking the following command: - -
$ sudo apt-get install libcupti-dev+ * [GPU drivers](http://nvidia.com/driver) supporting your version of the CUDA + Toolkit. + * [cuDNN SDK](http://developer.nvidia.com/cudnn) (>= v3). We recommend version 7.0. For details, see + [NVIDIA's documentation](http://docs.nvidia.com/deeplearning/sdk/cudnn-install/). + * [CUPTI](http://docs.nvidia.com/cuda/cupti/) ships with the CUDA Toolkit, but + you also need to append its path to the `LD_LIBRARY_PATH` environment + variable: +
$ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda/extras/CUPTI/lib64### Next @@ -221,7 +212,7 @@ problem, do either of the following: * Download Xcode 7.2 and select it as your default by issuing the following command: -
$ sudo xcode-select -s /Application/Xcode-7.2/Xcode.app+
$ sudo xcode-select -s /Applications/Xcode-7.2/Xcode.app**NOTE:** Your system must fulfill the NVIDIA software requirements described in one of the following documents: @@ -240,8 +231,8 @@ such as compiler flags. You must run this script *prior* to creating the pip package and installing TensorFlow. If you wish to build TensorFlow with GPU, `configure` will ask -you to specify the version numbers of Cuda and cuDNN. If several -versions of Cuda or cuDNN are installed on your system, explicitly select +you to specify the version numbers of CUDA and cuDNN. If several +versions of CUDA or cuDNN are installed on your system, explicitly select the desired version instead of relying on the default. One of the questions that `configure` will ask is as follows: @@ -289,12 +280,12 @@ Do you wish to build TensorFlow with CUDA support? [y/N] Y CUDA support will be enabled for TensorFlow Do you want to use clang as CUDA compiler? [y/N] nvcc will be used as CUDA compiler -Please specify the Cuda SDK version you want to use, e.g. 7.0. [Leave empty to default to CUDA 9.0]: 9.0 +Please specify the CUDA SDK version you want to use, e.g. 7.0. [Leave empty to default to CUDA 9.0]: 9.0 Please specify the location where CUDA 9.0 toolkit is installed. Refer to README.md for more details. [Default is /usr/local/cuda]: Please specify which gcc should be used by nvcc as the host compiler. [Default is /usr/bin/gcc]: Please specify the cuDNN version you want to use. [Leave empty to default to cuDNN 7.0]: 7 Please specify the location where cuDNN 7 library is installed. Refer to README.md for more details. [Default is /usr/local/cuda]: -Please specify a list of comma-separated Cuda compute capabilities you want to build with. +Please specify a list of comma-separated CUDA compute capabilities you want to build with. You can find the compute capability of your device at: https://developer.nvidia.com/cuda-gpus. Please note that each additional compute capability significantly increases your build time and binary size. [Default is: "3.5,5.2"]: 3.0 @@ -304,14 +295,14 @@ Configuration finished If you told `configure` to build for GPU support, then `configure` -will create a canonical set of symbolic links to the Cuda libraries -on your system. Therefore, every time you change the Cuda library paths, +will create a canonical set of symbolic links to the CUDA libraries +on your system. Therefore, every time you change the CUDA library paths, you must rerun the `configure` script before re-invoking the
bazel build command.
Note the following:
- * Although it is possible to build both Cuda and non-Cuda configs
+ * Although it is possible to build both CUDA and non-CUDA configs
under the same source tree, we recommend running `bazel clean` when
switching between these two configurations in the same source tree.
* If you don't run the `configure` script *before* running the
@@ -359,10 +350,10 @@ Invoke `pip install` to install that pip package.
The filename of the `.whl` file depends on your platform.
For example, the following command will install the pip package
-for TensorFlow 1.6.0 on Linux:
+for TensorFlow 1.8.0 on Linux:
-$ sudo pip install /tmp/tensorflow_pkg/tensorflow-1.6.0-py2-none-any.whl +$ sudo pip install /tmp/tensorflow_pkg/tensorflow-1.8.0-py2-none-any.whl## Validate your installation @@ -393,8 +384,7 @@ TensorFlow programs:
Hello, TensorFlow!-If you are new to TensorFlow, see @{$get_started/get_started$Getting Started with -TensorFlow}. +If you are new to TensorFlow, see @{$get_started/premade_estimators$Getting Started with TensorFlow}. If the system outputs an error message instead of a greeting, see [Common installation problems](#common_installation_problems). @@ -460,6 +450,10 @@ Stack Overflow and specify the `tensorflow` tag. **Linux**
Starting with a trained TensorFlow model on disk, you'll convert that model to
the TensorFlow Lite file format (`.tflite`) using the TensorFlow Lite
diff --git a/performance/datasets_performance.md b/performance/datasets_performance.md
index 4f95e17..46b43b7 100644
--- a/performance/datasets_performance.md
+++ b/performance/datasets_performance.md
@@ -92,11 +92,11 @@ transform the data.
Without pipelining, the CPU and the GPU/TPU sit idle much of the time:
-
+
With pipelining, idle time diminishes significantly:
-
+
The `tf.data` API provides a software pipelining mechanism through the
@{tf.data.Dataset.prefetch} transformation, which can be used to decouple the
@@ -139,7 +139,7 @@ multiple CPU cores. To make this possible, the `map` transformation provides the
the following diagram illustrates the effect of setting `num_parallel_calls=2`
to the `map` transformation:
-
+
Choosing the best value for the `num_parallel_calls` argument depends on your
hardware, characteristics of your training data (such as its size and shape),
@@ -213,7 +213,7 @@ number of datasets to overlap can be specified by the `cycle_length` argument.
The following diagram illustrates the effect of supplying `cycle_length=2` to
the `parallel_interleave` transformation:
-
+
To apply this change to our running example, change:
diff --git a/performance/leftnav_files b/performance/leftnav_files
index 316f023..1f894c3 100644
--- a/performance/leftnav_files
+++ b/performance/leftnav_files
@@ -1,7 +1,9 @@
+index.md
performance_guide.md
datasets_performance.md
performance_models.md
benchmarks.md
+quantization.md
### XLA
xla/index.md
@@ -11,6 +13,3 @@ xla/jit.md
xla/operation_semantics.md
xla/shapes.md
xla/tfcompile.md
-
-### Quantization
-quantization.md
diff --git a/performance/performance_guide.md b/performance/performance_guide.md
index cd47fc2..580a899 100644
--- a/performance/performance_guide.md
+++ b/performance/performance_guide.md
@@ -78,7 +78,7 @@ training CIFAR-10 illustrates the use of the `tf.data` API along with
The `tf.data` API utilizes C++ multi-threading and has a much lower overhead
than the Python-based `queue_runner` that is limited by Python's multi-threading
performance. A detailed performance guide for the `tf.data` API can be found
-[here](#datasets_performance).
+[here](@{$datasets_performance}).
While feeding data using a `feed_dict` offers a high level of flexibility, in
general `feed_dict` does not provide a scalable solution. If only a single GPU
diff --git a/performance/quantization.md b/performance/quantization.md
index 544274c..411889c 100644
--- a/performance/quantization.md
+++ b/performance/quantization.md
@@ -1,226 +1,253 @@
-# How to Quantize Neural Networks with TensorFlow
-
-When modern neural networks were being developed, the biggest challenge was
-getting them to work at all! That meant that accuracy and speed during training
-were the top priorities. Using floating point arithmetic was the easiest way to
-preserve accuracy, and GPUs were well-equipped to accelerate those calculations,
-so it's natural that not much attention was paid to other numerical formats.
-
-These days, we actually have a lot of models being deployed in commercial
-applications. The computation demands of training grow with the number of
-researchers, but the cycles needed for inference expand in proportion to users.
-That means pure inference efficiency has become a burning issue for a lot of
-teams.
-
-That is where quantization comes in. It's an umbrella term that covers a lot of
-different techniques to store numbers and perform calculations on them in more
-compact formats than 32-bit floating point. I am going to focus on eight-bit
-fixed point, for reasons I'll go into more detail on later.
-
-[TOC]
-
-## Why does Quantization Work?
-
-Training neural networks is done by applying many tiny nudges to the weights,
-and these small increments typically need floating point precision to work
-(though there are research efforts to use quantized representations here too).
-
-Taking a pre-trained model and running inference is very different. One of the
-magical qualities of deep networks is that they tend to cope very well with high
-levels of noise in their inputs. If you think about recognizing an object in a
-photo you've just taken, the network has to ignore all the CCD noise, lighting
-changes, and other non-essential differences between it and the training
-examples it's seen before, and focus on the important similarities instead. This
-ability means that they seem to treat low-precision calculations as just another
-source of noise, and still produce accurate results even with numerical formats
-that hold less information.
-
-## Why Quantize?
-
-Neural network models can take up a lot of space on disk, with the original
-AlexNet being over 200 MB in float format for example. Almost all of that size
-is taken up with the weights for the neural connections, since there are often
-many millions of these in a single model. Because they're all slightly different
-floating point numbers, simple compression formats like zip don't compress them
-well. They are arranged in large layers though, and within each layer the
-weights tend to be normally distributed within a certain range, for example -3.0
-to 6.0.
-
-The simplest motivation for quantization is to shrink file sizes by storing the
-min and max for each layer, and then compressing each float value to an
-eight-bit integer representing the closest real number in a linear set of 256
-within the range. For example with the -3.0 to 6.0 range, a 0 byte would
-represent -3.0, a 255 would stand for 6.0, and 128 would represent about 1.5.
-I'll go into the exact calculations later, since there's some subtleties, but
-this means you can get the benefit of a file on disk that's shrunk by 75%, and
-then convert back to float after loading so that your existing floating-point
-code can work without any changes.
-
-Another reason to quantize is to reduce the computational resources you need to
-do the inference calculations, by running them entirely with eight-bit inputs
-and outputs. This is a lot more difficult since it requires changes everywhere
-you do calculations, but offers a lot of potential rewards. Fetching eight-bit
-values only requires 25% of the memory bandwidth of floats, so you'll make much
-better use of caches and avoid bottlenecking on RAM access. You can also
-typically use SIMD operations that do many more operations per clock cycle. In
-some case you'll have a DSP chip available that can accelerate eight-bit
-calculations too, which can offer a lot of advantages.
-
-Moving calculations over to eight bit will help you run your models faster, and
-use less power (which is especially important on mobile devices). It also opens
-the door to a lot of embedded systems that can't run floating point code
-efficiently, so it can enable a lot of applications in the IoT world.
-
-## Why Not Train in Lower Precision Directly?
-
-There have been some experiments training at lower bit depths, but the results
-seem to indicate that you need higher than eight bit to handle the back
-propagation and gradients. That makes implementing the training more
-complicated, and so starting with inference made sense. We also already have a
-lot of float models already that we use and know well, so being able to convert
-them directly is very convenient.
-
-## How Can You Quantize Your Models?
-
-TensorFlow has production-grade support for eight-bit calculations built in. It
-also has a process for converting many models trained in floating-point over to
-equivalent graphs using quantized calculations for inference. For example,
-here's how you can translate the latest GoogLeNet model into a version that uses
-eight-bit computations:
-
-```sh
-curl -L "https://storage.googleapis.com/download.tensorflow.org/models/inception_v3_2016_08_28_frozen.pb.tar.gz" |
- tar -C tensorflow/examples/label_image/data -xz
-bazel build tensorflow/tools/graph_transforms:transform_graph
-bazel-bin/tensorflow/tools/graph_transforms/transform_graph \
- --in_graph=tensorflow/examples/label_image/data/inception_v3_2016_08_28_frozen.pb \
- --out_graph=/tmp/quantized_graph.pb \
- --inputs=input \
- --outputs=InceptionV3/Predictions/Reshape_1 \
- --transforms='add_default_attributes strip_unused_nodes(type=float, shape="1,299,299,3")
- remove_nodes(op=Identity, op=CheckNumerics) fold_constants(ignore_errors=true)
- fold_batch_norms fold_old_batch_norms quantize_weights quantize_nodes
- strip_unused_nodes sort_by_execution_order'
+# Fixed Point Quantization
+
+Quantization techniques store and calculate numbers in more compact formats.
+[TensorFlow Lite](/mobile/tflite/) adds quantization that uses an 8-bit fixed
+point representation.
+
+Since a challenge for modern neural networks is optimizing for high accuracy, the
+priority has been improving accuracy and speed during training. Using floating
+point arithmetic is an easy way to preserve accuracy and GPUs are designed to
+accelerate these calculations.
+
+However, as more machine learning models are deployed to mobile devices,
+inference efficiency has become a critical issue. Where the computational demand
+for *training* grows with the amount of models trained on different
+architectures, the computational demand for *inference* grows in proportion to
+the amount of users.
+
+## Quantization benefits
+
+
+Using 8-bit calculations help your models run faster and use less power. This is
+especially important for mobile devices and embedded applications that can't run
+floating point code efficiently, for example, Internet of Things (IoT) and
+robotics devices. There are additional opportunities to extend this support to
+more backends and research lower precision networks.
+
+### Smaller file sizes {: .hide-from-toc}
+
+Neural network models require a lot of space on disk. For example, the original
+AlexNet requires over 200 MB for the float format—almost all of that for the
+model's millions of weights. Because the weights are slightly different
+floating point numbers, simple compression formats perform poorly (like zip).
+
+Weights fall in large layers of numerical values. For each layer, weights tend to
+be normally distributed within a range. Quantization can shrink file sizes by
+storing the minimum and maximum weight for each layer, then compress each
+weight's float value to an 8-bit integer representing the closest real number in
+a linear set of 256 within the range.
+
+### Faster inference {: .hide-from-toc}
+
+Since calculations are run entirely on 8-bit inputs and outputs, quantization
+reduces the computational resources needed for inference calculations. This is
+more involved, requiring changes to all floating point calculations, but results
+in a large speed-up for inference time.
+
+### Memory efficiency {: .hide-from-toc}
+
+Since fetching 8-bit values only requires 25% of the memory bandwidth of floats,
+more efficient caches avoid bottlenecks for RAM access. In many cases, the power
+consumption for running a neural network is dominated by memory access. The
+savings from using fixed-point 8-bit weights and activations are significant.
+
+Typically, SIMD operations are available that run more operations per clock
+cycle. In some cases, a DSP chip is available that accelerates 8-bit calculations
+resulting in a massive speedup.
+
+## Fixed point quantization techniques
+
+The goal is to use the same precision for weights and activations during both
+training and inference. But an important difference is that training consists of
+a forward pass and a backward pass, while inference only uses a forward pass.
+When we train the model with quantization in the loop, we ensure that the forward
+pass matches precision for both training and inference.
+
+To minimize the loss in accuracy for fully fixed point models (weights and
+activations), train the model with quantization in the loop. This simulates
+quantization in the forward pass of a model so weights tend towards values that
+perform better during quantized inference. The backward pass uses quantized
+weights and activations and models quantization as a straight through estimator.
+(See Bengio et al., [2013](https://arxiv.org/abs/1308.3432))
+
+Additionally, the minimum and maximum values for activations are determined
+during training. This allows a model trained with quantization in the loop to be
+converted to a fixed point inference model with little effort, eliminating the
+need for a separate calibration step.
+
+## Quantization training with TensorFlow
+
+TensorFlow can train models with quantization in the loop. Because training
+requires small gradient adjustments, floating point values are still used. To
+keep models as floating point while adding the quantization error in the training
+loop, @{$array_ops#Fake_quantization$fake quantization} nodes simulate the
+effect of quantization in the forward and backward passes.
+
+Since it's difficult to add these fake quantization operations to all the
+required locations in the model, there's a function available that rewrites the
+training graph. To create a fake quantized training graph:
+
+```
+# Build forward pass of model.
+loss = tf.losses.get_total_loss()
+
+# Call the training rewrite which rewrites the graph in-place with
+# FakeQuantization nodes and folds batchnorm for training. It is
+# often needed to fine tune a floating point model for quantization
+# with this training tool. When training from scratch, quant_delay
+# can be used to activate quantization after training to converge
+# with the float graph, effectively fine-tuning the model.
+tf.contrib.quantize.create_training_graph(quant_delay=2000000)
+
+# Call backward pass optimizer as usual.
+optimizer = tf.train.GradientDescentOptimizer(learning_rate)
+optimizer.minimize(loss)
```
-This will produce a new model that runs the same operations as the original, but
-with eight bit calculations internally, and all weights quantized as well. If
-you look at the file size, you'll see it's about a quarter of the original (23MB
-versus 91MB). You can still run this model using exactly the same inputs and
-outputs though, and you should get equivalent results. Here's an example:
+The rewritten *eval graph* is non-trivially different from the *training graph*
+since the quantization ops affect the batch normalization step. Because of this,
+we've added a separate rewrite for the *eval graph*:
-```sh
-bazel build tensorflow/examples/label_image:label_image
-bazel-bin/tensorflow/examples/label_image/label_image \
---graph=/tmp/quantized_graph.pb \
+```
+# Build eval model
+logits = tf.nn.softmax_cross_entropy_with_logits(...)
+
+# Call the eval rewrite which rewrites the graph in-place with
+# FakeQuantization nodes and fold batchnorm for eval.
+tf.contrib.quantize.create_eval_graph()
+
+# Save the checkpoint and eval graph proto to disk for freezing
+# and providing to TFLite.
+with open(eval_graph_file, ‘w’) as f:
+ f.write(str(g.as_graph_def()))
+saver = tf.train.Saver()
+saver.save(sess, checkpoint_name)
+```
+
+Methods to rewrite the training and eval graphs are an active area of research
+and experimentation. Although rewrites and quantized training might not work or
+improve performance for all models, we are working to generalize these
+techniques.
+
+## Generating fully quantized models
+
+The previously demonstrated after-rewrite eval graph only *simulates*
+quantization. To generate real fixed point computations from a trained
+quantization model, convert it to a fixed point kernel. Tensorflow Lite supports
+this conversion from the graph resulting from `create_eval_graph`.
+
+First, create a frozen graph that will be the input for the TensorFlow Lite
+toolchain:
+
+```
+bazel build tensorflow/python/tools:freeze_graph && \
+ bazel-bin/tensorflow/python/tools/freeze_graph \
+ --input_graph=eval_graph_def.pb \
+ --input_checkpoint=checkpoint \
+ --output_graph=frozen_eval_graph.pb --output_node_names=outputs
```
-You'll see that this runs the newly-quantized graph, and outputs a very similar
-answer to the original.
-
-You can run the same process on your own models saved out as GraphDefs, with the
-input and output names adapted to those your network requires. I recommend that
-you run them through the freeze_graph script first, to convert checkpoints into
-constants stored in the file.
-
-## How Does the Quantization Process Work?
-
-We've implemented quantization by writing equivalent eight-bit versions of
-operations that are commonly used during inference. These include convolution,
-matrix multiplication, activation functions, pooling operations and
-concatenation. The conversion script first replaces all the individual ops it
-knows about with quantized equivalents. These are small sub-graphs that have
-conversion functions before and after to move the data between float and
-eight-bit. Below is an example of what they look like. First here's the original
-Relu operation, with float inputs and outputs:
-
-
-
-Then, this is the equivalent converted subgraph, still with float inputs and
-outputs, but with internal conversions so the calculations are done in eight
-bit.
-
-
-
-The min and max operations actually look at the values in the input float
-tensor, and then feeds them into the Dequantize operation that converts the
-tensor into eight-bits. There are more details on how the quantized representation
-works later on.
-
-Once the individual operations have been converted, the next stage is to remove
-unnecessary conversions to and from float. If there are consecutive sequences of
-operations that all have float equivalents, then there will be a lot of adjacent
-Dequantize/Quantize ops. This stage spots that pattern, recognizes that they
-cancel each other out, and removes them, like this:
-
-
-
-Applied on a large scale to models where all of the operations have quantized
-equivalents, this gives a graph where all of the tensor calculations are done in
-eight bit, without having to convert to float.
-
-## What Representation is Used for Quantized Tensors?
-
-We approach converting floating-point arrays of numbers into eight-bit
-representations as a compression problem. We know that the weights and
-activation tensors in trained neural network models tend to have values that are
-distributed across comparatively small ranges (for example you might have -15 to
-+15 for weights, -500 to 1000 for activations on an image model, though the
-exact numbers will vary). We also know from experiment that neural nets tend to
-be very robust in the face of noise, and so the noise-like error produced by
-quantizing down to a small set of values will not hurt the precision of the
-overall results very much. We also want to pick a representation that's easy to
-perform calculations on, especially the large matrix multiplications that form
-the bulk of the work that's needed to run a model.
-
-These led us to pick a representation that has two floats to store the overall
-minimum and maximum values that are represented by the lowest and highest
-quantized value. Each entry in the quantized array represents a float value in
-that range, distributed linearly between the minimum and maximum. For example,
-if we have minimum = -10.0, and maximum = 30.0f, and an eight-bit array, here's
-what the quantized values represent:
+Provide this to the TensorFlow Lite Optimizing Converter (TOCO) to get a fully
+quantized TensorFLow Lite model:
```
-Quantized | Float
---------- | -----
-0 | -10.0
-255 | 30.0
-128 | 10.0
+bazel build tensorflow/contrib/lite/toco:toco && \
+ ./bazel-bin/third_party/tensorflow/contrib/lite/toco/toco \
+ --input_file=frozen_eval_graph.pb \
+ --output_file=tflite_model.tflite \
+ --input_format=TENSORFLOW_GRAPHDEF --output_format=TFLITE \
+ --inference_type=QUANTIZED_UINT8 \
+ --input_shape="1,224, 224,3" \
+ --input_array=input \
+ --output_array=outputs \
+ --std_value=127.5 --mean_value=127.5
```
-The advantages of this format are that it can represent arbitrary magnitudes of
-ranges, they don't have to be symmetrical, it can represent signed and unsigned
-values, and the linear spread makes doing multiplications straightforward. There
-are alternatives like [Song Han's code books](http://arxiv.org/pdf/1510.00149.pdf)
-that can use lower bit depths by non-linearly distributing the float values
-across the representation, but these tend to be more expensive to calculate on.
-
-The advantage of having a strong and clear definition of the quantized format is
-that it's always possible to convert back and forth from float for operations
-that aren't quantization-ready, or to inspect the tensors for debugging
-purposes. One implementation detail in TensorFlow that we're hoping to improve
-in the future is that the minimum and maximum float values need to be passed as
-separate tensors to the one holding the quantized values, so graphs can get a
-bit dense!
-
-The nice thing about the minimum and maximum ranges is that they can often be
-pre-calculated. Weight parameters are constants known at load time, so their
-ranges can also be stored as constants. We often know the ranges for inputs (for
-examples images are usually RGB values in the range 0.0 to 255.0), and many
-activation functions have known ranges too. This can avoid having to analyze the
-outputs of an operation to determine the range, which we need to do for math ops
-like convolution or matrix multiplication which produce 32-bit accumulated
-results from 8-bit inputs.
-
-## What's Next?
-
-We've found that we can get extremely good performance on mobile and embedded
-devices by using eight-bit arithmetic rather than floating-point. You can see
-the framework we use to optimize matrix multiplications at
-[gemmlowp](https://github.com/google/gemmlowp). We still need to apply all the
-lessons we've learned to the TensorFlow ops to get maximum performance on
-mobile, but we're actively working on that. Right now, this quantized
-implementation is a reasonably fast and accurate reference implementation that
-we're hoping will enable wider support for our eight-bit models on a wider
-variety of devices. We also hope that this demonstration will encourage the
-community to explore what's possible with low-precision neural networks.
+See the documentation for @{tf.contrib.quantize} and
+[TensorFlow Lite](/mobile/tflite/).
+
+## Quantized accuracy
+
+Fixed point [MobileNet](https://arxiv.org/abs/1704.0486) models are released with
+8-bit weights and activations. Using the rewriters, these models achieve the
+Top-1 accuracies listed in Table 1. For comparison, the floating point accuracies
+are listed for the same models. The code used to generate these models
+[is available](https://github.com/tensorflow/models/blob/master/research/slim/nets/mobilenet_v1.md)
+along with links to all of the pretrained mobilenet_v1 models.
+
+
+
+
+
diff --git a/programmers_guide/datasets.md b/programmers_guide/datasets.md
index 9ede4ab..9ccdbde 100644
--- a/programmers_guide/datasets.md
+++ b/programmers_guide/datasets.md
@@ -18,11 +18,11 @@ The `tf.data` API introduces two new abstractions to TensorFlow:
tensors representing the image data and a label. There are two distinct
ways to create a dataset:
- * Creating a **source** (e.g. `Dataset.from_tensor_slices()`) constructs a
+ * Creating a **source** (e.g. `Dataset.from_tensor_slices()`) constructs a
dataset from
one or more `tf.Tensor` objects.
- * Applying a **transformation** (e.g. `Dataset.batch()`) constructs a dataset
+ * Applying a **transformation** (e.g. `Dataset.batch()`) constructs a dataset
from one or more `tf.data.Dataset` objects.
* A `tf.data.Iterator` provides the main way to extract elements from a
@@ -322,9 +322,39 @@ sess.run(iterator.initializer)
next1, (next2, next3) = iterator.get_next()
```
-Note that evaluating *any* of `next1`, `next2`, or `next3` will advance the
-iterator for all components. A typical consumer of an iterator will include all
-components in a single expression.
+Note that `next1`, `next2`, and `next3` are tensors produced by the
+same op/node (created by `Iterator.get_next()`). Therefore, evaluating *any* of
+these tensors will advance the iterator for all components. A typical consumer
+of an iterator will include all components in a single expression.
+
+### Saving iterator state
+
+The @{tf.contrib.data.make_saveable_from_iterator} function creates a
+`SaveableObject` from an iterator, which can be used to save and
+restore the current state of the iterator (and, effectively, the whole input
+pipeline). A saveable object thus created can be added to @{tf.train.Saver}
+variables list or the `tf.GraphKeys.SAVEABLE_OBJECTS` collection for saving and
+restoring in the same manner as a @{tf.Variable}. Refer to
+@{$saved_model$Saving and Restoring} for details on how to save and restore
+variables.
+
+```python
+# Create saveable object from iterator.
+saveable = tf.contrib.data.make_saveable_from_iterator(iterator)
+
+# Save the iterator state by adding it to the saveable objects collection.
+tf.add_to_collection(tf.GraphKeys.SAVEABLE_OBJECTS, saveable)
+saver = tf.train.Saver()
+
+with tf.Session() as sess:
+
+ if should_checkpoint:
+ saver.save(path_to_checkpoint)
+
+# Restore the iterator state.
+with tf.Session() as sess:
+ saver.restore(sess, path_to_checkpoint)
+```
## Reading input data
diff --git a/programmers_guide/debugger.md b/programmers_guide/debugger.md
index dbc4517..f5a0eb0 100644
--- a/programmers_guide/debugger.md
+++ b/programmers_guide/debugger.md
@@ -4,27 +4,31 @@
[TOC]
-TensorFlow debugger (**tfdbg**) is a specialized debugger for TensorFlow. It
-lets you view the internal structure and states of running TensorFlow graphs
-during training and inference, which is difficult to debug with general-purpose
-debuggers such as Python's `pdb` due to TensorFlow's computation-graph paradigm.
-
-> NOTE: TensorFlow debugger uses a
-> [curses](https://en.wikipedia.org/wiki/Curses_\(programming_library\))-based
-> text user interface. On Mac OS X, the `ncurses` library is required and can
-> be installed with `brew install homebrew/dupes/ncurses`. On Windows, curses
-> isn't as well supported, so a
-> [readline](https://en.wikipedia.org/wiki/GNU_Readline)-based interface can
-> be used with tfdbg by installing `pyreadline` with pip.
-> If you use Anaconda3, you can install it with a command
-> such as `"C:\Program Files\Anaconda3\Scripts\pip.exe" install pyreadline`.
-> Unofficial Windows curses packages can be downloaded
-> [here](https://www.lfd.uci.edu/~gohlke/pythonlibs/#curses), then subsequently
-> installed using `pip install