Adding auto doc links for READMEs.

Jon Wayne Parrott · Jon Wayne Parrott · commit b9cd3b03c3fa · 2015-11-13T11:59:31.000-08:00
diff --git a/appengine/bigquery/README.md b/appengine/bigquery/README.md
@@ -2,6 +2,15 @@
 
 This sample demonstrates [authenticating to BigQuery in App Engine using OAuth2](https://cloud.google.com/bigquery/authentication).
 
+<!-- auto-doc-link -->
+These samples are used on the following documentation pages:
+
+>
+* https://cloud.google.com/bigquery/authentication
+* https://cloud.google.com/monitoring/api/authentication
+
+<!-- end-auto-doc-link -->
+
 Refer to the [App Engine Samples README](../README.md) for information on how to run and deploy this sample.
 
 ## Setup
diff --git a/appengine/blobstore/README.md b/appengine/blobstore/README.md
@@ -0,0 +1,9 @@
+# App Engine Blobstore Sample
+
+<!-- auto-doc-link -->
+These samples are used on the following documentation page:
+
+> https://cloud.google.com/appengine/docs/python/blobstore/
+
+<!-- end-auto-doc-link -->
+ -->
diff --git a/appengine/images/README.md b/appengine/images/README.md
@@ -3,4 +3,11 @@
 This is a sample app for Google App Engine that demonstrates the [Images Python
 API](https://cloud.google.com/appengine/docs/python/images/usingimages).
 
+<!-- auto-doc-link -->
+These samples are used on the following documentation page:
+
+> https://cloud.google.com/appengine/docs/python/images/usingimages
+
+<!-- end-auto-doc-link -->
+
 Refer to the [App Engine Samples README](../README.md) for information on how to run and deploy this sample.
diff --git a/appengine/localtesting/README.md b/appengine/localtesting/README.md
@@ -0,0 +1,10 @@
+# App Engine Local Testing Samples
+
+These samples show how to do automated testing of App Engine applications.
+
+<!-- auto-doc-link -->
+These samples are used on the following documentation page:
+
+> https://cloud.google.com/appengine/docs/python/tools/localunittesting
+
+<!-- end-auto-doc-link -->
diff --git a/appengine/memcache/guestbook/README.md b/appengine/memcache/guestbook/README.md
@@ -1,5 +1,12 @@
 # Memcache Guestbook Sample
 
-This is a sample app for Google App Engine that demonstrates the [Memcache Python API](https://cloud.google.com/appengine/docs/python/memcache/usingmemcache).
+This is a sample app for Google App Engine that demonstrates the Memcache Python API.
+
+<!-- auto-doc-link -->
+These samples are used on the following documentation page:
+
+> https://cloud.google.com/appengine/docs/python/memcache/usingmemcache
+
+<!-- end-auto-doc-link -->
 
 Refer to the [App Engine Samples README](../../README.md) for information on how to run and deploy this sample.
diff --git a/appengine/multitenancy/README.md b/appengine/multitenancy/README.md
@@ -2,4 +2,11 @@
 
 This sample demonstrates how to use Google App Engine's [Namespace Manager API](https://cloud.google.com/appengine/docs/python/multitenancy/multitenancy).
 
+<!-- auto-doc-link -->
+These samples are used on the following documentation page:
+
+> https://cloud.google.com/appengine/docs/python/multitenancy/namespaces
+
+<!-- end-auto-doc-link -->
+
 Refer to the [App Engine Samples README](../README.md) for information on how to run and deploy this sample.
diff --git a/appengine/ndb/overview/README.md b/appengine/ndb/overview/README.md
@@ -2,4 +2,11 @@
 
 This is a sample app for Google App Engine that demonstrates the [Datastore NDB Python API](https://cloud.google.com/appengine/docs/python/ndb/).
 
+<!-- auto-doc-link -->
+These samples are used on the following documentation page:
+
+> https://cloud.google.com/appengine/docs/python/ndb/
+
+<!-- end-auto-doc-link -->
+
 Refer to the [App Engine Samples README](../../README.md) for information on how to run and deploy this sample.
diff --git a/appengine/ndb/transactions/README.md b/appengine/ndb/transactions/README.md
@@ -1,7 +1,14 @@
 ## App Engine Datastore NDB Transactions Sample
 
-This is a sample app for Google App Engine that exercises the [NDB Transactions Python API](https://cloud.google.com/appengine/docs/python/ndb/transactions)
+This is a sample app for Google App Engine that demonstrates the [NDB Transactions Python API](https://cloud.google.com/appengine/docs/python/ndb/transactions)
 
 This app presents a list of notes. After you submit a note with a particular title, you may not change that note or submit a new note with the same title. There are multiple note pages available.
 
+<!-- auto-doc-link -->
+These samples are used on the following documentation page:
+
+> https://cloud.google.com/appengine/docs/python/ndb/transactions
+
+<!-- end-auto-doc-link -->
+
 Refer to the [App Engine Samples README](../../README.md) for information on how to run and deploy this sample.
diff --git a/bigquery/api/README.md b/bigquery/api/README.md
@@ -0,0 +1,16 @@
+# BigQuery API Samples
+
+<!-- auto-doc-link -->
+These samples are used on the following documentation pages:
+
+>
+* https://cloud.google.com/bigquery/exporting-data-from-bigquery
+* https://cloud.google.com/bigquery/authentication
+* https://cloud.google.com/bigquery/bigquery-api-quickstart
+* https://cloud.google.com/bigquery/docs/managing_jobs_datasets_projects
+* https://cloud.google.com/bigquery/streaming-data-into-bigquery
+* https://cloud.google.com/bigquery/docs/data
+* https://cloud.google.com/bigquery/querying-data
+* https://cloud.google.com/bigquery/loading-data-into-bigquery
+
+<!-- end-auto-doc-link -->
diff --git a/compute/api/README.md b/compute/api/README.md
@@ -0,0 +1,13 @@
+# Compute Engine API Samples
+
+<!-- auto-doc-link -->
+These samples are used on the following documentation page:
+
+> https://cloud.google.com/compute/docs/tutorials/python-guide
+
+<!-- end-auto-doc-link -->
+
+
+
+
+
diff --git a/monitoring/api/README.md b/monitoring/api/README.md
@@ -0,0 +1,10 @@
+# Cloud Monitoring API Samples
+
+<!-- auto-doc-link -->
+These samples are used on the following documentation pages:
+
+>
+* https://cloud.google.com/monitoring/demos/
+* https://cloud.google.com/monitoring/api/authentication
+
+<!-- end-auto-doc-link -->
diff --git a/scripts/README.md b/scripts/README.md
@@ -0,0 +1 @@
+These scripts are used for the maintenance of this repository and are not necessarily meant as samples.
diff --git a/scripts/auto_link_to_docs.py b/scripts/auto_link_to_docs.py
@@ -0,0 +1,144 @@
+#!/usr/bin/env python
+
+# Copyright (C) 2013 Google Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#            http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+Process docs-links.json and updates all READMEs and replaces
+
+<!-- auto-doc-link --><!-- end-auto-doc-link -->
+
+With a generated list of documentation backlinks.
+"""
+
+from collections import defaultdict
+import json
+import os
+import re
+
+
+REPO_ROOT = os.path.abspath(os.path.join(
+    os.path.dirname(__file__),
+    '..'))
+DOC_SITE_ROOT = 'https://cloud.google.com/'
+AUTO_DOC_LINK_EXP = re.compile(
+    r'<!-- auto-doc-link -->.*?<!-- end-auto-doc-link -->\n',
+    re.DOTALL)
+
+
+def invert_docs_link_map(docs_links):
+    """
+    The docs links map is in this format:
+
+        {
+            "doc_path": [
+                "file_path",
+            ]
+        }
+
+    This transforms it to:
+
+        {
+            "file_path": [
+                "doc_path",
+            ]
+        }
+    """
+    files_to_docs = defaultdict(list)
+    for doc, files in docs_links.iteritems():
+        for file in files:
+            files_to_docs[file].append(doc)
+            files_to_docs[file] = list(set(files_to_docs[file]))
+
+    return files_to_docs
+
+
+def collect_docs_for_readmes(files_to_docs):
+    """
+    There's a one-to-many relationship between readmes and files. This method
+    finds the readme for each file and consolidates all docs references.
+    """
+    readmes_to_docs = defaultdict(list)
+    for file, docs in files_to_docs.iteritems():
+        readme = get_readme_path(file)
+        readmes_to_docs[readme].extend(docs)
+        readmes_to_docs[readme] = list(set(readmes_to_docs[readme]))
+    return readmes_to_docs
+
+
+def linkify(docs):
+    """Adds the documentation site root to doc paths, creating a full URL."""
+    return [DOC_SITE_ROOT + x for x in docs]
+
+
+def replace_contents(file_path, regex, new_content):
+    with open(file_path, 'r+') as f:
+        content = f.read()
+        content = regex.sub(new_content, content)
+        f.seek(0)
+        f.write(content)
+
+
+def get_readme_path(file_path):
+    """Gets the readme for an associated sample file, basically just the
+    README.md in the same directory."""
+    dir = os.path.dirname(file_path)
+    readme = os.path.join(
+        REPO_ROOT, dir, 'README.md')
+    return readme
+
+
+def generate_doc_link_statement(docs):
+    links = linkify(docs)
+    if len(links) == 1:
+        return """<!-- auto-doc-link -->
+These samples are used on the following documentation page:
+
+> {}
+
+<!-- end-auto-doc-link -->
+""".format(links.pop())
+    else:
+        return """<!-- auto-doc-link -->
+These samples are used on the following documentation pages:
+
+>
+{}
+
+<!-- end-auto-doc-link -->
+""".format('\n'.join(['* {}'.format(x) for x in links]))
+
+
+def update_readme(readme_path, docs):
+    if not os.path.exists(readme_path):
+        print('{} doesn\'t exist'.format(readme_path))
+        return
+    replace_contents(
+        readme_path,
+        AUTO_DOC_LINK_EXP,
+        generate_doc_link_statement(docs))
+    print('Updated {}'.format(readme_path))
+
+
+def main():
+    docs_links = json.load(open(
+        os.path.join(REPO_ROOT, 'scripts', 'docs-links.json'), 'r'))
+    files_to_docs = invert_docs_link_map(docs_links)
+    readmes_to_docs = collect_docs_for_readmes(files_to_docs)
+
+    for readme, docs in readmes_to_docs.iteritems():
+        update_readme(readme, docs)
+
+if __name__ == '__main__':
+    main()
diff --git a/scripts/docs-links.json b/scripts/docs-links.json
@@ -0,0 +1,87 @@
+{
+  "/appengine/docs/python/ndb/": [
+    "appengine/ndb/overview/main.py"
+  ], 
+  "/storage/transfer/create-client": [
+    "storage/transfer_service/create_client.py"
+  ], 
+  "/storage/docs/json_api/v1/json-api-python-samples": [
+    "storage/api/list_objects.py"
+  ], 
+  "/bigquery/querying-data": [
+    "bigquery/api/async_query.py", 
+    "bigquery/api/sync_query.py"
+  ], 
+  "/bigquery/bigquery-api-quickstart": [
+    "bigquery/api/getting_started.py"
+  ], 
+  "/bigquery/docs/managing_jobs_datasets_projects": [
+    "bigquery/api/list_datasets_projects.py"
+  ], 
+  "/appengine/docs/python/blobstore/": [
+    "appengine/blobstore/main.py"
+  ], 
+  "/appengine/docs/python/images/usingimages": [
+    "appengine/images/main.py"
+  ], 
+  "/compute/docs/tutorials/python-guide": [
+    "compute/api/create_instance.py", 
+    "compute/api/startup-script.sh"
+  ], 
+  "/docs/authentication": [
+    "storage/api/list_objects.py"
+  ], 
+  "/bigquery/loading-data-into-bigquery": [
+    "bigquery/api/load_data_from_csv.py", 
+    "bigquery/api/load_data_by_post.py"
+  ], 
+  "/bigquery/streaming-data-into-bigquery": [
+    "bigquery/api/streaming.py"
+  ], 
+  "/appengine/docs/python/memcache/usingmemcache": [
+    "appengine/memcache/guestbook/main.py"
+  ], 
+  "/monitoring/demos/": [
+    "monitoring/api/auth.py"
+  ], 
+  "/appengine/docs/python/multitenancy/namespaces": [
+    "appengine/multitenancy/memcache.py", 
+    "appengine/multitenancy/datastore.py", 
+    "appengine/multitenancy/taskqueue.py"
+  ], 
+  "/bigquery/docs/data": [
+    "bigquery/api/sync_query.py"
+  ], 
+  "/bigquery/authentication": [
+    "appengine/bigquery/main.py", 
+    "bigquery/api/getting_started.py"
+  ], 
+  "/bigquery/exporting-data-from-bigquery": [
+    "bigquery/api/export_data_to_cloud_storage.py"
+  ], 
+  "/appengine/docs/python/ndb/transactions": [
+    "appengine/ndb/transactions/main.py"
+  ], 
+  "/monitoring/api/authentication": [
+    "appengine/bigquery/app.yaml", 
+    "monitoring/api/auth.py", 
+    "appengine/bigquery/main.py"
+  ], 
+  "/storage/transfer/create-transfer": [
+    "storage/transfer_service/nearline_request.py", 
+    "storage/transfer_service/aws_request.py", 
+    "storage/transfer_service/transfer_check.py"
+  ], 
+  "/storage/docs/authentication": [
+    "storage/api/list_objects.py"
+  ], 
+  "/appengine/docs/python/tools/localunittesting": [
+    "appengine/localtesting/runner.py", 
+    "appengine/localtesting/test_task_queue.py", 
+    "appengine/localtesting/test_login.py", 
+    "appengine/localtesting/queue.yaml", 
+    "appengine/localtesting/test_mail.py", 
+    "appengine/localtesting/test_env_vars.py", 
+    "appengine/localtesting/test_datastore.py"
+  ]
+}
diff --git a/storage/api/README.md b/storage/api/README.md
@@ -0,0 +1,11 @@
+# Cloud Storage API Samples
+
+<!-- auto-doc-link -->
+These samples are used on the following documentation pages:
+
+>
+* https://cloud.google.com/storage/docs/json_api/v1/json-api-python-samples
+* https://cloud.google.com/storage/docs/authentication
+* https://cloud.google.com/docs/authentication
+
+<!-- end-auto-doc-link -->
diff --git a/storage/transfer_service/README.md b/storage/transfer_service/README.md
