From 5808366e7cf870be642840dc8d56c678da0dc22e Mon Sep 17 00:00:00 2001 From: wesley chun Date: Wed, 23 Jun 2021 10:59:23 -0700 Subject: [PATCH 01/56] add video to README --- README.md | 32 ++++++++++++++++---------------- 1 file changed, 16 insertions(+), 16 deletions(-) diff --git a/README.md b/README.md index 93db30a..1fdeecc 100644 --- a/README.md +++ b/README.md @@ -1,12 +1,12 @@ # Python App Engine app migration ### To modern runtime, Cloud services, Python 3, and Cloud Run containers -[Google App Engine](https://cloud.google.com/appengine) (Standard) has undergone significant changes between the legacy and next generation platforms. To address this, we've created a set of codelabs (free, online, self-paced, hands-on tutorials) to show developers how to perform individual migrations they can apply to modernize their apps for the latest runtimes, with this repo managing the samples from those codelabs. +[Google App Engine](https://cloud.google.com/appengine) (Standard) has undergone significant changes between the legacy and next generation platforms. To address this, we've created a set of codelabs (free, online, self-paced, hands-on tutorials) and corresponding videos (when available) to show developers how to perform individual migrations they can apply to modernize their apps for the latest runtimes, with this repo managing the samples from those codelabs. Each codelab begins with a "START" code base then walks developers through that migration step, resulting in a "FINISH" repo. If you make any mistakes along the way, you can always go back to START or compare your code with what's in the FINISH folder to see the differences. We also want to help you port to the Python 3 runtime, so some codelabs contain a bonus section for that purpose. > **NOTE:** These migrations are *typically* aimed at Python 2 users -> 1. *Python 3.x App Engine users*: You're *already* on the next-gen platform, so only for **non**-legacy service migrations +> 1. *Python 3.x App Engine users*: You're *already* on the next-gen platform, so only look for **non**-legacy service migrations > 1. *Python 2.5 App Engine users*: to revive apps from the original 2.5 runtime, [deprecated in 2013](http://googleappengine.blogspot.com/2013/03/python-25-thanks-for-good-times.html) and [shutdown in 2017](https://cloud.google.com/appengine/docs/standard/python/python25), you must [migrate from `db` to `ndb`](http://cloud.google.com/appengine/docs/standard/python/ndb/db_to_ndb) and get those apps running on Python 2.7 before attempting these migrations. @@ -67,20 +67,20 @@ The table below summarizes migration module resources currently available along ### Summary table -Module | Topic | Codelab | START here | FINISH here ---- | --- | --- | --- | --- -0|Baseline app| _N/A_ (no tutorial; just review the code) | _N/A_ | Module 0 [code](/mod0-baseline) (2.x) -1|Migrate to Flask| [link](http://g.co/codelabs/pae-migrate-flask) | Module 0 [code](/mod0-baseline) (2.x) | Module 1 [code](/mod1-flask) (2.x) -2|Migrate to Cloud NDB| [link](http://g.co/codelabs/pae-migrate-cloudndb) | Module 1 [code](/mod1-flask) (2.x) | Module 2 [code](/mod2a-cloudndb) (2.x) & [code](/mod2b-cloudndb) (3.x) -3|Migrate to Cloud Datastore| [link](http://g.co/codelabs/pae-migrate-datastore) | Module 2 [code](/mod2a-cloudndb) (2.x) & [code](/mod2b-cloudndb) (3.x) | Module 3 [code](/mod3a-datastore) (2.x) & [code](/mod3b-datastore) (3.x) -4|Migrate to Cloud Run with Docker| [link](http://g.co/codelabs/pae-migrate-rundocker) | Module 2 [code](/mod2a-cloudndb) (2.x) & Module 3 [code](/mod3b-datastore) (3.x) | Module 4 [code](/mod4a-rundocker) (2.x) & [code](/mod4b-rundocker) (3.x) -5|Migrate to Cloud Run with Buildpacks| [link](http://g.co/codelabs/pae-migrate-runbldpks) | Module 2 [code](/mod2b-cloudndb) (3.x) | Module 5 [code](/mod5-runbldpks) (3.x) -6|Migrate to Cloud Firestore (app)| [link](http://g.co/codelabs/pae-migrate-firestore) | Module 3 [code](/mod3b-datastore) (3.x) | Module 6 [code](/mod6-firestore) (3.x) -7|Add App Engine push tasks| [link](http://g.co/codelabs/pae-migrate-gaetasks) | Module 1 [code]() (2.x) | Module 7 [code](/mod7-gaetasks) (2.x) -8|Migrate to Cloud Tasks| [link](http://g.co/codelabs/pae-migrate-cloudtasks) | Module 7 [code](/mod7-gaetasks) (2.x) | Module 8 [code](/mod8-cloudtasks) (2.x) -9|Migrate to Python 3 (Cloud Datastore & Cloud Tasks v2)| [link](http://g.co/codelabs/pae-migrate-py3dstasks) | Module 8 [code](/mod8-cloudtasks) (2.x) | Module 9 [code](/mod9-py3dstasks) (3.x) -10|Migrate to Cloud Firestore (data)| _N/A_ | _N/A_ | _TBD_ -11|Migrate to Cloud Functions| _TBD_ | Module 2 [code](/mod2b-cloudndb) (3.x) | Module 11 [code](/mod11-functions) (3.x) +Module | Topic | Video | Codelab | START here | FINISH here +--- | --- | --- | --- | --- | --- +0|Baseline app| [link](http://twitter.com/googledevs/status/1407755281786867714?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_smsintro_201023&utm_content=-) | _N/A_ (no tutorial; just review the code) | _N/A_ | Module 0 [code](/mod0-baseline) (2.x) +1|Migrate to Flask| _TBD_ | [link](http://g.co/codelabs/pae-migrate-flask) | Module 0 [code](/mod0-baseline) (2.x) | Module 1 [code](/mod1-flask) (2.x) +2|Migrate to Cloud NDB| _TBD_ | [link](http://g.co/codelabs/pae-migrate-cloudndb) | Module 1 [code](/mod1-flask) (2.x) | Module 2 [code](/mod2a-cloudndb) (2.x) & [code](/mod2b-cloudndb) (3.x) +3|Migrate to Cloud Datastore| _TBD_ | [link](http://g.co/codelabs/pae-migrate-datastore) | Module 2 [code](/mod2a-cloudndb) (2.x) & [code](/mod2b-cloudndb) (3.x) | Module 3 [code](/mod3a-datastore) (2.x) & [code](/mod3b-datastore) (3.x) +4|Migrate to Cloud Run with Docker| _TBD_ | [link](http://g.co/codelabs/pae-migrate-rundocker) | Module 2 [code](/mod2a-cloudndb) (2.x) & Module 3 [code](/mod3b-datastore) (3.x) | Module 4 [code](/mod4a-rundocker) (2.x) & [code](/mod4b-rundocker) (3.x) +5|Migrate to Cloud Run with Buildpacks| _TBD_ | [link](http://g.co/codelabs/pae-migrate-runbldpks) | Module 2 [code](/mod2b-cloudndb) (3.x) | Module 5 [code](/mod5-runbldpks) (3.x) +6|Migrate to Cloud Firestore (app)| _TBD_ | [link](http://g.co/codelabs/pae-migrate-firestore) | Module 3 [code](/mod3b-datastore) (3.x) | Module 6 [code](/mod6-firestore) (3.x) +7|Add App Engine push tasks| _TBD_ | [link](http://g.co/codelabs/pae-migrate-gaetasks) | Module 1 [code]() (2.x) | Module 7 [code](/mod7-gaetasks) (2.x) +8|Migrate to Cloud Tasks| _TBD_ | [link](http://g.co/codelabs/pae-migrate-cloudtasks) | Module 7 [code](/mod7-gaetasks) (2.x) | Module 8 [code](/mod8-cloudtasks) (2.x) +9|Migrate to Python 3 (Cloud Datastore & Cloud Tasks v2)| _TBD_ | [link](http://g.co/codelabs/pae-migrate-py3dstasks) | Module 8 [code](/mod8-cloudtasks) (2.x) | Module 9 [code](/mod9-py3dstasks) (3.x) +10|Migrate to Cloud Firestore (data)| _TBD_ | _N/A_ | _N/A_ | _TBD_ +11|Migrate to Cloud Functions| _TBD_ | _TBD_ | Module 2 [code](/mod2b-cloudndb) (3.x) | Module 11 [code](/mod11-functions) (3.x) ### Table of contents From 3c6eff6a85dabb001509d1b98afdbe3636195bb1 Mon Sep 17 00:00:00 2001 From: wesley chun Date: Tue, 29 Jun 2021 00:01:46 -0700 Subject: [PATCH 02/56] rev pkg versions, switch Mod9 from DS to FS --- mod6-firestore/requirements.txt | 2 +- mod8-cloudtasks/main.py | 3 +- mod9-py3dstasks/README.md | 2 +- mod9-py3dstasks/main.py | 60 ++++++++++++++++++++------------ mod9-py3dstasks/requirements.txt | 2 +- 5 files changed, 43 insertions(+), 26 deletions(-) diff --git a/mod6-firestore/requirements.txt b/mod6-firestore/requirements.txt index 72ec3d4..9602ece 100644 --- a/mod6-firestore/requirements.txt +++ b/mod6-firestore/requirements.txt @@ -1,2 +1,2 @@ flask==1.1.2 -google-cloud-firestore==2.0.2 +google-cloud-firestore==2.1.3 diff --git a/mod8-cloudtasks/main.py b/mod8-cloudtasks/main.py index 9150967..abd5876 100644 --- a/mod8-cloudtasks/main.py +++ b/mod8-cloudtasks/main.py @@ -17,13 +17,14 @@ import logging import time from flask import Flask, render_template, request +import google.auth from google.cloud import ndb, tasks app = Flask(__name__) ds_client = ndb.Client() ts_client = tasks.CloudTasksClient() -PROJECT_ID = 'PROJECT_ID' # replace w/your own +_, PROJECT_ID = google.auth.default() REGION_ID = 'REGION_ID' # replace w/your own QUEUE_NAME = 'default' # replace w/your own QUEUE_PATH = ts_client.queue_path(PROJECT_ID, REGION_ID, QUEUE_NAME) diff --git a/mod9-py3dstasks/README.md b/mod9-py3dstasks/README.md index 613ea03..9513ffd 100644 --- a/mod9-py3dstasks/README.md +++ b/mod9-py3dstasks/README.md @@ -1,3 +1,3 @@ # Module 9 - Migrate from Python 2 to 3 and Cloud NDB to Cloud Datastore -This repo folder is the corresponding Python 3 code to the [Module 9 codelab](http://g.co/codelabs/pae-migrate-py3dstasks). The tutorial STARTs with the Python 2 code in the [Module 8 repo folder](/mod7-cloudtasks) and leads developers through migrating from Python 2 to 3, Cloud NDB to Cloud Datastore (plus any changes from Cloud Tasks v1 to v2), culminating in the code in this folder. +This repo folder is the corresponding Python 3 code to the [Module 9 codelab](http://g.co/codelabs/pae-migrate-py3dstasks). The tutorial STARTs with the Python 2 code in the [Module 8 repo folder](/mod7-cloudtasks) and leads developers through migrating from Python 2 to 3, Cloud NDB to Cloud Firestore (plus any changes from Cloud Tasks v1 to v2), culminating in the code in this folder. diff --git a/mod9-py3dstasks/main.py b/mod9-py3dstasks/main.py index e4222cf..9720083 100644 --- a/mod9-py3dstasks/main.py +++ b/mod9-py3dstasks/main.py @@ -16,32 +16,43 @@ import json import time from flask import Flask, render_template, request -from google.cloud import datastore, tasks +import google.auth +from google.cloud import firestore, tasks app = Flask(__name__) -ds_client = datastore.Client() +fs_client = firestore.Client() ts_client = tasks.CloudTasksClient() -PROJECT_ID = 'PROJECT_ID' # replace w/your own +_, PROJECT_ID = google.auth.default() REGION_ID = 'REGION_ID' # replace w/your own QUEUE_NAME = 'default' # replace w/your own QUEUE_PATH = ts_client.queue_path(PROJECT_ID, REGION_ID, QUEUE_NAME) +PATH_PREFIX = QUEUE_PATH.rsplit('/', 2)[0] def store_visit(remote_addr, user_agent): - 'create new Visit entity in Datastore' - entity = datastore.Entity(key=ds_client.key('Visit')) - entity.update({ + 'create new Visit document in Firestore' + doc_ref = fs_client.collection('Visit') + doc_ref.add({ 'timestamp': datetime.now(), 'visitor': '{}: {}'.format(remote_addr, user_agent), }) - ds_client.put(entity) + +def create_queue_if(): + 'app-internal function creating default queue if it does not exist' + try: + ts_client.get_queue(name=QUEUE_PATH) + except Exception as e: + if 'does not exist' in str(e): + ts_client.create_queue(parent=PATH_PREFIX, + queue={'name': QUEUE_PATH}) + return True def fetch_visits(limit): 'get most recent visits & add task to delete older visits' - query = ds_client.query(kind='Visit') - query.order = ['-timestamp'] - data = list(query.fetch(limit=limit)) - oldest = time.mktime(data[-1]['timestamp'].timetuple()) + visits_ref = fs_client.collection('Visit') + visits = list(v.to_dict() for v in visits_ref.order_by('timestamp', + direction=firestore.Query.DESCENDING).limit(limit).stream()) + oldest = time.mktime(visits[-1]['timestamp'].timetuple()) oldest_str = time.ctime(oldest) print('Delete entities older than %s' % oldest_str) task = { @@ -53,22 +64,27 @@ def fetch_visits(limit): }, } } - ts_client.create_task(parent=QUEUE_PATH, task=task) - return data, oldest_str + if create_queue_if(): + ts_client.create_task(parent=QUEUE_PATH, task=task) + return visits, oldest_str + +def _delete_docs(visits): + 'app-internal generator deleteing all old FS visit documents' + for visit in visits: + visit.reference.delete() + yield visit.id @app.route('/trim', methods=['POST']) def trim(): '(push) task queue handler to delete oldest visits' oldest = float(request.get_json().get('oldest')) - query = ds_client.query(kind='Visit') - query.add_filter('timestamp', '<', datetime.fromtimestamp(oldest)) - query.keys_only() - keys = list(visit.key for visit in query.fetch()) - nkeys = len(keys) - if nkeys: - print('Deleting %d entities: %s' % ( - nkeys, ', '.join(str(k.id) for k in keys))) - ds_client.delete_multi(keys) + query = fs_client.collection('Visit') + visits = list(query.where('timestamp', '<', + datetime.fromtimestamp(oldest)).stream()) + nvisits = len(visits) + if nvisits: + print('Deleting %d entities: ' % nvisits, end='') + print(', '.join(str(v_id) for v_id in _delete_docs(visits))) else: print('No entities older than: %s' % time.ctime(oldest)) return '' # need to return SOME string w/200 diff --git a/mod9-py3dstasks/requirements.txt b/mod9-py3dstasks/requirements.txt index acc1e1b..ed5a882 100644 --- a/mod9-py3dstasks/requirements.txt +++ b/mod9-py3dstasks/requirements.txt @@ -1,3 +1,3 @@ flask==1.1.2 -google-cloud-datastore==2.1.3 +google-cloud-firestore==2.1.3 google-cloud-tasks==2.3.0 From 341e58cfd602603a23e63279f51c40c6d5e60f2a Mon Sep 17 00:00:00 2001 From: wesley chun Date: Tue, 29 Jun 2021 00:12:12 -0700 Subject: [PATCH 03/56] change Mod9 from DS to FS --- README.md | 14 ++++++-------- mod9-py3dstasks/README.md | 3 --- {mod9-py3dstasks => mod9-py3fstasks}/.gcloudignore | 0 mod9-py3fstasks/README.md | 3 +++ {mod9-py3dstasks => mod9-py3fstasks}/app.yaml | 0 {mod9-py3dstasks => mod9-py3fstasks}/main.py | 0 .../requirements.txt | 0 .../templates/index.html | 0 8 files changed, 9 insertions(+), 11 deletions(-) delete mode 100644 mod9-py3dstasks/README.md rename {mod9-py3dstasks => mod9-py3fstasks}/.gcloudignore (100%) create mode 100644 mod9-py3fstasks/README.md rename {mod9-py3dstasks => mod9-py3fstasks}/app.yaml (100%) rename {mod9-py3dstasks => mod9-py3fstasks}/main.py (100%) rename {mod9-py3dstasks => mod9-py3fstasks}/requirements.txt (100%) rename {mod9-py3dstasks => mod9-py3fstasks}/templates/index.html (100%) diff --git a/README.md b/README.md index 1fdeecc..ca8b522 100644 --- a/README.md +++ b/README.md @@ -78,7 +78,7 @@ Module | Topic | Video | Codelab | START here | FINISH here 6|Migrate to Cloud Firestore (app)| _TBD_ | [link](http://g.co/codelabs/pae-migrate-firestore) | Module 3 [code](/mod3b-datastore) (3.x) | Module 6 [code](/mod6-firestore) (3.x) 7|Add App Engine push tasks| _TBD_ | [link](http://g.co/codelabs/pae-migrate-gaetasks) | Module 1 [code]() (2.x) | Module 7 [code](/mod7-gaetasks) (2.x) 8|Migrate to Cloud Tasks| _TBD_ | [link](http://g.co/codelabs/pae-migrate-cloudtasks) | Module 7 [code](/mod7-gaetasks) (2.x) | Module 8 [code](/mod8-cloudtasks) (2.x) -9|Migrate to Python 3 (Cloud Datastore & Cloud Tasks v2)| _TBD_ | [link](http://g.co/codelabs/pae-migrate-py3dstasks) | Module 8 [code](/mod8-cloudtasks) (2.x) | Module 9 [code](/mod9-py3dstasks) (3.x) +9|Migrate to Python 3, Cloud Firestore & Cloud Tasks v2| _TBD_ | _TBD_ | Module 8 [code](/mod8-cloudtasks) (2.x) | Module 9 [code](/mod9-py3fstasks) (3.x) 10|Migrate to Cloud Firestore (data)| _TBD_ | _N/A_ | _N/A_ | _TBD_ 11|Migrate to Cloud Functions| _TBD_ | _TBD_ | Module 2 [code](/mod2b-cloudndb) (3.x) | Module 11 [code](/mod11-functions) (3.x) @@ -130,20 +130,18 @@ If there is a logical codelab to do immediately after completing one, they will - FINISH: [Module 8 code - Cloud Tasks](/mod8-cloudtasks) (2.x) - NEXT: Module 9 - migrate to Python 3 and Cloud Datastore -- [Module 9 codelab](http://g.co/codelabs/pae-migrate-py3dstasks): **Migrate a Python 2 Cloud NDB & Cloud Tasks app to a Python 3 Cloud Datastore app** - - **Mixed migration recommendation** - - Migrating to Python 3 is required, but... - - Migrating to Cloud Datastore is optional as Cloud NDB works on 3.x; it's to give you the experience of doing it - - This codelab includes the [Module 3 migration codelab](http://g.co/codelabs/pae-migrate-datastore), so skip if you complete this one +- **Module 9 codelab** (TBD): **Migrate a Python 2 Cloud NDB & Cloud Tasks (v1) app to a Python 3 Cloud Firestore & Cloud Tasks (v2) app** + - **Optional** migrations + - Migrating to Python 3 is not required but recommended as Python 2 has been sunset + - Migrating to Cloud Firestore is *very* optional as Cloud NDB works on 3.x and most importantly, Cloud Firestore requires a completely new GCP project - Python 2 - START: [Module 8 code - Cloud Tasks](/mod8-cloudtasks) (2.x) - Python 3 - - FINISH: [Module 9 code - Cloud Datastore & Tasks](/mod9-py3dstasks) (3.x) + - FINISH: [Module 9 code - Cloud Firestore & Tasks](/mod9-py3fstasks) (3.x) - RECOMMENDED: - Module 11 - migrate to Cloud Functions - Module 5 - migrate to Cloud Run container with Cloud Buildpacks - Module 4 - migrate to Cloud Run container with Docker - - Module 6 - migrate to Cloud Firestore (app) - [Module 4 codelab](http://g.co/codelabs/pae-migrate-rundocker): **Migrate from App Engine to [Cloud Run](http://cloud.google.com/run) with Docker** - **Optional** migration diff --git a/mod9-py3dstasks/README.md b/mod9-py3dstasks/README.md deleted file mode 100644 index 9513ffd..0000000 --- a/mod9-py3dstasks/README.md +++ /dev/null @@ -1,3 +0,0 @@ -# Module 9 - Migrate from Python 2 to 3 and Cloud NDB to Cloud Datastore - -This repo folder is the corresponding Python 3 code to the [Module 9 codelab](http://g.co/codelabs/pae-migrate-py3dstasks). The tutorial STARTs with the Python 2 code in the [Module 8 repo folder](/mod7-cloudtasks) and leads developers through migrating from Python 2 to 3, Cloud NDB to Cloud Firestore (plus any changes from Cloud Tasks v1 to v2), culminating in the code in this folder. diff --git a/mod9-py3dstasks/.gcloudignore b/mod9-py3fstasks/.gcloudignore similarity index 100% rename from mod9-py3dstasks/.gcloudignore rename to mod9-py3fstasks/.gcloudignore diff --git a/mod9-py3fstasks/README.md b/mod9-py3fstasks/README.md new file mode 100644 index 0000000..4b636ca --- /dev/null +++ b/mod9-py3fstasks/README.md @@ -0,0 +1,3 @@ +# Module 9 - Migrate from Python 2 to 3 and Cloud NDB to Cloud Firestore + +This repo folder is the corresponding Python 3 code to the [Module 9 codelab](http://g.co/codelabs/pae-migrate-py3fstasks). The tutorial STARTs with the Python 2 code in the [Module 8 repo folder](/mod7-cloudtasks) and leads developers through migrating from Python 2 to 3, Cloud NDB to Cloud Firestore (skipping over a Cloud Datstore migration) plus any changes from Cloud Tasks v1 to v2, culminating in the code in this folder. One major addition to look for here vs. Module 8 is that App Engine `taskqueue` creates a `default` push queue while Cloud Tasks does not, so that now has to be done in code. diff --git a/mod9-py3dstasks/app.yaml b/mod9-py3fstasks/app.yaml similarity index 100% rename from mod9-py3dstasks/app.yaml rename to mod9-py3fstasks/app.yaml diff --git a/mod9-py3dstasks/main.py b/mod9-py3fstasks/main.py similarity index 100% rename from mod9-py3dstasks/main.py rename to mod9-py3fstasks/main.py diff --git a/mod9-py3dstasks/requirements.txt b/mod9-py3fstasks/requirements.txt similarity index 100% rename from mod9-py3dstasks/requirements.txt rename to mod9-py3fstasks/requirements.txt diff --git a/mod9-py3dstasks/templates/index.html b/mod9-py3fstasks/templates/index.html similarity index 100% rename from mod9-py3dstasks/templates/index.html rename to mod9-py3fstasks/templates/index.html From fad00bb6af59cbe2f79f411dd7b787a7c111c609 Mon Sep 17 00:00:00 2001 From: wesley chun Date: Sat, 3 Jul 2021 00:35:04 -0700 Subject: [PATCH 04/56] README & gcloudignore tweaks, minor memory usage fix in Mod9 --- README.md | 11 +++++------ mod0-baseline/.gcloudignore | 5 ++--- mod9-py3fstasks/main.py | 15 +++++++-------- 3 files changed, 14 insertions(+), 17 deletions(-) diff --git a/README.md b/README.md index ca8b522..5fc7ac1 100644 --- a/README.md +++ b/README.md @@ -246,8 +246,8 @@ If your original app users does *not* have a user interface, i.e., mobile backen - This repo, along with corresponding codelabs & videos are complementary to the official docs & code samples. - The [official Python 2 to 3 migration documentation](https://cloud.google.com/appengine/docs/standard/python/migrate-to-python3) - [Canonical migration code samples repo](https://github.com/GoogleCloudPlatform/python-docs-samples/tree/master/appengine/standard/migration) - - Example: [GAE `ndb` to Cloud NDB](https://github.com/GoogleCloudPlatform/python-docs-samples/tree/master/appengine/standard/migration/ndb/overview) - - Example: [GAE `taskqueue` to Cloud Tasks](https://github.com/GoogleCloudPlatform/python-docs-samples/tree/master/appengine/standard/migration/taskqueue) + - *Example:* [GAE `ndb` to Cloud NDB](https://github.com/GoogleCloudPlatform/python-docs-samples/tree/master/appengine/standard/migration/ndb/overview) (similar to Module 2) + - *Example:* [GAE `taskqueue` to Cloud Tasks](https://github.com/GoogleCloudPlatform/python-docs-samples/tree/master/appengine/standard/migration/taskqueue) (similar to Module 8) ## References @@ -255,15 +255,14 @@ If your original app users does *not* have a user interface, i.e., mobile backen - App Engine Migration - [Migrate from Python 2 to 3](http://cloud.google.com/appengine/docs/standard/python/migrate-to-python3) - [Migrate from App Engine `ndb` to Cloud NDB](http://cloud.google.com/appengine/docs/standard/python/migrate-to-python3/migrate-to-cloud-ndb) (Module 2) - - [App Engine `ndb` to Cloud NDB official sample app](https://github.com/GoogleCloudPlatform/python-docs-samples/tree/master/appengine/standard/migration/ndb/overview) (Module 2) - [Migrate from App Engine `taskqueue` to Cloud Tasks](http://cloud.google.com/appengine/docs/standard/python/migrate-to-python3/migrate-to-cloud-ndb) (Modules 7-9) - - [App Engine `app.yaml` to Cloud Run `service.yaml` tool](http://googlecloudplatform.github.io/app-engine-cloud-run-converter) (Modules 4 and 5) - [Migrate from App Engine `db` to `ndb`](http://cloud.google.com/appengine/docs/standard/python/ndb/db_to_ndb) ("Module -1"; only for reviving "dead" Python 2.5 apps for 2.7) - [Community contributed migration samples](https://github.com/GoogleCloudPlatform/appengine-python2-3-migration) - Python App Engine - - [Python 2 App Engine (Standard)](https://cloud.google.com/appengine/docs/standard/python/runtime) - - [Python 3 App Engine (Standard)](https://cloud.google.com/appengine/docs/standard/python3/runtime) + - [App Engine 1st vs. 2nd generation runtimes](https://cloud.google.com/appengine/docs/standard/runtimes) + - [Python 2 App Engine (Standard) runtime](https://cloud.google.com/appengine/docs/standard/python/runtime) + - [Python 3 App Engine (Standard) runtime](https://cloud.google.com/appengine/docs/standard/python3/runtime) - [Python App Engine (Flexible)](https://cloud.google.com/appengine/docs/flexible/python) - Google Cloud Platform (GCP) diff --git a/mod0-baseline/.gcloudignore b/mod0-baseline/.gcloudignore index bcf97a8..ee1a5ea 100644 --- a/mod0-baseline/.gcloudignore +++ b/mod0-baseline/.gcloudignore @@ -9,14 +9,13 @@ .gcloudignore # Ignore source code control maintenance files -.git +.git/ .gitignore .hgignore .hg/ # Python files -*.pyc -*.pyo +*.py[cod] __pycache__/ /setup.cfg diff --git a/mod9-py3fstasks/main.py b/mod9-py3fstasks/main.py index 9720083..3727ca5 100644 --- a/mod9-py3fstasks/main.py +++ b/mod9-py3fstasks/main.py @@ -37,7 +37,7 @@ def store_visit(remote_addr, user_agent): 'visitor': '{}: {}'.format(remote_addr, user_agent), }) -def create_queue_if(): +def _create_queue_if(): 'app-internal function creating default queue if it does not exist' try: ts_client.get_queue(name=QUEUE_PATH) @@ -64,7 +64,7 @@ def fetch_visits(limit): }, } } - if create_queue_if(): + if _create_queue_if(): ts_client.create_task(parent=QUEUE_PATH, task=task) return visits, oldest_str @@ -79,12 +79,11 @@ def trim(): '(push) task queue handler to delete oldest visits' oldest = float(request.get_json().get('oldest')) query = fs_client.collection('Visit') - visits = list(query.where('timestamp', '<', - datetime.fromtimestamp(oldest)).stream()) - nvisits = len(visits) - if nvisits: - print('Deleting %d entities: ' % nvisits, end='') - print(', '.join(str(v_id) for v_id in _delete_docs(visits))) + visits = query.where('timestamp', '<', + datetime.fromtimestamp(oldest)).stream() + dlist = ', '.join(str(v_id) for v_id in _delete_docs(visits)) + if dlist: + print('Deleting %d entities: %s' % (dlist.count(',')+1, dlist)) else: print('No entities older than: %s' % time.ctime(oldest)) return '' # need to return SOME string w/200 From 51818ac4b985bd27f0481379f85e7aea6bc15710 Mon Sep 17 00:00:00 2001 From: wesley chun Date: Sat, 17 Jul 2021 01:53:21 -0700 Subject: [PATCH 05/56] update copyright year & pkg version#s --- mod0-baseline/app.yaml | 2 +- mod0-baseline/main.py | 2 +- mod1-flask/app.yaml | 2 +- mod1-flask/appengine_config.py | 2 +- mod11-functions/main.py | 2 +- mod2a-cloudndb/app.yaml | 2 +- mod2a-cloudndb/appengine_config.py | 2 +- mod2a-cloudndb/main.py | 2 +- mod2b-cloudndb/app.yaml | 2 +- mod2b-cloudndb/main.py | 2 +- mod3a-datastore/app.yaml | 2 +- mod3a-datastore/appengine_config.py | 2 +- mod3a-datastore/main.py | 2 +- mod3b-datastore/app.yaml | 2 +- mod3b-datastore/main.py | 2 +- mod4a-rundocker/main.py | 2 +- mod4b-rundocker/main.py | 2 +- mod5-runbldpks/main.py | 2 +- mod6-firestore/app.yaml | 2 +- mod6-firestore/main.py | 2 +- mod7-gaetasks/app.yaml | 2 +- mod7-gaetasks/appengine_config.py | 2 +- mod7-gaetasks/main.py | 2 +- mod8-cloudtasks/app.yaml | 2 +- mod8-cloudtasks/appengine_config.py | 2 +- mod8-cloudtasks/main.py | 2 +- mod9-py3fstasks/app.yaml | 2 +- mod9-py3fstasks/main.py | 2 +- 28 files changed, 28 insertions(+), 28 deletions(-) diff --git a/mod0-baseline/app.yaml b/mod0-baseline/app.yaml index 80fb603..0d4d8d6 100644 --- a/mod0-baseline/app.yaml +++ b/mod0-baseline/app.yaml @@ -1,4 +1,4 @@ -# Copyright 2020 Google LLC +# Copyright 2021 Google LLC # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/mod0-baseline/main.py b/mod0-baseline/main.py index f21d8b0..29ef681 100644 --- a/mod0-baseline/main.py +++ b/mod0-baseline/main.py @@ -1,4 +1,4 @@ -# Copyright 2020 Google LLC +# Copyright 2021 Google LLC # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/mod1-flask/app.yaml b/mod1-flask/app.yaml index 80fb603..0d4d8d6 100644 --- a/mod1-flask/app.yaml +++ b/mod1-flask/app.yaml @@ -1,4 +1,4 @@ -# Copyright 2020 Google LLC +# Copyright 2021 Google LLC # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/mod1-flask/appengine_config.py b/mod1-flask/appengine_config.py index 0ca8634..9760ebb 100644 --- a/mod1-flask/appengine_config.py +++ b/mod1-flask/appengine_config.py @@ -1,4 +1,4 @@ -# Copyright 2020 Google LLC +# Copyright 2021 Google LLC # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/mod11-functions/main.py b/mod11-functions/main.py index 30e2f02..6c0acc6 100644 --- a/mod11-functions/main.py +++ b/mod11-functions/main.py @@ -1,4 +1,4 @@ -# Copyright 2020 Google LLC +# Copyright 2021 Google LLC # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/mod2a-cloudndb/app.yaml b/mod2a-cloudndb/app.yaml index 03e7e70..0877f79 100644 --- a/mod2a-cloudndb/app.yaml +++ b/mod2a-cloudndb/app.yaml @@ -1,4 +1,4 @@ -# Copyright 2020 Google LLC +# Copyright 2021 Google LLC # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/mod2a-cloudndb/appengine_config.py b/mod2a-cloudndb/appengine_config.py index 4773cd7..2a41fb4 100644 --- a/mod2a-cloudndb/appengine_config.py +++ b/mod2a-cloudndb/appengine_config.py @@ -1,4 +1,4 @@ -# Copyright 2020 Google LLC +# Copyright 2021 Google LLC # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/mod2a-cloudndb/main.py b/mod2a-cloudndb/main.py index 1d95ed7..4c3015a 100644 --- a/mod2a-cloudndb/main.py +++ b/mod2a-cloudndb/main.py @@ -1,4 +1,4 @@ -# Copyright 2020 Google LLC +# Copyright 2021 Google LLC # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/mod2b-cloudndb/app.yaml b/mod2b-cloudndb/app.yaml index a74df21..a4ce819 100644 --- a/mod2b-cloudndb/app.yaml +++ b/mod2b-cloudndb/app.yaml @@ -1,4 +1,4 @@ -# Copyright 2020 Google LLC +# Copyright 2021 Google LLC # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/mod2b-cloudndb/main.py b/mod2b-cloudndb/main.py index 1d95ed7..4c3015a 100644 --- a/mod2b-cloudndb/main.py +++ b/mod2b-cloudndb/main.py @@ -1,4 +1,4 @@ -# Copyright 2020 Google LLC +# Copyright 2021 Google LLC # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/mod3a-datastore/app.yaml b/mod3a-datastore/app.yaml index 03e7e70..0877f79 100644 --- a/mod3a-datastore/app.yaml +++ b/mod3a-datastore/app.yaml @@ -1,4 +1,4 @@ -# Copyright 2020 Google LLC +# Copyright 2021 Google LLC # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/mod3a-datastore/appengine_config.py b/mod3a-datastore/appengine_config.py index 4773cd7..2a41fb4 100644 --- a/mod3a-datastore/appengine_config.py +++ b/mod3a-datastore/appengine_config.py @@ -1,4 +1,4 @@ -# Copyright 2020 Google LLC +# Copyright 2021 Google LLC # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/mod3a-datastore/main.py b/mod3a-datastore/main.py index 4eaa4dd..73a8eeb 100644 --- a/mod3a-datastore/main.py +++ b/mod3a-datastore/main.py @@ -1,4 +1,4 @@ -# Copyright 2020 Google LLC +# Copyright 2021 Google LLC # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/mod3b-datastore/app.yaml b/mod3b-datastore/app.yaml index a74df21..a4ce819 100644 --- a/mod3b-datastore/app.yaml +++ b/mod3b-datastore/app.yaml @@ -1,4 +1,4 @@ -# Copyright 2020 Google LLC +# Copyright 2021 Google LLC # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/mod3b-datastore/main.py b/mod3b-datastore/main.py index 4eaa4dd..73a8eeb 100644 --- a/mod3b-datastore/main.py +++ b/mod3b-datastore/main.py @@ -1,4 +1,4 @@ -# Copyright 2020 Google LLC +# Copyright 2021 Google LLC # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/mod4a-rundocker/main.py b/mod4a-rundocker/main.py index 1d95ed7..4c3015a 100644 --- a/mod4a-rundocker/main.py +++ b/mod4a-rundocker/main.py @@ -1,4 +1,4 @@ -# Copyright 2020 Google LLC +# Copyright 2021 Google LLC # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/mod4b-rundocker/main.py b/mod4b-rundocker/main.py index 4eaa4dd..73a8eeb 100644 --- a/mod4b-rundocker/main.py +++ b/mod4b-rundocker/main.py @@ -1,4 +1,4 @@ -# Copyright 2020 Google LLC +# Copyright 2021 Google LLC # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/mod5-runbldpks/main.py b/mod5-runbldpks/main.py index 1d95ed7..4c3015a 100644 --- a/mod5-runbldpks/main.py +++ b/mod5-runbldpks/main.py @@ -1,4 +1,4 @@ -# Copyright 2020 Google LLC +# Copyright 2021 Google LLC # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/mod6-firestore/app.yaml b/mod6-firestore/app.yaml index a173c6e..48dfc91 100644 --- a/mod6-firestore/app.yaml +++ b/mod6-firestore/app.yaml @@ -1,4 +1,4 @@ -# Copyright 2020 Google LLC +# Copyright 2021 Google LLC # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/mod6-firestore/main.py b/mod6-firestore/main.py index 5db626b..b9a83a7 100644 --- a/mod6-firestore/main.py +++ b/mod6-firestore/main.py @@ -1,4 +1,4 @@ -# Copyright 2020 Google LLC +# Copyright 2021 Google LLC # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/mod7-gaetasks/app.yaml b/mod7-gaetasks/app.yaml index 80fb603..0d4d8d6 100644 --- a/mod7-gaetasks/app.yaml +++ b/mod7-gaetasks/app.yaml @@ -1,4 +1,4 @@ -# Copyright 2020 Google LLC +# Copyright 2021 Google LLC # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/mod7-gaetasks/appengine_config.py b/mod7-gaetasks/appengine_config.py index 0ca8634..9760ebb 100644 --- a/mod7-gaetasks/appengine_config.py +++ b/mod7-gaetasks/appengine_config.py @@ -1,4 +1,4 @@ -# Copyright 2020 Google LLC +# Copyright 2021 Google LLC # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/mod7-gaetasks/main.py b/mod7-gaetasks/main.py index 289bd28..49e2b08 100644 --- a/mod7-gaetasks/main.py +++ b/mod7-gaetasks/main.py @@ -1,4 +1,4 @@ -# Copyright 2020 Google LLC +# Copyright 2021 Google LLC # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/mod8-cloudtasks/app.yaml b/mod8-cloudtasks/app.yaml index 03e7e70..0877f79 100644 --- a/mod8-cloudtasks/app.yaml +++ b/mod8-cloudtasks/app.yaml @@ -1,4 +1,4 @@ -# Copyright 2020 Google LLC +# Copyright 2021 Google LLC # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/mod8-cloudtasks/appengine_config.py b/mod8-cloudtasks/appengine_config.py index 4773cd7..2a41fb4 100644 --- a/mod8-cloudtasks/appengine_config.py +++ b/mod8-cloudtasks/appengine_config.py @@ -1,4 +1,4 @@ -# Copyright 2020 Google LLC +# Copyright 2021 Google LLC # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/mod8-cloudtasks/main.py b/mod8-cloudtasks/main.py index abd5876..10a1085 100644 --- a/mod8-cloudtasks/main.py +++ b/mod8-cloudtasks/main.py @@ -1,4 +1,4 @@ -# Copyright 2020 Google LLC +# Copyright 2021 Google LLC # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/mod9-py3fstasks/app.yaml b/mod9-py3fstasks/app.yaml index a74df21..a4ce819 100644 --- a/mod9-py3fstasks/app.yaml +++ b/mod9-py3fstasks/app.yaml @@ -1,4 +1,4 @@ -# Copyright 2020 Google LLC +# Copyright 2021 Google LLC # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/mod9-py3fstasks/main.py b/mod9-py3fstasks/main.py index 3727ca5..d6c524a 100644 --- a/mod9-py3fstasks/main.py +++ b/mod9-py3fstasks/main.py @@ -1,4 +1,4 @@ -# Copyright 2020 Google LLC +# Copyright 2021 Google LLC # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. From 7550d7f1f76954506c90b2b8e33f7d9958eaa509 Mon Sep 17 00:00:00 2001 From: wesley chun Date: Sat, 17 Jul 2021 02:08:51 -0700 Subject: [PATCH 06/56] add Py3 legacy services app; update README --- README.md | 9 +++++-- mod1b-flask/.gcloudignore | 24 +++++++++++++++++++ mod1b-flask/README.md | 5 ++++ mod1b-flask/app.yaml | 16 +++++++++++++ mod1b-flask/main.py | 41 ++++++++++++++++++++++++++++++++ mod1b-flask/requirements.txt | 2 ++ mod1b-flask/templates/index.html | 16 +++++++++++++ 7 files changed, 111 insertions(+), 2 deletions(-) create mode 100644 mod1b-flask/.gcloudignore create mode 100644 mod1b-flask/README.md create mode 100644 mod1b-flask/app.yaml create mode 100644 mod1b-flask/main.py create mode 100644 mod1b-flask/requirements.txt create mode 100644 mod1b-flask/templates/index.html diff --git a/README.md b/README.md index 5fc7ac1..cccdc1c 100644 --- a/README.md +++ b/README.md @@ -70,7 +70,7 @@ The table below summarizes migration module resources currently available along Module | Topic | Video | Codelab | START here | FINISH here --- | --- | --- | --- | --- | --- 0|Baseline app| [link](http://twitter.com/googledevs/status/1407755281786867714?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_smsintro_201023&utm_content=-) | _N/A_ (no tutorial; just review the code) | _N/A_ | Module 0 [code](/mod0-baseline) (2.x) -1|Migrate to Flask| _TBD_ | [link](http://g.co/codelabs/pae-migrate-flask) | Module 0 [code](/mod0-baseline) (2.x) | Module 1 [code](/mod1-flask) (2.x) +1|Migrate to Flask| _TBD_ | [link](http://g.co/codelabs/pae-migrate-flask) | Module 0 [code](/mod0-baseline) (2.x) | Module 1 [code](/mod1-flask) (2.x) & [code](/mod1b-flask) (3.x) 2|Migrate to Cloud NDB| _TBD_ | [link](http://g.co/codelabs/pae-migrate-cloudndb) | Module 1 [code](/mod1-flask) (2.x) | Module 2 [code](/mod2a-cloudndb) (2.x) & [code](/mod2b-cloudndb) (3.x) 3|Migrate to Cloud Datastore| _TBD_ | [link](http://g.co/codelabs/pae-migrate-datastore) | Module 2 [code](/mod2a-cloudndb) (2.x) & [code](/mod2b-cloudndb) (3.x) | Module 3 [code](/mod3a-datastore) (2.x) & [code](/mod3b-datastore) (3.x) 4|Migrate to Cloud Run with Docker| _TBD_ | [link](http://g.co/codelabs/pae-migrate-rundocker) | Module 2 [code](/mod2a-cloudndb) (2.x) & Module 3 [code](/mod3b-datastore) (3.x) | Module 4 [code](/mod4a-rundocker) (2.x) & [code](/mod4b-rundocker) (3.x) @@ -241,7 +241,7 @@ If your original app users does *not* have a user interface, i.e., mobile backen - [Firebase mobile & web app platform](https://firebase.google.com) (and [Cloud Functions for Firebase](https://firebase.google.com/products/functions) [customized for Firebase]) -### Canonical code samples +## Canonical code samples - This repo, along with corresponding codelabs & videos are complementary to the official docs & code samples. - The [official Python 2 to 3 migration documentation](https://cloud.google.com/appengine/docs/standard/python/migrate-to-python3) @@ -250,6 +250,11 @@ If your original app users does *not* have a user interface, i.e., mobile backen - *Example:* [GAE `taskqueue` to Cloud Tasks](https://github.com/GoogleCloudPlatform/python-docs-samples/tree/master/appengine/standard/migration/taskqueue) (similar to Module 8) +## Accessing legacy service in second generation + +Legacy App Engine services created for the first generation platform (Python 2, Java 8, PHP 5, and Go 1.11 & older) are available on a limited basis for second generation runtimes (Python 3, Java 11, PHP 7, and Go 1.12 & newer) as part of an early-access program. There is no video or codelab available yet, however, the Module 1 Flask migration [Python 2 sample](/mod1-flask) is available in [Python 3](/mod1b-flask) if you have access. (See its README to sign-up if interested.) + + ## References - App Engine Migration diff --git a/mod1b-flask/.gcloudignore b/mod1b-flask/.gcloudignore new file mode 100644 index 0000000..bcf97a8 --- /dev/null +++ b/mod1b-flask/.gcloudignore @@ -0,0 +1,24 @@ +# This file specifies files that are *not* uploaded to Google Cloud Platform +# using gcloud. It follows the same syntax as .gitignore, with the addition of +# "#!include" directives (which insert the entries of the given .gitignore-style +# file at that point). +# +# For more information, run: +# $ gcloud topic gcloudignore +# +.gcloudignore + +# Ignore source code control maintenance files +.git +.gitignore +.hgignore +.hg/ + +# Python files +*.pyc +*.pyo +__pycache__/ +/setup.cfg + +# no need to upload README +README.md diff --git a/mod1b-flask/README.md b/mod1b-flask/README.md new file mode 100644 index 0000000..70fe0aa --- /dev/null +++ b/mod1b-flask/README.md @@ -0,0 +1,5 @@ +# Module 1 - Migrate from `webapp2` to Flask + +This repo folder is the corresponding Python 3 code to the [Module 1 codelab](http://g.co/codelabs/pae-migrate-flask). The tutorial STARTs with the Python 2 code in the [Module 0 repo folder](/mod0-baseline) and leads developers through migrating away from App Engine's `webapp2` web framework to Flask, culminating in the code in the [mod1-flask](/mod1-flask) folder. That is followed by a BONUS migration to Python 3, culminating in the code in *this* (`mod1b-flask`) folder. In the [next (Module 2) codelab](http://g.co/codelabs/pae-migrate-cloudndb), you will convert this app from the bundled `ndb` library to the Cloud NDB library for Datastore access. + +**NOTE**: Accessing a legacy service such as App Engine `ndb` from Python 3 (and next generation App Engine in general) is currently in private preview and requires access. Please complete [this form](https://docs.google.com/forms/d/e/1FAIpQLSd1hFLA2UFSYwIMxm9ZI3pwigORZBgjJRH0qrnhtE7nvhhRCQ/viewform) to register for this early-access program. Doing so opts you into the preview and give you access to the documentation. diff --git a/mod1b-flask/app.yaml b/mod1b-flask/app.yaml new file mode 100644 index 0000000..288ce87 --- /dev/null +++ b/mod1b-flask/app.yaml @@ -0,0 +1,16 @@ +# Copyright 2021 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +runtime: python38 +app_engine_apis: true diff --git a/mod1b-flask/main.py b/mod1b-flask/main.py new file mode 100644 index 0000000..bfc4c80 --- /dev/null +++ b/mod1b-flask/main.py @@ -0,0 +1,41 @@ +# Copyright 2021 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from flask import Flask, render_template, request +from google.appengine.api import wrap_wsgi_app +from google.appengine.ext import ndb + +app = Flask(__name__) +app.wsgi_app = wrap_wsgi_app(app.wsgi_app) + +class Visit(ndb.Model): + 'Visit entity registers visitor IP address & timestamp' + visitor = ndb.StringProperty() + timestamp = ndb.DateTimeProperty(auto_now_add=True) + +def store_visit(remote_addr, user_agent): + 'create new Visit entity in Datastore' + Visit(visitor='{}: {}'.format(remote_addr, user_agent)).put() + +def fetch_visits(limit): + 'get most recent visits' + return (v.to_dict() for v in Visit.query().order( + -Visit.timestamp).fetch(limit)) + +@app.route('/') +def root(): + 'main application (GET) handler' + store_visit(request.remote_addr, request.user_agent) + visits = fetch_visits(10) + return render_template('index.html', visits=visits) diff --git a/mod1b-flask/requirements.txt b/mod1b-flask/requirements.txt new file mode 100644 index 0000000..c661cf7 --- /dev/null +++ b/mod1b-flask/requirements.txt @@ -0,0 +1,2 @@ +flask==1.1.2 +appengine-python-standard diff --git a/mod1b-flask/templates/index.html b/mod1b-flask/templates/index.html new file mode 100644 index 0000000..e140206 --- /dev/null +++ b/mod1b-flask/templates/index.html @@ -0,0 +1,16 @@ + + + +VisitMe Example + + +

VisitMe example

+

Last 10 visits

+
    +{% for visit in visits %} +
  • {{ visit.timestamp.ctime() }} from {{ visit.visitor }}
  • +{% endfor %} +
+ + + From b5c89997adb69e4d5a5389b93a3d2c69a0630cc4 Mon Sep 17 00:00:00 2001 From: wesley chun Date: Sat, 17 Jul 2021 17:07:44 -0700 Subject: [PATCH 07/56] README cleanup --- README.md | 8 +++++--- mod1b-flask/README.md | 7 +++++-- 2 files changed, 10 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index cccdc1c..ebc796c 100644 --- a/README.md +++ b/README.md @@ -31,7 +31,9 @@ Furthermore, deploying to GCP serverless platforms incur [minor build and storag In App Engine's early days, users wanted Google to make the platform more flexible for developers and make their apps more portable. As a result, the team made significant changes to its 2nd-generation service which [launched in 2018](https://cloud.google.com/blog/products/gcp/introducing-app-engine-second-generation-runtimes-and-python-3-7). As a result, there are no longer any built-in services, allowing users to select from standalone GCP products or best-of-breed 3rd-party tools used by the broader community. Summary: - **Legacy platform**: *Python 2* only, legacy built-in services -- **Next generation**: *Python 3* only, external services, flexible platform +- **Next generation**: *Python 3* only, use standalone services, flexible platform, some* built-in services available + +* see [Legacy services](#accessing-legacy-services-in-second-generation) section below While the 2nd-gen platform is more flexible, users of the legacy platform have two challenges: @@ -250,9 +252,9 @@ If your original app users does *not* have a user interface, i.e., mobile backen - *Example:* [GAE `taskqueue` to Cloud Tasks](https://github.com/GoogleCloudPlatform/python-docs-samples/tree/master/appengine/standard/migration/taskqueue) (similar to Module 8) -## Accessing legacy service in second generation +## Accessing legacy services in second generation -Legacy App Engine services created for the first generation platform (Python 2, Java 8, PHP 5, and Go 1.11 & older) are available on a limited basis for second generation runtimes (Python 3, Java 11, PHP 7, and Go 1.12 & newer) as part of an early-access program. There is no video or codelab available yet, however, the Module 1 Flask migration [Python 2 sample](/mod1-flask) is available in [Python 3](/mod1b-flask) if you have access. (See its README to sign-up if interested.) +Some legacy App Engine first generation platform (Python 2, Java 8, PHP 5, and Go 1.11 & older) services are available on a limited basis to second generation runtimes (Python 3, Java 11, PHP 7, and Go 1.12 & newer) as part of an early-access program. There is no video or codelab available yet, however the Module 1 Flask migration [Python 2 sample](/mod1-flask) is available in [Python 3](/mod1b-flask) if you have access. (See its README to sign-up if interested.) ## References diff --git a/mod1b-flask/README.md b/mod1b-flask/README.md index 70fe0aa..b264059 100644 --- a/mod1b-flask/README.md +++ b/mod1b-flask/README.md @@ -1,5 +1,8 @@ # Module 1 - Migrate from `webapp2` to Flask -This repo folder is the corresponding Python 3 code to the [Module 1 codelab](http://g.co/codelabs/pae-migrate-flask). The tutorial STARTs with the Python 2 code in the [Module 0 repo folder](/mod0-baseline) and leads developers through migrating away from App Engine's `webapp2` web framework to Flask, culminating in the code in the [mod1-flask](/mod1-flask) folder. That is followed by a BONUS migration to Python 3, culminating in the code in *this* (`mod1b-flask`) folder. In the [next (Module 2) codelab](http://g.co/codelabs/pae-migrate-cloudndb), you will convert this app from the bundled `ndb` library to the Cloud NDB library for Datastore access. +This repo folder is the corresponding Python 3 code to the [Module 1 codelab](http://g.co/codelabs/pae-migrate-flask). The tutorial STARTs with the Python 2 code in the [Module 0 repo folder](/mod0-baseline) and leads developers through migrating away from App Engine's `webapp2` web framework to Flask, culminating in the code in the [mod1-flask](/mod1-flask) folder. The codelab does **not** currently feature any bonus migration to Python 3, however, if you do so, and add in components from the legacy services private preview program (see sidebar below), it will culminate in the code in *this* (`mod1b-flask`) folder. In the [next (Module 2) codelab](http://g.co/codelabs/pae-migrate-cloudndb), users will migrate (the original Python 2 version of) this app from App Engine `ndb` to Cloud NDB for Datastore access. + +> **LEGACY SERVICES PRIVATE PREVIEW**: Accessing a legacy service such as App Engine `ndb` from Python 3 (and next generation App Engine in general) is currently in private preview and requires access. Please complete [this form](https://docs.google.com/forms/d/e/1FAIpQLSd1hFLA2UFSYwIMxm9ZI3pwigORZBgjJRH0qrnhtE7nvhhRCQ/viewform) to sign-up for this early-access program. Registering will get you in the queue to access the private preview, its documentation, and the announcements mailing list. For more info on what services are (or are not) available at this time, see this Subreddit](https://reddit.com/r/AppEngine/comments/o9wr72). + + -**NOTE**: Accessing a legacy service such as App Engine `ndb` from Python 3 (and next generation App Engine in general) is currently in private preview and requires access. Please complete [this form](https://docs.google.com/forms/d/e/1FAIpQLSd1hFLA2UFSYwIMxm9ZI3pwigORZBgjJRH0qrnhtE7nvhhRCQ/viewform) to register for this early-access program. Doing so opts you into the preview and give you access to the documentation. From c4e27b3aeff701b36a1a2bd0c331ed66ef12b26e Mon Sep 17 00:00:00 2001 From: wesley chun Date: Fri, 23 Jul 2021 18:37:38 -0700 Subject: [PATCH 08/56] add video links & README updates --- README.md | 4 ++-- mod1b-flask/README.md | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index ebc796c..90a6682 100644 --- a/README.md +++ b/README.md @@ -72,8 +72,8 @@ The table below summarizes migration module resources currently available along Module | Topic | Video | Codelab | START here | FINISH here --- | --- | --- | --- | --- | --- 0|Baseline app| [link](http://twitter.com/googledevs/status/1407755281786867714?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_smsintro_201023&utm_content=-) | _N/A_ (no tutorial; just review the code) | _N/A_ | Module 0 [code](/mod0-baseline) (2.x) -1|Migrate to Flask| _TBD_ | [link](http://g.co/codelabs/pae-migrate-flask) | Module 0 [code](/mod0-baseline) (2.x) | Module 1 [code](/mod1-flask) (2.x) & [code](/mod1b-flask) (3.x) -2|Migrate to Cloud NDB| _TBD_ | [link](http://g.co/codelabs/pae-migrate-cloudndb) | Module 1 [code](/mod1-flask) (2.x) | Module 2 [code](/mod2a-cloudndb) (2.x) & [code](/mod2b-cloudndb) (3.x) +1|Migrate to Flask| [link](https://twitter.com/googledevs/status/1413273119071031299?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrwa2flsk_201008&u%20tm_content=-) | [link](http://g.co/codelabs/pae-migrate-flask) | Module 0 [code](/mod0-baseline) (2.x) | Module 1 [code](/mod1-flask) (2.x) & [code](/mod1b-flask) (3.x) +2|Migrate to Cloud NDB| [link](http://twitter.com/googledevs/status/1413273119071031299?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrwa2flsk_201008&utm_content=-) | [link](http://g.co/codelabs/pae-migrate-cloudndb) | Module 1 [code](/mod1-flask) (2.x) | Module 2 [code](/mod2a-cloudndb) (2.x) & [code](/mod2b-cloudndb) (3.x) 3|Migrate to Cloud Datastore| _TBD_ | [link](http://g.co/codelabs/pae-migrate-datastore) | Module 2 [code](/mod2a-cloudndb) (2.x) & [code](/mod2b-cloudndb) (3.x) | Module 3 [code](/mod3a-datastore) (2.x) & [code](/mod3b-datastore) (3.x) 4|Migrate to Cloud Run with Docker| _TBD_ | [link](http://g.co/codelabs/pae-migrate-rundocker) | Module 2 [code](/mod2a-cloudndb) (2.x) & Module 3 [code](/mod3b-datastore) (3.x) | Module 4 [code](/mod4a-rundocker) (2.x) & [code](/mod4b-rundocker) (3.x) 5|Migrate to Cloud Run with Buildpacks| _TBD_ | [link](http://g.co/codelabs/pae-migrate-runbldpks) | Module 2 [code](/mod2b-cloudndb) (3.x) | Module 5 [code](/mod5-runbldpks) (3.x) diff --git a/mod1b-flask/README.md b/mod1b-flask/README.md index b264059..f90ca49 100644 --- a/mod1b-flask/README.md +++ b/mod1b-flask/README.md @@ -2,7 +2,7 @@ This repo folder is the corresponding Python 3 code to the [Module 1 codelab](http://g.co/codelabs/pae-migrate-flask). The tutorial STARTs with the Python 2 code in the [Module 0 repo folder](/mod0-baseline) and leads developers through migrating away from App Engine's `webapp2` web framework to Flask, culminating in the code in the [mod1-flask](/mod1-flask) folder. The codelab does **not** currently feature any bonus migration to Python 3, however, if you do so, and add in components from the legacy services private preview program (see sidebar below), it will culminate in the code in *this* (`mod1b-flask`) folder. In the [next (Module 2) codelab](http://g.co/codelabs/pae-migrate-cloudndb), users will migrate (the original Python 2 version of) this app from App Engine `ndb` to Cloud NDB for Datastore access. -> **LEGACY SERVICES PRIVATE PREVIEW**: Accessing a legacy service such as App Engine `ndb` from Python 3 (and next generation App Engine in general) is currently in private preview and requires access. Please complete [this form](https://docs.google.com/forms/d/e/1FAIpQLSd1hFLA2UFSYwIMxm9ZI3pwigORZBgjJRH0qrnhtE7nvhhRCQ/viewform) to sign-up for this early-access program. Registering will get you in the queue to access the private preview, its documentation, and the announcements mailing list. For more info on what services are (or are not) available at this time, see this Subreddit](https://reddit.com/r/AppEngine/comments/o9wr72). +> **LEGACY SERVICES PRIVATE PREVIEW**: Accessing a legacy service such as App Engine `ndb` from Python 3 (and next generation App Engine in general) is currently in private preview and requires access. Please complete [this form](https://docs.google.com/forms/d/e/1FAIpQLSd1hFLA2UFSYwIMxm9ZI3pwigORZBgjJRH0qrnhtE7nvhhRCQ/viewform) to sign-up for the program. Registering will get you in the queue to access the private preview, its documentation, and the announcements mailing list. For more info on what services are (or are not) available (at this time), see [this thread in the r/AppEngine Subreddit](https://reddit.com/r/AppEngine/comments/o9wr72). From c62451d306d7f8aef77cd61650ee8e47c491d9eb Mon Sep 17 00:00:00 2001 From: wesley chun Date: Fri, 30 Jul 2021 23:47:56 -0700 Subject: [PATCH 09/56] add memcache sample; update TQ templates --- README.md | 3 +- mod0-baseline/.gcloudignore | 5 +-- mod12a-memcache/.gcloudignore | 24 +++++++++++++ mod12a-memcache/README.md | 3 ++ mod12a-memcache/app.yaml | 21 ++++++++++++ mod12a-memcache/appengine_config.py | 20 +++++++++++ mod12a-memcache/main.py | 51 ++++++++++++++++++++++++++++ mod12a-memcache/requirements.txt | 1 + mod12a-memcache/templates/index.html | 16 +++++++++ mod7-gaetasks/templates/index.html | 2 +- mod8-cloudtasks/templates/index.html | 2 +- mod9-py3fstasks/templates/index.html | 2 +- 12 files changed, 144 insertions(+), 6 deletions(-) create mode 100644 mod12a-memcache/.gcloudignore create mode 100644 mod12a-memcache/README.md create mode 100644 mod12a-memcache/app.yaml create mode 100644 mod12a-memcache/appengine_config.py create mode 100644 mod12a-memcache/main.py create mode 100644 mod12a-memcache/requirements.txt create mode 100644 mod12a-memcache/templates/index.html diff --git a/README.md b/README.md index 90a6682..233540f 100644 --- a/README.md +++ b/README.md @@ -78,11 +78,12 @@ Module | Topic | Video | Codelab | START here | FINISH here 4|Migrate to Cloud Run with Docker| _TBD_ | [link](http://g.co/codelabs/pae-migrate-rundocker) | Module 2 [code](/mod2a-cloudndb) (2.x) & Module 3 [code](/mod3b-datastore) (3.x) | Module 4 [code](/mod4a-rundocker) (2.x) & [code](/mod4b-rundocker) (3.x) 5|Migrate to Cloud Run with Buildpacks| _TBD_ | [link](http://g.co/codelabs/pae-migrate-runbldpks) | Module 2 [code](/mod2b-cloudndb) (3.x) | Module 5 [code](/mod5-runbldpks) (3.x) 6|Migrate to Cloud Firestore (app)| _TBD_ | [link](http://g.co/codelabs/pae-migrate-firestore) | Module 3 [code](/mod3b-datastore) (3.x) | Module 6 [code](/mod6-firestore) (3.x) -7|Add App Engine push tasks| _TBD_ | [link](http://g.co/codelabs/pae-migrate-gaetasks) | Module 1 [code]() (2.x) | Module 7 [code](/mod7-gaetasks) (2.x) +7|Add App Engine push tasks| _TBD_ | [link](http://g.co/codelabs/pae-migrate-gaetasks) | Module 1 [code](/mod1-flask) (2.x) | Module 7 [code](/mod7-gaetasks) (2.x) 8|Migrate to Cloud Tasks| _TBD_ | [link](http://g.co/codelabs/pae-migrate-cloudtasks) | Module 7 [code](/mod7-gaetasks) (2.x) | Module 8 [code](/mod8-cloudtasks) (2.x) 9|Migrate to Python 3, Cloud Firestore & Cloud Tasks v2| _TBD_ | _TBD_ | Module 8 [code](/mod8-cloudtasks) (2.x) | Module 9 [code](/mod9-py3fstasks) (3.x) 10|Migrate to Cloud Firestore (data)| _TBD_ | _N/A_ | _N/A_ | _TBD_ 11|Migrate to Cloud Functions| _TBD_ | _TBD_ | Module 2 [code](/mod2b-cloudndb) (3.x) | Module 11 [code](/mod11-functions) (3.x) +12|Add App Engine `memcache`| _TBD_ | _TBD_ | Module 1 [code](/mod1-flask) (2.x) | Module 12 [code](/mod12a-memcache) (2.x) ### Table of contents diff --git a/mod0-baseline/.gcloudignore b/mod0-baseline/.gcloudignore index ee1a5ea..bcf97a8 100644 --- a/mod0-baseline/.gcloudignore +++ b/mod0-baseline/.gcloudignore @@ -9,13 +9,14 @@ .gcloudignore # Ignore source code control maintenance files -.git/ +.git .gitignore .hgignore .hg/ # Python files -*.py[cod] +*.pyc +*.pyo __pycache__/ /setup.cfg diff --git a/mod12a-memcache/.gcloudignore b/mod12a-memcache/.gcloudignore new file mode 100644 index 0000000..bcf97a8 --- /dev/null +++ b/mod12a-memcache/.gcloudignore @@ -0,0 +1,24 @@ +# This file specifies files that are *not* uploaded to Google Cloud Platform +# using gcloud. It follows the same syntax as .gitignore, with the addition of +# "#!include" directives (which insert the entries of the given .gitignore-style +# file at that point). +# +# For more information, run: +# $ gcloud topic gcloudignore +# +.gcloudignore + +# Ignore source code control maintenance files +.git +.gitignore +.hgignore +.hg/ + +# Python files +*.pyc +*.pyo +__pycache__/ +/setup.cfg + +# no need to upload README +README.md diff --git a/mod12a-memcache/README.md b/mod12a-memcache/README.md new file mode 100644 index 0000000..f4447f9 --- /dev/null +++ b/mod12a-memcache/README.md @@ -0,0 +1,3 @@ +# Module 12 - Add usage of App Engine `memcache` to Flask `ndb` sample app + +This repo folder is the corresponding Python 2 code to the Module 12 codelab (TBD). The tutorial STARTs with the Python 2 code in the [Module 1 repo folder](/mod1-flask) and leads developers through adding usage of App Engine's `memcache`, culminating in the code in this folder. diff --git a/mod12a-memcache/app.yaml b/mod12a-memcache/app.yaml new file mode 100644 index 0000000..0d4d8d6 --- /dev/null +++ b/mod12a-memcache/app.yaml @@ -0,0 +1,21 @@ +# Copyright 2021 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +runtime: python27 +threadsafe: yes +api_version: 1 + +handlers: +- url: /.* + script: main.app diff --git a/mod12a-memcache/appengine_config.py b/mod12a-memcache/appengine_config.py new file mode 100644 index 0000000..9760ebb --- /dev/null +++ b/mod12a-memcache/appengine_config.py @@ -0,0 +1,20 @@ +# Copyright 2021 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from google.appengine.ext import vendor + +# Set PATH to your libraries folder. +PATH = 'lib' +# Add libraries installed in the PATH folder. +vendor.add(PATH) diff --git a/mod12a-memcache/main.py b/mod12a-memcache/main.py new file mode 100644 index 0000000..621032a --- /dev/null +++ b/mod12a-memcache/main.py @@ -0,0 +1,51 @@ +# Copyright 2020 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from flask import Flask, render_template, request +from google.appengine.api import memcache +from google.appengine.ext import ndb + +app = Flask(__name__) +HOUR = 3600 + +class Visit(ndb.Model): + 'Visit entity registers visitor IP address & timestamp' + visitor = ndb.StringProperty() + timestamp = ndb.DateTimeProperty(auto_now_add=True) + +def store_visit(remote_addr, user_agent): + 'create new Visit entity in Datastore' + Visit(visitor='{}: {}'.format(remote_addr, user_agent)).put() + +def fetch_visits(limit): + 'get most recent visits' + return [v.to_dict() for v in Visit.query().order( + -Visit.timestamp).fetch(limit)] + +@app.route('/') +def root(): + 'main application (GET) handler' + # check for cached visits + ip_addr, usr_agt = request.remote_addr, request.user_agent + visitor = '{}: {}'.format(ip_addr, usr_agt) + visits = memcache.get('visits') + + # run DB query if cache empty or new visitor + if not visits or visits[0]['visitor'] != visitor: + store_visit(ip_addr, usr_agt) + visits = fetch_visits(10) + memcache.set('visits', visits, HOUR) + + # send context to template renderer + return render_template('index.html', visits=visits) diff --git a/mod12a-memcache/requirements.txt b/mod12a-memcache/requirements.txt new file mode 100644 index 0000000..5c508e5 --- /dev/null +++ b/mod12a-memcache/requirements.txt @@ -0,0 +1 @@ +flask==1.1.2 diff --git a/mod12a-memcache/templates/index.html b/mod12a-memcache/templates/index.html new file mode 100644 index 0000000..e140206 --- /dev/null +++ b/mod12a-memcache/templates/index.html @@ -0,0 +1,16 @@ + + + +VisitMe Example + + +

VisitMe example

+

Last 10 visits

+
    +{% for visit in visits %} +
  • {{ visit.timestamp.ctime() }} from {{ visit.visitor }}
  • +{% endfor %} +
+ + + diff --git a/mod7-gaetasks/templates/index.html b/mod7-gaetasks/templates/index.html index 9f33130..952961e 100644 --- a/mod7-gaetasks/templates/index.html +++ b/mod7-gaetasks/templates/index.html @@ -12,7 +12,7 @@

Last 10 visits

{% endfor %} -{% if oldest %} +{% if oldest is defined %} Deleting visits older than: {{ oldest }}

{% endif %} diff --git a/mod8-cloudtasks/templates/index.html b/mod8-cloudtasks/templates/index.html index 9f33130..952961e 100644 --- a/mod8-cloudtasks/templates/index.html +++ b/mod8-cloudtasks/templates/index.html @@ -12,7 +12,7 @@

Last 10 visits

{% endfor %} -{% if oldest %} +{% if oldest is defined %} Deleting visits older than: {{ oldest }}

{% endif %} diff --git a/mod9-py3fstasks/templates/index.html b/mod9-py3fstasks/templates/index.html index 9f33130..952961e 100644 --- a/mod9-py3fstasks/templates/index.html +++ b/mod9-py3fstasks/templates/index.html @@ -12,7 +12,7 @@

Last 10 visits

{% endfor %} -{% if oldest %} +{% if oldest is defined %} Deleting visits older than: {{ oldest }}

{% endif %} From b316a2d3a867aa2d84b850791e0413888dde93c6 Mon Sep 17 00:00:00 2001 From: wesley chun Date: Sun, 1 Aug 2021 23:56:39 -0700 Subject: [PATCH 10/56] add 3.x memcache sample; README updates --- README.md | 14 ++++---- mod12a-memcache/main.py | 6 ++-- mod12b-memcache/.gcloudignore | 24 +++++++++++++ mod12b-memcache/README.md | 5 +++ mod12b-memcache/app.yaml | 16 +++++++++ mod12b-memcache/main.py | 52 ++++++++++++++++++++++++++++ mod12b-memcache/requirements.txt | 2 ++ mod12b-memcache/templates/index.html | 16 +++++++++ mod1b-flask/README.md | 5 +-- 9 files changed, 126 insertions(+), 14 deletions(-) create mode 100644 mod12b-memcache/.gcloudignore create mode 100644 mod12b-memcache/README.md create mode 100644 mod12b-memcache/app.yaml create mode 100644 mod12b-memcache/main.py create mode 100644 mod12b-memcache/requirements.txt create mode 100644 mod12b-memcache/templates/index.html diff --git a/README.md b/README.md index 233540f..4b354f5 100644 --- a/README.md +++ b/README.md @@ -48,7 +48,7 @@ Neither upgrade may be particularly straightforward and can only be done seriall These are the challenges developers are facing, so the purpose of this content is to reduce the friction in this process and make things more prescriptive. Review the [runtimes chart](https://cloud.google.com/appengine/docs/standard/runtimes) to see the legacy services and current migration recommendation. The [migration guide overview](https://cloud.google.com/appengine/docs/standard/python/migrate-to-python3/migrating-services) has more information. -> **NOTE:** App Engine ([Flexible](https://cloud.google.com/appengine/docs/flexible/python/runtime?hl=en#interpreter)) is a next-gen service but is not within the scope of these tutorials. Curious developers can compare App Engine [Standard vs. Flexible](https://cloud.google.com/appengine/docs/the-appengine-environments) to find out more. Also, many of the Flexible use cases can now be handled by [Cloud Run](http://cloud.google.com/run). +> **NOTE:** App Engine ([Flexible](https://cloud.google.com/appengine/docs/flexible/python/runtime?hl=en#interpreter)) is a next-gen service but is not within the scope of these tutorials. Curious developers can compare App Engine [Standard vs. Flexible](https://cloud.google.com/appengine/docs/the-appengine-environments) to find out more. Also, many of the Flexible use cases can now be handled by [Cloud Run](http://cloud.run). ## Progression (START and FINISH) @@ -78,12 +78,12 @@ Module | Topic | Video | Codelab | START here | FINISH here 4|Migrate to Cloud Run with Docker| _TBD_ | [link](http://g.co/codelabs/pae-migrate-rundocker) | Module 2 [code](/mod2a-cloudndb) (2.x) & Module 3 [code](/mod3b-datastore) (3.x) | Module 4 [code](/mod4a-rundocker) (2.x) & [code](/mod4b-rundocker) (3.x) 5|Migrate to Cloud Run with Buildpacks| _TBD_ | [link](http://g.co/codelabs/pae-migrate-runbldpks) | Module 2 [code](/mod2b-cloudndb) (3.x) | Module 5 [code](/mod5-runbldpks) (3.x) 6|Migrate to Cloud Firestore (app)| _TBD_ | [link](http://g.co/codelabs/pae-migrate-firestore) | Module 3 [code](/mod3b-datastore) (3.x) | Module 6 [code](/mod6-firestore) (3.x) -7|Add App Engine push tasks| _TBD_ | [link](http://g.co/codelabs/pae-migrate-gaetasks) | Module 1 [code](/mod1-flask) (2.x) | Module 7 [code](/mod7-gaetasks) (2.x) +7|Add App Engine `taskqueue` push tasks| _TBD_ | [link](http://g.co/codelabs/pae-migrate-gaetasks) | Module 1 [code](/mod1-flask) (2.x) | Module 7 [code](/mod7-gaetasks) (2.x) 8|Migrate to Cloud Tasks| _TBD_ | [link](http://g.co/codelabs/pae-migrate-cloudtasks) | Module 7 [code](/mod7-gaetasks) (2.x) | Module 8 [code](/mod8-cloudtasks) (2.x) 9|Migrate to Python 3, Cloud Firestore & Cloud Tasks v2| _TBD_ | _TBD_ | Module 8 [code](/mod8-cloudtasks) (2.x) | Module 9 [code](/mod9-py3fstasks) (3.x) 10|Migrate to Cloud Firestore (data)| _TBD_ | _N/A_ | _N/A_ | _TBD_ 11|Migrate to Cloud Functions| _TBD_ | _TBD_ | Module 2 [code](/mod2b-cloudndb) (3.x) | Module 11 [code](/mod11-functions) (3.x) -12|Add App Engine `memcache`| _TBD_ | _TBD_ | Module 1 [code](/mod1-flask) (2.x) | Module 12 [code](/mod12a-memcache) (2.x) +12|Add App Engine `memcache`| _TBD_ | _TBD_ | Module 1 [code](/mod1-flask) (2.x) | Module 12 [code](/mod12a-memcache) (2.x) & [code](/mod12b-memcache) (3.x) ### Table of contents @@ -146,7 +146,7 @@ If there is a logical codelab to do immediately after completing one, they will - Module 5 - migrate to Cloud Run container with Cloud Buildpacks - Module 4 - migrate to Cloud Run container with Docker -- [Module 4 codelab](http://g.co/codelabs/pae-migrate-rundocker): **Migrate from App Engine to [Cloud Run](http://cloud.google.com/run) with Docker** +- [Module 4 codelab](http://g.co/codelabs/pae-migrate-rundocker): **Migrate from App Engine to Cloud Run with Docker** - **Optional** migration - "Containerize" your app (migrate your app to a container) with Docker - Python 2 @@ -161,7 +161,7 @@ If there is a logical codelab to do immediately after completing one, they will - Module 7 - add App Engine (push) tasks - Module 11 - migrate to Cloud Functions -- [Module 5 codelab](http://g.co/codelabs/pae-migrate-runbldpks): **Migrate from App Engine to [Cloud Run](http://cloud.google.com/run) with Cloud Buildpacks** +- [Module 5 codelab](http://g.co/codelabs/pae-migrate-runbldpks): **Migrate from App Engine to Cloud Run with Cloud Buildpacks** - **Optional** migration - "Containerize" your app (migrate your app to a container) with... - [Cloud Buildpacks]() which lets you containerize your app without `Dockerfile`s @@ -220,7 +220,7 @@ If there is a logical codelab to do immediately after completing one, they will - Module 5 - migrate to Cloud Run container with Cloud Buildpacks - Module 4 - migrate to Cloud Run container with Docker -- **Module 11 codelab** (TBD): **Migrate from App Engine to [Cloud Functions](http://cloud.google.com/run)** +- **Module 11 codelab** (TBD): **Migrate from App Engine to Cloud Functions** - **Optional** migration - Recommende for small apps or for breaking up large apps into multiple microservices - Python 3 only @@ -255,7 +255,7 @@ If your original app users does *not* have a user interface, i.e., mobile backen ## Accessing legacy services in second generation -Some legacy App Engine first generation platform (Python 2, Java 8, PHP 5, and Go 1.11 & older) services are available on a limited basis to second generation runtimes (Python 3, Java 11, PHP 7, and Go 1.12 & newer) as part of an early-access program. There is no video or codelab available yet, however the Module 1 Flask migration [Python 2 sample](/mod1-flask) is available in [Python 3](/mod1b-flask) if you have access. (See its README to sign-up if interested.) +Some legacy App Engine first generation platform (Python 2, Java 8, PHP 5, and Go 1.11 & older) services are available on a limited basis to second generation runtimes (Python 3, Java 11, PHP 7, and Go 1.12 & newer) as part of an early-access program. There are no videos or codelabs yet, however the Module 1 Flask migration using App Engine `ndb` [Python 2 sample ](/mod1-flask) is available in [Python 3](/mod1b-flask) if you have access. Similarly, the Module 12 addition of App Engine `memcache` [Python 2 sample](/mod12a-memcache) is also available in [Python 3](/mod12b-memcache). See their respective `README` files for more info and to sign-up if interested. ## References diff --git a/mod12a-memcache/main.py b/mod12a-memcache/main.py index 621032a..f0d887d 100644 --- a/mod12a-memcache/main.py +++ b/mod12a-memcache/main.py @@ -36,16 +36,16 @@ def fetch_visits(limit): @app.route('/') def root(): 'main application (GET) handler' - # check for cached visits + # check for (hour-)cached visits ip_addr, usr_agt = request.remote_addr, request.user_agent visitor = '{}: {}'.format(ip_addr, usr_agt) visits = memcache.get('visits') - # run DB query if cache empty or new visitor + # register visit & run DB query if cache empty or new visitor if not visits or visits[0]['visitor'] != visitor: store_visit(ip_addr, usr_agt) visits = fetch_visits(10) - memcache.set('visits', visits, HOUR) + memcache.set('visits', visits, HOUR) # must set() not add() here # send context to template renderer return render_template('index.html', visits=visits) diff --git a/mod12b-memcache/.gcloudignore b/mod12b-memcache/.gcloudignore new file mode 100644 index 0000000..bcf97a8 --- /dev/null +++ b/mod12b-memcache/.gcloudignore @@ -0,0 +1,24 @@ +# This file specifies files that are *not* uploaded to Google Cloud Platform +# using gcloud. It follows the same syntax as .gitignore, with the addition of +# "#!include" directives (which insert the entries of the given .gitignore-style +# file at that point). +# +# For more information, run: +# $ gcloud topic gcloudignore +# +.gcloudignore + +# Ignore source code control maintenance files +.git +.gitignore +.hgignore +.hg/ + +# Python files +*.pyc +*.pyo +__pycache__/ +/setup.cfg + +# no need to upload README +README.md diff --git a/mod12b-memcache/README.md b/mod12b-memcache/README.md new file mode 100644 index 0000000..3def656 --- /dev/null +++ b/mod12b-memcache/README.md @@ -0,0 +1,5 @@ +# Module 12 - Add usage of App Engine `memcache` to Flask `ndb` sample app + +This repo folder is the corresponding Python 3 code to the Module 12 codelab (TBD). The tutorial STARTs with the Python 2 code in the [Module 1 repo folder](/mod1-flask) and leads developers through adding usage of App Engine's `memcache`, followed by a bonus migration to Python 3, culminating in the code in this folder. + +> **LEGACY SERVICES PRIVATE PREVIEW**: Accessing legacy services such as App Engine `ndb` and `memcache` from Python 3 (and next generation App Engine in general) is currently in private preview and requires allowlisted access. Please complete [this form](https://docs.google.com/forms/d/e/1FAIpQLSd1hFLA2UFSYwIMxm9ZI3pwigORZBgjJRH0qrnhtE7nvhhRCQ/viewform) to sign-up for the program. Registering will get you in the queue to access the private preview, its documentation, and the announcements mailing list. For more info on what services are (or are not) available (at this time), see [this thread in the r/AppEngine Subreddit](https://reddit.com/r/AppEngine/comments/o9wr72). diff --git a/mod12b-memcache/app.yaml b/mod12b-memcache/app.yaml new file mode 100644 index 0000000..288ce87 --- /dev/null +++ b/mod12b-memcache/app.yaml @@ -0,0 +1,16 @@ +# Copyright 2021 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +runtime: python38 +app_engine_apis: true diff --git a/mod12b-memcache/main.py b/mod12b-memcache/main.py new file mode 100644 index 0000000..47fd8e3 --- /dev/null +++ b/mod12b-memcache/main.py @@ -0,0 +1,52 @@ +# Copyright 2020 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from flask import Flask, render_template, request +from google.appengine.api import memcache, wrap_wsgi_app +from google.appengine.ext import ndb + +app = Flask(__name__) +app.wsgi_app = wrap_wsgi_app(app.wsgi_app) +HOUR = 3600 + +class Visit(ndb.Model): + 'Visit entity registers visitor IP address & timestamp' + visitor = ndb.StringProperty() + timestamp = ndb.DateTimeProperty(auto_now_add=True) + +def store_visit(remote_addr, user_agent): + 'create new Visit entity in Datastore' + Visit(visitor='{}: {}'.format(remote_addr, user_agent)).put() + +def fetch_visits(limit): + 'get most recent visits' + return [v.to_dict() for v in Visit.query().order( + -Visit.timestamp).fetch(limit)] + +@app.route('/') +def root(): + 'main application (GET) handler' + # check for (hour-)cached visits + ip_addr, usr_agt = request.remote_addr, request.user_agent + visitor = '{}: {}'.format(ip_addr, usr_agt) + visits = memcache.get('visits') + + # register visit & run DB query if cache empty or new visitor + if not visits or visits[0]['visitor'] != visitor: + store_visit(ip_addr, usr_agt) + visits = fetch_visits(10) + memcache.set('visits', visits, HOUR) # must set() not add() here + + # send context to template renderer + return render_template('index.html', visits=visits) diff --git a/mod12b-memcache/requirements.txt b/mod12b-memcache/requirements.txt new file mode 100644 index 0000000..c661cf7 --- /dev/null +++ b/mod12b-memcache/requirements.txt @@ -0,0 +1,2 @@ +flask==1.1.2 +appengine-python-standard diff --git a/mod12b-memcache/templates/index.html b/mod12b-memcache/templates/index.html new file mode 100644 index 0000000..e140206 --- /dev/null +++ b/mod12b-memcache/templates/index.html @@ -0,0 +1,16 @@ + + + +VisitMe Example + + +

VisitMe example

+

Last 10 visits

+
    +{% for visit in visits %} +
  • {{ visit.timestamp.ctime() }} from {{ visit.visitor }}
  • +{% endfor %} +
+ + + diff --git a/mod1b-flask/README.md b/mod1b-flask/README.md index f90ca49..111eded 100644 --- a/mod1b-flask/README.md +++ b/mod1b-flask/README.md @@ -2,7 +2,4 @@ This repo folder is the corresponding Python 3 code to the [Module 1 codelab](http://g.co/codelabs/pae-migrate-flask). The tutorial STARTs with the Python 2 code in the [Module 0 repo folder](/mod0-baseline) and leads developers through migrating away from App Engine's `webapp2` web framework to Flask, culminating in the code in the [mod1-flask](/mod1-flask) folder. The codelab does **not** currently feature any bonus migration to Python 3, however, if you do so, and add in components from the legacy services private preview program (see sidebar below), it will culminate in the code in *this* (`mod1b-flask`) folder. In the [next (Module 2) codelab](http://g.co/codelabs/pae-migrate-cloudndb), users will migrate (the original Python 2 version of) this app from App Engine `ndb` to Cloud NDB for Datastore access. -> **LEGACY SERVICES PRIVATE PREVIEW**: Accessing a legacy service such as App Engine `ndb` from Python 3 (and next generation App Engine in general) is currently in private preview and requires access. Please complete [this form](https://docs.google.com/forms/d/e/1FAIpQLSd1hFLA2UFSYwIMxm9ZI3pwigORZBgjJRH0qrnhtE7nvhhRCQ/viewform) to sign-up for the program. Registering will get you in the queue to access the private preview, its documentation, and the announcements mailing list. For more info on what services are (or are not) available (at this time), see [this thread in the r/AppEngine Subreddit](https://reddit.com/r/AppEngine/comments/o9wr72). - - - +> **LEGACY SERVICES PRIVATE PREVIEW**: Accessing legacy services such as App Engine `ndb` from Python 3 (and next generation App Engine in general) is currently in private preview and requires allowlisted access. Please complete [this form](https://docs.google.com/forms/d/e/1FAIpQLSd1hFLA2UFSYwIMxm9ZI3pwigORZBgjJRH0qrnhtE7nvhhRCQ/viewform) to sign-up for the program. Registering will get you in the queue to access the private preview, its documentation, and the announcements mailing list. For more info on what services are (or are not) available (at this time), see [this thread in the r/AppEngine Subreddit](https://reddit.com/r/AppEngine/comments/o9wr72). From 195ad49d45997d3e247321f4d014024952e98fe1 Mon Sep 17 00:00:00 2001 From: wesley chun Date: Mon, 16 Aug 2021 18:56:45 -0700 Subject: [PATCH 11/56] add Mod7b; Mod9 updates --- README.md | 6 +-- mod7b-gaetasks/.gcloudignore | 24 +++++++++++ mod7b-gaetasks/README.md | 5 +++ mod7b-gaetasks/app.yaml | 16 +++++++ mod7b-gaetasks/main.py | 66 +++++++++++++++++++++++++++++ mod7b-gaetasks/requirements.txt | 2 + mod7b-gaetasks/templates/index.html | 19 +++++++++ mod9-py3fstasks/README.md | 10 +++++ mod9-py3fstasks/main.py | 2 +- mod9-py3fstasks/requirements.txt | 4 +- 10 files changed, 148 insertions(+), 6 deletions(-) create mode 100644 mod7b-gaetasks/.gcloudignore create mode 100644 mod7b-gaetasks/README.md create mode 100644 mod7b-gaetasks/app.yaml create mode 100644 mod7b-gaetasks/main.py create mode 100644 mod7b-gaetasks/requirements.txt create mode 100644 mod7b-gaetasks/templates/index.html diff --git a/README.md b/README.md index 4b354f5..bdd80c3 100644 --- a/README.md +++ b/README.md @@ -74,11 +74,11 @@ Module | Topic | Video | Codelab | START here | FINISH here 0|Baseline app| [link](http://twitter.com/googledevs/status/1407755281786867714?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_smsintro_201023&utm_content=-) | _N/A_ (no tutorial; just review the code) | _N/A_ | Module 0 [code](/mod0-baseline) (2.x) 1|Migrate to Flask| [link](https://twitter.com/googledevs/status/1413273119071031299?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrwa2flsk_201008&u%20tm_content=-) | [link](http://g.co/codelabs/pae-migrate-flask) | Module 0 [code](/mod0-baseline) (2.x) | Module 1 [code](/mod1-flask) (2.x) & [code](/mod1b-flask) (3.x) 2|Migrate to Cloud NDB| [link](http://twitter.com/googledevs/status/1413273119071031299?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrwa2flsk_201008&utm_content=-) | [link](http://g.co/codelabs/pae-migrate-cloudndb) | Module 1 [code](/mod1-flask) (2.x) | Module 2 [code](/mod2a-cloudndb) (2.x) & [code](/mod2b-cloudndb) (3.x) -3|Migrate to Cloud Datastore| _TBD_ | [link](http://g.co/codelabs/pae-migrate-datastore) | Module 2 [code](/mod2a-cloudndb) (2.x) & [code](/mod2b-cloudndb) (3.x) | Module 3 [code](/mod3a-datastore) (2.x) & [code](/mod3b-datastore) (3.x) +3|Migrate to Cloud Datastore| [link](http://twitter.com/googledevs/status/1422966928910393347?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrcloudds_201003&utm_content=-) | [link](http://g.co/codelabs/pae-migrate-datastore) | Module 2 [code](/mod2a-cloudndb) (2.x) & [code](/mod2b-cloudndb) (3.x) | Module 3 [code](/mod3a-datastore) (2.x) & [code](/mod3b-datastore) (3.x) 4|Migrate to Cloud Run with Docker| _TBD_ | [link](http://g.co/codelabs/pae-migrate-rundocker) | Module 2 [code](/mod2a-cloudndb) (2.x) & Module 3 [code](/mod3b-datastore) (3.x) | Module 4 [code](/mod4a-rundocker) (2.x) & [code](/mod4b-rundocker) (3.x) 5|Migrate to Cloud Run with Buildpacks| _TBD_ | [link](http://g.co/codelabs/pae-migrate-runbldpks) | Module 2 [code](/mod2b-cloudndb) (3.x) | Module 5 [code](/mod5-runbldpks) (3.x) 6|Migrate to Cloud Firestore (app)| _TBD_ | [link](http://g.co/codelabs/pae-migrate-firestore) | Module 3 [code](/mod3b-datastore) (3.x) | Module 6 [code](/mod6-firestore) (3.x) -7|Add App Engine `taskqueue` push tasks| _TBD_ | [link](http://g.co/codelabs/pae-migrate-gaetasks) | Module 1 [code](/mod1-flask) (2.x) | Module 7 [code](/mod7-gaetasks) (2.x) +7|Add App Engine `taskqueue` push tasks| _TBD_ | [link](http://g.co/codelabs/pae-migrate-gaetasks) | Module 1 [code](/mod1-flask) (2.x) | Module 7 [code](/mod7-gaetasks) (2.x) & [code](/mod7b-gaetasks) (3.x) 8|Migrate to Cloud Tasks| _TBD_ | [link](http://g.co/codelabs/pae-migrate-cloudtasks) | Module 7 [code](/mod7-gaetasks) (2.x) | Module 8 [code](/mod8-cloudtasks) (2.x) 9|Migrate to Python 3, Cloud Firestore & Cloud Tasks v2| _TBD_ | _TBD_ | Module 8 [code](/mod8-cloudtasks) (2.x) | Module 9 [code](/mod9-py3fstasks) (3.x) 10|Migrate to Cloud Firestore (data)| _TBD_ | _N/A_ | _N/A_ | _TBD_ @@ -255,7 +255,7 @@ If your original app users does *not* have a user interface, i.e., mobile backen ## Accessing legacy services in second generation -Some legacy App Engine first generation platform (Python 2, Java 8, PHP 5, and Go 1.11 & older) services are available on a limited basis to second generation runtimes (Python 3, Java 11, PHP 7, and Go 1.12 & newer) as part of an early-access program. There are no videos or codelabs yet, however the Module 1 Flask migration using App Engine `ndb` [Python 2 sample ](/mod1-flask) is available in [Python 3](/mod1b-flask) if you have access. Similarly, the Module 12 addition of App Engine `memcache` [Python 2 sample](/mod12a-memcache) is also available in [Python 3](/mod12b-memcache). See their respective `README` files for more info and to sign-up if interested. +Some legacy App Engine first generation platform (Python 2, Java 8, PHP 5, and Go 1.11 & older) services are available on a limited basis to second generation runtimes (Python 3, Java 11, PHP 7, and Go 1.12 & newer) as part of an early-access program. There are no videos or codelabs yet, however the Module 1 Flask migration using App Engine `ndb` [Python 2 sample ](/mod1-flask) is available in [Python 3](/mod1b-flask) if you have access. Similarly, Python 3 editions are also available for for Module 7 and 12 which add usage of App Engine (`taskqueue`) and (`memcache`), respectively. See their respective `README` files for more info and to sign-up if interested. ## References diff --git a/mod7b-gaetasks/.gcloudignore b/mod7b-gaetasks/.gcloudignore new file mode 100644 index 0000000..bcf97a8 --- /dev/null +++ b/mod7b-gaetasks/.gcloudignore @@ -0,0 +1,24 @@ +# This file specifies files that are *not* uploaded to Google Cloud Platform +# using gcloud. It follows the same syntax as .gitignore, with the addition of +# "#!include" directives (which insert the entries of the given .gitignore-style +# file at that point). +# +# For more information, run: +# $ gcloud topic gcloudignore +# +.gcloudignore + +# Ignore source code control maintenance files +.git +.gitignore +.hgignore +.hg/ + +# Python files +*.pyc +*.pyo +__pycache__/ +/setup.cfg + +# no need to upload README +README.md diff --git a/mod7b-gaetasks/README.md b/mod7b-gaetasks/README.md new file mode 100644 index 0000000..f065500 --- /dev/null +++ b/mod7b-gaetasks/README.md @@ -0,0 +1,5 @@ +# Module 7 - Add usage of App Engine `taskqueue` to Flask `ndb` sample app + +This repo folder is the corresponding Python 3 code to the [Module 7 codelab](http://g.co/codelabs/pae-migrate-gaetasks). The tutorial STARTs with the Python 2 code in the [Module 1 repo folder](/mod1-flask) and leads developers through adding usage of App Engine's `taskqueue`, culminating in the code in the [mod7-gaetasks](/mod7-gaetasks) folder. The codelab does **not** currently feature any bonus migration to Python 3, however, if you do so, and add in components from the legacy services private preview program (see sidebar below), it will culminate in the code in *this* (`mod7b-gaetasks`) folder. In the [next (Module 8) codelab](http://g.co/codelabs/pae-migrate-cloudtasks), users will migrate (the original Python 2 version of) this app from App Engine `taskqueue` to Cloud Tasks. + +> **LEGACY SERVICES PRIVATE PREVIEW**: Accessing legacy services such as App Engine `ndb` and `taskqueue` from Python 3 (and next generation App Engine in general) is currently in private preview and requires allowlisted access. Please complete [this form](https://docs.google.com/forms/d/e/1FAIpQLSd1hFLA2UFSYwIMxm9ZI3pwigORZBgjJRH0qrnhtE7nvhhRCQ/viewform) to sign-up for the program. Registering will get you in the queue to access the private preview, its documentation, and the announcements mailing list. For more info on what services are (or are not) available (at this time), see [this thread in the r/AppEngine Subreddit](https://reddit.com/r/AppEngine/comments/o9wr72). diff --git a/mod7b-gaetasks/app.yaml b/mod7b-gaetasks/app.yaml new file mode 100644 index 0000000..288ce87 --- /dev/null +++ b/mod7b-gaetasks/app.yaml @@ -0,0 +1,16 @@ +# Copyright 2021 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +runtime: python38 +app_engine_apis: true diff --git a/mod7b-gaetasks/main.py b/mod7b-gaetasks/main.py new file mode 100644 index 0000000..35a1188 --- /dev/null +++ b/mod7b-gaetasks/main.py @@ -0,0 +1,66 @@ +# Copyright 2021 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from datetime import datetime +import logging +import time +from flask import Flask, render_template, request +from google.appengine.api import taskqueue, wrap_wsgi_app +from google.appengine.ext import ndb + +app = Flask(__name__) +app.wsgi_app = wrap_wsgi_app(app.wsgi_app) + +class Visit(ndb.Model): + 'Visit entity registers visitor IP address & timestamp' + visitor = ndb.StringProperty() + timestamp = ndb.DateTimeProperty(auto_now_add=True) + +def store_visit(remote_addr, user_agent): + 'create new Visit entity in Datastore' + Visit(visitor='{}: {}'.format(remote_addr, user_agent)).put() + +def fetch_visits(limit): + 'get most recent visits & add task to delete older visits' + data = Visit.query().order(-Visit.timestamp).fetch(limit) + oldest = time.mktime(data[-1].timestamp.timetuple()) + oldest_str = time.ctime(oldest) + logging.info('Delete entities older than %s' % oldest_str) + taskqueue.add(url='/trim', params={'oldest': oldest}) + return (v.to_dict() for v in data), oldest_str + +@app.route('/trim', methods=['POST']) +def trim(): + '(push) task queue handler to delete oldest visits' + oldest = request.form.get('oldest', type=float) + keys = Visit.query( + Visit.timestamp < datetime.fromtimestamp(oldest) + ).fetch(keys_only=True) + nkeys = len(keys) + if nkeys: + logging.info('Deleting %d entities: %s' % ( + nkeys, ', '.join(str(k.id()) for k in keys))) + ndb.delete_multi(keys) + else: + logging.info( + 'No entities older than: %s' % time.ctime(oldest)) + return '' # need to return SOME string w/200 + +@app.route('/') +def root(): + 'main application (GET) handler' + store_visit(request.remote_addr, request.user_agent) + visits, oldest = fetch_visits(10) + context = {'visits': visits, 'oldest': oldest} + return render_template('index.html', **context) diff --git a/mod7b-gaetasks/requirements.txt b/mod7b-gaetasks/requirements.txt new file mode 100644 index 0000000..c661cf7 --- /dev/null +++ b/mod7b-gaetasks/requirements.txt @@ -0,0 +1,2 @@ +flask==1.1.2 +appengine-python-standard diff --git a/mod7b-gaetasks/templates/index.html b/mod7b-gaetasks/templates/index.html new file mode 100644 index 0000000..952961e --- /dev/null +++ b/mod7b-gaetasks/templates/index.html @@ -0,0 +1,19 @@ + + + +VisitMe Example + + +

VisitMe example

+

Last 10 visits

+
    +{% for visit in visits %} +
  • {{ visit.timestamp.ctime() }} from {{ visit.visitor }}
  • +{% endfor %} +
+ +{% if oldest is defined %} + Deleting visits older than: {{ oldest }}

+{% endif %} + + diff --git a/mod9-py3fstasks/README.md b/mod9-py3fstasks/README.md index 4b636ca..6abc0ea 100644 --- a/mod9-py3fstasks/README.md +++ b/mod9-py3fstasks/README.md @@ -1,3 +1,13 @@ # Module 9 - Migrate from Python 2 to 3 and Cloud NDB to Cloud Firestore This repo folder is the corresponding Python 3 code to the [Module 9 codelab](http://g.co/codelabs/pae-migrate-py3fstasks). The tutorial STARTs with the Python 2 code in the [Module 8 repo folder](/mod7-cloudtasks) and leads developers through migrating from Python 2 to 3, Cloud NDB to Cloud Firestore (skipping over a Cloud Datstore migration) plus any changes from Cloud Tasks v1 to v2, culminating in the code in this folder. One major addition to look for here vs. Module 8 is that App Engine `taskqueue` creates a `default` push queue while Cloud Tasks does not, so that now has to be done in code. + +NOTE: The deletion process in this app is "one-at-a-time." If your app requires deletion of more than a few documents, consider switching to the batch model. In this case, you would replace `_delete_docs()` with: + + def _delete_docs(visits): + 'app-internal generator deleting old FS visit documents' + batch = fs_client.batch() + for visit in visits: + batch.delete(visit.reference) + yield visit.id + batch.commit() diff --git a/mod9-py3fstasks/main.py b/mod9-py3fstasks/main.py index d6c524a..0f7b12c 100644 --- a/mod9-py3fstasks/main.py +++ b/mod9-py3fstasks/main.py @@ -69,7 +69,7 @@ def fetch_visits(limit): return visits, oldest_str def _delete_docs(visits): - 'app-internal generator deleteing all old FS visit documents' + 'app-internal generator deleting old FS visit documents' for visit in visits: visit.reference.delete() yield visit.id diff --git a/mod9-py3fstasks/requirements.txt b/mod9-py3fstasks/requirements.txt index ed5a882..ca6d2f7 100644 --- a/mod9-py3fstasks/requirements.txt +++ b/mod9-py3fstasks/requirements.txt @@ -1,3 +1,3 @@ flask==1.1.2 -google-cloud-firestore==2.1.3 -google-cloud-tasks==2.3.0 +google-cloud-firestore==2.2.0 +google-cloud-tasks==2.5.1 From 234b878e791cafb45cbc4c96db12801f3eb3742d Mon Sep 17 00:00:00 2001 From: wesley chun Date: Thu, 19 Aug 2021 10:07:02 -0700 Subject: [PATCH 12/56] add Module 4 video; typo fixes --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index bdd80c3..3c7165f 100644 --- a/README.md +++ b/README.md @@ -75,7 +75,7 @@ Module | Topic | Video | Codelab | START here | FINISH here 1|Migrate to Flask| [link](https://twitter.com/googledevs/status/1413273119071031299?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrwa2flsk_201008&u%20tm_content=-) | [link](http://g.co/codelabs/pae-migrate-flask) | Module 0 [code](/mod0-baseline) (2.x) | Module 1 [code](/mod1-flask) (2.x) & [code](/mod1b-flask) (3.x) 2|Migrate to Cloud NDB| [link](http://twitter.com/googledevs/status/1413273119071031299?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrwa2flsk_201008&utm_content=-) | [link](http://g.co/codelabs/pae-migrate-cloudndb) | Module 1 [code](/mod1-flask) (2.x) | Module 2 [code](/mod2a-cloudndb) (2.x) & [code](/mod2b-cloudndb) (3.x) 3|Migrate to Cloud Datastore| [link](http://twitter.com/googledevs/status/1422966928910393347?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrcloudds_201003&utm_content=-) | [link](http://g.co/codelabs/pae-migrate-datastore) | Module 2 [code](/mod2a-cloudndb) (2.x) & [code](/mod2b-cloudndb) (3.x) | Module 3 [code](/mod3a-datastore) (2.x) & [code](/mod3b-datastore) (3.x) -4|Migrate to Cloud Run with Docker| _TBD_ | [link](http://g.co/codelabs/pae-migrate-rundocker) | Module 2 [code](/mod2a-cloudndb) (2.x) & Module 3 [code](/mod3b-datastore) (3.x) | Module 4 [code](/mod4a-rundocker) (2.x) & [code](/mod4b-rundocker) (3.x) +4|Migrate to Cloud Run with Docker| [link](https://twitter.com/googledevs/status/1428041270702735362?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrcrdckr_sms_201017&utm_content=-)| [link](http://g.co/codelabs/pae-migrate-rundocker) | Module 2 [code](/mod2a-cloudndb) (2.x) & Module 3 [code](/mod3b-datastore) (3.x) | Module 4 [code](/mod4a-rundocker) (2.x) & [code](/mod4b-rundocker) (3.x) 5|Migrate to Cloud Run with Buildpacks| _TBD_ | [link](http://g.co/codelabs/pae-migrate-runbldpks) | Module 2 [code](/mod2b-cloudndb) (3.x) | Module 5 [code](/mod5-runbldpks) (3.x) 6|Migrate to Cloud Firestore (app)| _TBD_ | [link](http://g.co/codelabs/pae-migrate-firestore) | Module 3 [code](/mod3b-datastore) (3.x) | Module 6 [code](/mod6-firestore) (3.x) 7|Add App Engine `taskqueue` push tasks| _TBD_ | [link](http://g.co/codelabs/pae-migrate-gaetasks) | Module 1 [code](/mod1-flask) (2.x) | Module 7 [code](/mod7-gaetasks) (2.x) & [code](/mod7b-gaetasks) (3.x) @@ -255,7 +255,7 @@ If your original app users does *not* have a user interface, i.e., mobile backen ## Accessing legacy services in second generation -Some legacy App Engine first generation platform (Python 2, Java 8, PHP 5, and Go 1.11 & older) services are available on a limited basis to second generation runtimes (Python 3, Java 11, PHP 7, and Go 1.12 & newer) as part of an early-access program. There are no videos or codelabs yet, however the Module 1 Flask migration using App Engine `ndb` [Python 2 sample ](/mod1-flask) is available in [Python 3](/mod1b-flask) if you have access. Similarly, Python 3 editions are also available for for Module 7 and 12 which add usage of App Engine (`taskqueue`) and (`memcache`), respectively. See their respective `README` files for more info and to sign-up if interested. +Some legacy App Engine first generation platform (Python 2, Java 8, PHP 5, and Go 1.11 & older) services are available on a limited basis to second generation runtimes (Python 3, Java 11, PHP 7, and Go 1.12 & newer) as part of an early-access program. There are no videos or codelabs yet, however the Module 1 Flask migration using App Engine `ndb` [Python 2 sample ](/mod1-flask) is available in [Python 3](/mod1b-flask) if you have access. Similarly, Python 3 editions are also available for Modules 7 and 12 which add usage of App Engine `taskqueue` and `memcache`, respectively. See their respective `README` files for more info and to sign-up if interested. ## References From 093ab9138404bd8f877eacb979c7e0af548e9ecf Mon Sep 17 00:00:00 2001 From: wesley chun Date: Tue, 24 Aug 2021 19:34:50 -0700 Subject: [PATCH 13/56] README update, Mod9 2.x backport, Mod12 naming convention --- README.md | 12 ++++++---- .../.gcloudignore | 0 {mod12a-memcache => mod12-memcache}/README.md | 0 {mod12a-memcache => mod12-memcache}/app.yaml | 0 .../appengine_config.py | 0 {mod12a-memcache => mod12-memcache}/main.py | 0 .../requirements.txt | 0 .../templates/index.html | 0 mod9-py3fstasks/README.md | 23 ++++++++++++++++++- 9 files changed, 29 insertions(+), 6 deletions(-) rename {mod12a-memcache => mod12-memcache}/.gcloudignore (100%) rename {mod12a-memcache => mod12-memcache}/README.md (100%) rename {mod12a-memcache => mod12-memcache}/app.yaml (100%) rename {mod12a-memcache => mod12-memcache}/appengine_config.py (100%) rename {mod12a-memcache => mod12-memcache}/main.py (100%) rename {mod12a-memcache => mod12-memcache}/requirements.txt (100%) rename {mod12a-memcache => mod12-memcache}/templates/index.html (100%) diff --git a/README.md b/README.md index 3c7165f..d2354e7 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,7 @@ -# Python App Engine app migration -### To modern runtime, Cloud services, Python 3, and Cloud Run containers +# Modernizing Google Cloud serverless compute applications +### To newer Cloud services and other serverless platforms + +This is the corresponding repository to the [Serverless Migration Station](https://bit.ly/3xk2Swi) video series whose goal is to help users on a Google Cloud serverless compute platform modernize to newer Cloud products or other serverless compute platforms. Each modernization migration aims to feature a video, codelab (self-paced, hands-on tutorial), and code samples. The content initially focuses on App Engine, the earliest Google Cloud users. [Google App Engine](https://cloud.google.com/appengine) (Standard) has undergone significant changes between the legacy and next generation platforms. To address this, we've created a set of codelabs (free, online, self-paced, hands-on tutorials) and corresponding videos (when available) to show developers how to perform individual migrations they can apply to modernize their apps for the latest runtimes, with this repo managing the samples from those codelabs. @@ -21,9 +23,9 @@ Each codelab begins with a "START" code base then walks developers through that ## Cost -App Engine is not a free service. While you may not have needed to enable billing in App Engine's early days, [all applications now require an active billing account](https://cloud.google.com/appengine/docs/standard/payment-instrument) backed by a financial instrument (usually a credit card). Don't worry, App Engine (and other GCP products) still have an ["Always Free" tier](https://cloud.google.com/free/docs/gcp-free-tier#free-tier-usage-limits) and as long as you stay within those limits, you won't incur any charges. Also check the App Engine [pricing](https://cloud.google.com/appengine/pricing) and [quotas](https://cloud.google.com/appengine/quotas) pages for more information. +App Engine, Cloud Functions, and Cloud Run are not free services. While you may not have needed to enable billing in App Engine's early days, [all applications now require an active billing account](https://cloud.google.com/appengine/docs/standard/payment-instrument) backed by a financial instrument (usually a credit card). Don't worry, App Engine (and other GCP products) still have an ["Always Free" tier](https://cloud.google.com/free/docs/gcp-free-tier#free-tier-usage-limits) and as long as you stay within those limits, you won't incur any charges. Also check the App Engine [pricing](https://cloud.google.com/appengine/pricing) and [quotas](https://cloud.google.com/appengine/quotas) pages for more information. -Furthermore, deploying to GCP serverless platforms incur [minor build and storage costs](https://cloud.google.com/appengine/pricing#pricing-for-related-google-cloud-products). [Cloud Build](https://cloud.google.com/build/pricing) has its own free quota as does [Cloud Storage](https://cloud.google.com/storage/pricing#cloud-storage-always-free). For greater transparency, Cloud Build builds your application image which is than sent to the [Cloud Container Registry](https://cloud.google.com/container-registry/pricing); storage of that image uses up some of that (Cloud Storage) quota as does network egress when transferring that image to the service you're deploying to. However you may live in region that does not have such a free tier, so be aware of your storage usage to minimize potential costs. (You may look at what storage you're using and how much, including deleting build artifacts via [your Cloud Storage browser](https://console.cloud.google.com/storage/browser).) +Furthermore, deploying to GCP serverless platforms incur [minor build and storage costs](https://cloud.google.com/appengine/pricing#pricing-for-related-google-cloud-products). [Cloud Build](https://cloud.google.com/build/pricing) has its own free quota as does [Cloud Storage](https://cloud.google.com/storage/pricing#cloud-storage-always-free). For greater transparency, Cloud Build builds your application image which is than sent to the [Cloud Container Registry](https://cloud.google.com/container-registry/pricing) or [Artifact Registry](https://cloud.google.com/artifact-registry/pricing), its successor; storage of that image uses up some of that (Cloud Storage) quota as does network egress when transferring that image to the service you're deploying to. However you may live in region that does not have such a free tier, so be aware of your storage usage to minimize potential costs. (You may look at what storage you're using and how much, including deleting build artifacts via [your Cloud Storage browser](https://console.cloud.google.com/storage/browser).) ## Why @@ -83,7 +85,7 @@ Module | Topic | Video | Codelab | START here | FINISH here 9|Migrate to Python 3, Cloud Firestore & Cloud Tasks v2| _TBD_ | _TBD_ | Module 8 [code](/mod8-cloudtasks) (2.x) | Module 9 [code](/mod9-py3fstasks) (3.x) 10|Migrate to Cloud Firestore (data)| _TBD_ | _N/A_ | _N/A_ | _TBD_ 11|Migrate to Cloud Functions| _TBD_ | _TBD_ | Module 2 [code](/mod2b-cloudndb) (3.x) | Module 11 [code](/mod11-functions) (3.x) -12|Add App Engine `memcache`| _TBD_ | _TBD_ | Module 1 [code](/mod1-flask) (2.x) | Module 12 [code](/mod12a-memcache) (2.x) & [code](/mod12b-memcache) (3.x) +12|Add App Engine `memcache`| _TBD_ | _TBD_ | Module 1 [code](/mod1-flask) (2.x) | Module 12 [code](/mod12-memcache) (2.x) & [code](/mod12b-memcache) (3.x) ### Table of contents diff --git a/mod12a-memcache/.gcloudignore b/mod12-memcache/.gcloudignore similarity index 100% rename from mod12a-memcache/.gcloudignore rename to mod12-memcache/.gcloudignore diff --git a/mod12a-memcache/README.md b/mod12-memcache/README.md similarity index 100% rename from mod12a-memcache/README.md rename to mod12-memcache/README.md diff --git a/mod12a-memcache/app.yaml b/mod12-memcache/app.yaml similarity index 100% rename from mod12a-memcache/app.yaml rename to mod12-memcache/app.yaml diff --git a/mod12a-memcache/appengine_config.py b/mod12-memcache/appengine_config.py similarity index 100% rename from mod12a-memcache/appengine_config.py rename to mod12-memcache/appengine_config.py diff --git a/mod12a-memcache/main.py b/mod12-memcache/main.py similarity index 100% rename from mod12a-memcache/main.py rename to mod12-memcache/main.py diff --git a/mod12a-memcache/requirements.txt b/mod12-memcache/requirements.txt similarity index 100% rename from mod12a-memcache/requirements.txt rename to mod12-memcache/requirements.txt diff --git a/mod12a-memcache/templates/index.html b/mod12-memcache/templates/index.html similarity index 100% rename from mod12a-memcache/templates/index.html rename to mod12-memcache/templates/index.html diff --git a/mod9-py3fstasks/README.md b/mod9-py3fstasks/README.md index 6abc0ea..cbbd97f 100644 --- a/mod9-py3fstasks/README.md +++ b/mod9-py3fstasks/README.md @@ -2,7 +2,7 @@ This repo folder is the corresponding Python 3 code to the [Module 9 codelab](http://g.co/codelabs/pae-migrate-py3fstasks). The tutorial STARTs with the Python 2 code in the [Module 8 repo folder](/mod7-cloudtasks) and leads developers through migrating from Python 2 to 3, Cloud NDB to Cloud Firestore (skipping over a Cloud Datstore migration) plus any changes from Cloud Tasks v1 to v2, culminating in the code in this folder. One major addition to look for here vs. Module 8 is that App Engine `taskqueue` creates a `default` push queue while Cloud Tasks does not, so that now has to be done in code. -NOTE: The deletion process in this app is "one-at-a-time." If your app requires deletion of more than a few documents, consider switching to the batch model. In this case, you would replace `_delete_docs()` with: +**NOTE: Batch delete**: The deletion process in this app is "one-at-a-time." If your app requires deletion of more than a few documents, consider switching to the batch model. In this case, you would replace `_delete_docs()` with: def _delete_docs(visits): 'app-internal generator deleting old FS visit documents' @@ -11,3 +11,24 @@ NOTE: The deletion process in this app is "one-at-a-time." If your app requires batch.delete(visit.reference) yield visit.id batch.commit() + +**NOTE: Backport to Python 2**: When migrating this app to Python 3, we added a Python 3 dependency: the `print()` function. If for any reason you need to get back on Python 2 App Engine, you would have to: + + 1. Decide on your logging strategy. The Python 2 App Engine runtime now allows writing to `stdout`, so you don't have to revert back to `logging.info()` (or preferred logging level), however writing to `stdout` defaults to `logging.error()`. If that is acceptable and to continue with `print()`, add a `__future__.print_function` import above the others so the top of `main.py` looks like this: + + from __future__ import print_function + from datetime import datetime + import json + import time + from flask import Flask, render_template, request + import google.auth + from google.cloud import firestore, tasks + + 2. Revert back to your Python 2 configuration files. For this app, it would be Module 8's [`app.yaml`](https://github.com/googlecodelabs/migrate-python2-appengine/blob/master/mod8-cloudtasks/app.yaml) and [`appengine_config.py`](https://github.com/googlecodelabs/migrate-python2-appengine/blob/master/mod8-cloudtasks/appengine_config.py) files. + + 3. Revert all package versions from `requirements.txt` to get the latest/last(?) package versions for Python 2 as well as run `pip install -t lib -r requirements.txt` again. Here is what ours looks like: + + flask==1.1.2 + google-cloud-firestore==1.9.0 + google-cloud-tasks==1.5.0 + From 6a9fbb8049f357b3a207963f5c6b74e4a41a4a65 Mon Sep 17 00:00:00 2001 From: wesley chun Date: Wed, 1 Sep 2021 17:56:58 -0700 Subject: [PATCH 14/56] add Mod5 video, update description --- README.md | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index d2354e7..24092e0 100644 --- a/README.md +++ b/README.md @@ -1,9 +1,13 @@ # Modernizing Google Cloud serverless compute applications -### To newer Cloud services and other serverless platforms +### To the latest Cloud services and serverless platforms -This is the corresponding repository to the [Serverless Migration Station](https://bit.ly/3xk2Swi) video series whose goal is to help users on a Google Cloud serverless compute platform modernize to newer Cloud products or other serverless compute platforms. Each modernization migration aims to feature a video, codelab (self-paced, hands-on tutorial), and code samples. The content initially focuses on App Engine, the earliest Google Cloud users. +This is the corresponding repository to the [Serverless Migration Station](https://bit.ly/3xk2Swi) video series whose goal is to help users on a Google Cloud serverless compute platform modernize to newer Cloud products or other serverless compute platforms. Each modernization migration aims to feature a video, codelab (self-paced, hands-on tutorial), and code samples. The content initially focuses on App Engine and Google's earliest Cloud users. -[Google App Engine](https://cloud.google.com/appengine) (Standard) has undergone significant changes between the legacy and next generation platforms. To address this, we've created a set of codelabs (free, online, self-paced, hands-on tutorials) and corresponding videos (when available) to show developers how to perform individual migrations they can apply to modernize their apps for the latest runtimes, with this repo managing the samples from those codelabs. +[Google App Engine](https://cloud.google.com/appengine) (Standard) has undergone significant changes between the legacy and next generation platforms. To address this, we've created a set of codelabs (free, online, self-paced, hands-on tutorials) and corresponding videos (when available) to show developers how to perform individual migrations they can apply to modernize their apps for the latest runtimes, with this repo managing the samples from those codelabs. Codelab content typically falls into one of these three topics: + +1. Migrate from a legacy App Engine service to a Cloud equivalent product +1. Migrate to a newer Cloud product, service, or API client library +1. Migrate to another or newer Google Cloud serverless compute platform Each codelab begins with a "START" code base then walks developers through that migration step, resulting in a "FINISH" repo. If you make any mistakes along the way, you can always go back to START or compare your code with what's in the FINISH folder to see the differences. We also want to help you port to the Python 3 runtime, so some codelabs contain a bonus section for that purpose. @@ -78,7 +82,7 @@ Module | Topic | Video | Codelab | START here | FINISH here 2|Migrate to Cloud NDB| [link](http://twitter.com/googledevs/status/1413273119071031299?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrwa2flsk_201008&utm_content=-) | [link](http://g.co/codelabs/pae-migrate-cloudndb) | Module 1 [code](/mod1-flask) (2.x) | Module 2 [code](/mod2a-cloudndb) (2.x) & [code](/mod2b-cloudndb) (3.x) 3|Migrate to Cloud Datastore| [link](http://twitter.com/googledevs/status/1422966928910393347?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrcloudds_201003&utm_content=-) | [link](http://g.co/codelabs/pae-migrate-datastore) | Module 2 [code](/mod2a-cloudndb) (2.x) & [code](/mod2b-cloudndb) (3.x) | Module 3 [code](/mod3a-datastore) (2.x) & [code](/mod3b-datastore) (3.x) 4|Migrate to Cloud Run with Docker| [link](https://twitter.com/googledevs/status/1428041270702735362?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrcrdckr_sms_201017&utm_content=-)| [link](http://g.co/codelabs/pae-migrate-rundocker) | Module 2 [code](/mod2a-cloudndb) (2.x) & Module 3 [code](/mod3b-datastore) (3.x) | Module 4 [code](/mod4a-rundocker) (2.x) & [code](/mod4b-rundocker) (3.x) -5|Migrate to Cloud Run with Buildpacks| _TBD_ | [link](http://g.co/codelabs/pae-migrate-runbldpks) | Module 2 [code](/mod2b-cloudndb) (3.x) | Module 5 [code](/mod5-runbldpks) (3.x) +5|Migrate to Cloud Run with Buildpacks| [link](https://twitter.com/googledevs/status/1433113274984271875?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrcrbdpk_sms_201031&utm_content=-) | [link](http://g.co/codelabs/pae-migrate-runbldpks) | Module 2 [code](/mod2b-cloudndb) (3.x) | Module 5 [code](/mod5-runbldpks) (3.x) 6|Migrate to Cloud Firestore (app)| _TBD_ | [link](http://g.co/codelabs/pae-migrate-firestore) | Module 3 [code](/mod3b-datastore) (3.x) | Module 6 [code](/mod6-firestore) (3.x) 7|Add App Engine `taskqueue` push tasks| _TBD_ | [link](http://g.co/codelabs/pae-migrate-gaetasks) | Module 1 [code](/mod1-flask) (2.x) | Module 7 [code](/mod7-gaetasks) (2.x) & [code](/mod7b-gaetasks) (3.x) 8|Migrate to Cloud Tasks| _TBD_ | [link](http://g.co/codelabs/pae-migrate-cloudtasks) | Module 7 [code](/mod7-gaetasks) (2.x) | Module 8 [code](/mod8-cloudtasks) (2.x) From 11b40212f85a1ec3c1ea8f2a2356b3ca25a73d01 Mon Sep 17 00:00:00 2001 From: wesley chun Date: Fri, 10 Sep 2021 01:35:53 -0700 Subject: [PATCH 15/56] fix Mod2 video link --- README.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 24092e0..88ee684 100644 --- a/README.md +++ b/README.md @@ -79,7 +79,8 @@ Module | Topic | Video | Codelab | START here | FINISH here --- | --- | --- | --- | --- | --- 0|Baseline app| [link](http://twitter.com/googledevs/status/1407755281786867714?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_smsintro_201023&utm_content=-) | _N/A_ (no tutorial; just review the code) | _N/A_ | Module 0 [code](/mod0-baseline) (2.x) 1|Migrate to Flask| [link](https://twitter.com/googledevs/status/1413273119071031299?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrwa2flsk_201008&u%20tm_content=-) | [link](http://g.co/codelabs/pae-migrate-flask) | Module 0 [code](/mod0-baseline) (2.x) | Module 1 [code](/mod1-flask) (2.x) & [code](/mod1b-flask) (3.x) -2|Migrate to Cloud NDB| [link](http://twitter.com/googledevs/status/1413273119071031299?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrwa2flsk_201008&utm_content=-) | [link](http://g.co/codelabs/pae-migrate-cloudndb) | Module 1 [code](/mod1-flask) (2.x) | Module 2 [code](/mod2a-cloudndb) (2.x) & [code](/mod2b-cloudndb) (3.x) +2|Migrate to Cloud NDB| [link](https://twitter.com/googledevs/status/1417892493383802883?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrcloudndb_201021&ut +m_content=-) | [link](http://g.co/codelabs/pae-migrate-cloudndb) | Module 1 [code](/mod1-flask) (2.x) | Module 2 [code](/mod2a-cloudndb) (2.x) & [code](/mod2b-cloudndb) (3.x) 3|Migrate to Cloud Datastore| [link](http://twitter.com/googledevs/status/1422966928910393347?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrcloudds_201003&utm_content=-) | [link](http://g.co/codelabs/pae-migrate-datastore) | Module 2 [code](/mod2a-cloudndb) (2.x) & [code](/mod2b-cloudndb) (3.x) | Module 3 [code](/mod3a-datastore) (2.x) & [code](/mod3b-datastore) (3.x) 4|Migrate to Cloud Run with Docker| [link](https://twitter.com/googledevs/status/1428041270702735362?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrcrdckr_sms_201017&utm_content=-)| [link](http://g.co/codelabs/pae-migrate-rundocker) | Module 2 [code](/mod2a-cloudndb) (2.x) & Module 3 [code](/mod3b-datastore) (3.x) | Module 4 [code](/mod4a-rundocker) (2.x) & [code](/mod4b-rundocker) (3.x) 5|Migrate to Cloud Run with Buildpacks| [link](https://twitter.com/googledevs/status/1433113274984271875?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrcrbdpk_sms_201031&utm_content=-) | [link](http://g.co/codelabs/pae-migrate-runbldpks) | Module 2 [code](/mod2b-cloudndb) (3.x) | Module 5 [code](/mod5-runbldpks) (3.x) From d2d3b313aa45a9d7a918984faa9b18a788600478 Mon Sep 17 00:00:00 2001 From: wesley chun Date: Fri, 10 Sep 2021 01:42:17 -0700 Subject: [PATCH 16/56] fix Mod2 video link again --- README.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/README.md b/README.md index 88ee684..fb43771 100644 --- a/README.md +++ b/README.md @@ -79,8 +79,7 @@ Module | Topic | Video | Codelab | START here | FINISH here --- | --- | --- | --- | --- | --- 0|Baseline app| [link](http://twitter.com/googledevs/status/1407755281786867714?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_smsintro_201023&utm_content=-) | _N/A_ (no tutorial; just review the code) | _N/A_ | Module 0 [code](/mod0-baseline) (2.x) 1|Migrate to Flask| [link](https://twitter.com/googledevs/status/1413273119071031299?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrwa2flsk_201008&u%20tm_content=-) | [link](http://g.co/codelabs/pae-migrate-flask) | Module 0 [code](/mod0-baseline) (2.x) | Module 1 [code](/mod1-flask) (2.x) & [code](/mod1b-flask) (3.x) -2|Migrate to Cloud NDB| [link](https://twitter.com/googledevs/status/1417892493383802883?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrcloudndb_201021&ut -m_content=-) | [link](http://g.co/codelabs/pae-migrate-cloudndb) | Module 1 [code](/mod1-flask) (2.x) | Module 2 [code](/mod2a-cloudndb) (2.x) & [code](/mod2b-cloudndb) (3.x) +2|Migrate to Cloud NDB| [link](https://twitter.com/googledevs/status/1417892493383802883?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrcloudndb_201021&utm_content=-) | [link](http://g.co/codelabs/pae-migrate-cloudndb) | Module 1 [code](/mod1-flask) (2.x) | Module 2 [code](/mod2a-cloudndb) (2.x) & [code](/mod2b-cloudndb) (3.x) 3|Migrate to Cloud Datastore| [link](http://twitter.com/googledevs/status/1422966928910393347?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrcloudds_201003&utm_content=-) | [link](http://g.co/codelabs/pae-migrate-datastore) | Module 2 [code](/mod2a-cloudndb) (2.x) & [code](/mod2b-cloudndb) (3.x) | Module 3 [code](/mod3a-datastore) (2.x) & [code](/mod3b-datastore) (3.x) 4|Migrate to Cloud Run with Docker| [link](https://twitter.com/googledevs/status/1428041270702735362?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrcrdckr_sms_201017&utm_content=-)| [link](http://g.co/codelabs/pae-migrate-rundocker) | Module 2 [code](/mod2a-cloudndb) (2.x) & Module 3 [code](/mod3b-datastore) (3.x) | Module 4 [code](/mod4a-rundocker) (2.x) & [code](/mod4b-rundocker) (3.x) 5|Migrate to Cloud Run with Buildpacks| [link](https://twitter.com/googledevs/status/1433113274984271875?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrcrbdpk_sms_201031&utm_content=-) | [link](http://g.co/codelabs/pae-migrate-runbldpks) | Module 2 [code](/mod2b-cloudndb) (3.x) | Module 5 [code](/mod5-runbldpks) (3.x) From 8c60e99f5972608239be4cf63d172f548a60624b Mon Sep 17 00:00:00 2001 From: wesley chun Date: Thu, 16 Sep 2021 16:51:00 -0700 Subject: [PATCH 17/56] optimize Docker layering; ifurther discourage FS migration --- .README.md.swp | Bin 0 -> 24576 bytes mod4a-rundocker/Dockerfile | 2 +- mod4b-rundocker/Dockerfile | 2 +- 3 files changed, 2 insertions(+), 2 deletions(-) create mode 100644 .README.md.swp diff --git a/.README.md.swp b/.README.md.swp new file mode 100644 index 0000000000000000000000000000000000000000..21d57c65d1dec9b88c55842b93dcab79c3596090 GIT binary patch literal 24576 zcmeHOON=8&8E!}*;SmxNDdHt4W);@9rtP)8o86t2R+^oeeIV`bWM(!Y8AUr~ciC>H zyQ|ZW@j3^%Bp?BTL>%A<96`V#5ho5wB*-B`JT6EPa6@oFfFMFh5s&Y$s(y@TGV9$a zpd@Y0FOS>ZReyc;zyEUY$b%OyvJ=g=h1Xq{^~gs~@$inP-u>6ZmNgb}pDthTMfcf0 z@AstK4kOuaMw0u@zKp`=P|Bev@V}2iK1VmUd@kZ7lC2)^Z>M2P-(mC6jaz+BrcR58 zp$LX?Xcw`R*{{iLk>-GLk_DH)nUOtt>8ITxt5w^?0nluMl`!BakKo`@|hr zx}EkB!r{=|C)quBUwKkm$Sf9uq9Uzz&;UFLJf=Sx%HziP%m zWv>5h>ib*F=X=fj?Ph`5I?d(ktwNwephBQRphBQRphBQRphBQRphBQRphDm+i2z6E z{GUjIpQ6MX{r@cg|H++}^<~hPKrh~5Sw96m`##Hh2IPY71O4{Bmh~IZgYU7dE1>ab*zCGafkR;#wLo9tjr6gbgAgSS+F)w=d!*duZf(f=y(~ zd_GY(jQ9@j6v7b>OC%Ek@8OP~>v?W4WNyHUNe{nKZ)xsPJjkeKw(vzbBo>>0I;)wc8(>khz?j1;onlL z8TLb)KH2odq5MBPjrma6p6k0b7xBPj=1G8$ciJhyFkxdi8DTSU#wQj`-#o9r9jn#2 zK|hV~@Pze8JQ~tgFM6@WC(+-=Q=+`k%`kHN*xduzE>01ddo^d9kEJr<@k|fuTk9;u z@?tiS5%VSF!VU1RuLP!vqZWtQl8h5bz0J;vKt$NXcpGrXBzmjrOcU!URJyq3Ecac{ z<&m3cM3^IXgeOB1)lYFhJcSFPCnxl|csLAW6752e_Zvxum^`5e<4K$dpCuylAy-~2 z9tmtej%oSWoEXn8vUBR_mQlAFqy@{2nfP$qQ5twndXj&xi4oSNM^W+x*aY{xdl!N!vg^umYSjuPK9!WNIx#F68` zOmuuEv753SN`lO!onv*aGkzK;Y}1^a(<|O`dRqj=R)Bx<6{$maGg&A-4<2qgy{n15 zivRQ~h9WUqtKbEY0STEbq`P^X^#oAB1|$%?PzfP+0E(_zB>Udny_enQwf*+;VVNGr zI1TPaf;*E2O+$Mbh--;G6afo8o{;{{GZ+UMz&d(3n5eX~(FkPFXxN?@^8O_A86_6L zL8Ec$^2W+Sqro<&_)`w2kxW830~p-~1&A)v|jsNMlTF^4p|k3|gif2{bRm2+nu|5IYV;lZ4kr@H7^O zVQ~?dC`-z*yoAX?v6D3l5P(;JFbwap@CG*d24^1p!427@*(p2#q8<}A;7~Cp2xZ`q zf!T2#A!Ak>lLqk}?n2!>QCA{3LY!tiC#e&Cgbcs|l4*eCIxe0@@Kop)u`Li>kVYWl zo6!9}L_P}K$CW(0$`xrpuVBHtU}TNopaqj|(At$E!lMNosy|d1#&F$`uFUsPh+_-{ zpj$Ggf@GVvI=3-w082xP23}~V03T*nMg!P^L)fBt0qGS@ia%m&H%4R<;3{RL)R#LKtdHHg2Af3G#VW=sMjJ{BM=`6>W@Y`z9`l)X ztBkPRPqAXsC#|>#s3EH35W8sPl%(x?W{re9OJ^*~|Bv5hS>FRu{-5>7Ir9B2&{fa_pjFUG&`a+}7z=s= z`TsAF`#%DD7_+mp^yv7m05hjq@fD>XqRQUrZ_o>srhQ}qV*m8~B|DUY%m zJerSeGB<=hVD5kcb#0`MBGIlmWz{6@s1%{nWM%xCQHxB6)y{h2noD8+0h1uaF+U+x ztPjj1rbaT&bGKY(C?!cIu%CS%q}isJt4bx^mKkAs!p<8s;hy9eX( zuZgVc1=GGx#jCxgo`D-dAxnywWpk{S@jTByjhY5KG0#@dud-u01wAKG?ryhDCqPx) z(7Z9dFIAw->mWK2SA7&}BJo&f>Zq5Yd{Uk|Q{;T=u9>>s)^i^(&)qR|s~Q^WzcIQW zDBz(ofx?OEo!O}AQxymWEBtQTSD7;9+@L^aMH?tcR{vlVk)Y%w$7pn*eu(_n=u?}W z^&saZYmvsJP$lG5Et_p-m@#cm*rTkX;Ii6+nl%;?f8N>Nk?)7ROS{u%rhnuMyv1P1axlwttVPc=R} z4VgQjY7(lls3bd0mH)pCG5s?ggh&3Lo!fsNegEe`Ht0^!Gas<5dqB5>evCT6_d!=c zXFwkWy@;B?7eQYD-3j^|>I2^aeI0ZO^eXBDe*%3A^Z@8I=oQooUI0A}+5%k#{SLK) z=RqmxqoCVBFQ8`NfgT2Z81x66Blr>MNzfCZzu@e^uRtm2KG40OU*ZP^Pl9$pZP4#; z!9AdFgCyuQ=oOqZcoy^w=xz{U;DJ5{Is~GaiC%VAQExO|VjA_kKhx0Mj1S?KTbbITWi5>B6*lad2BW@!@wGh^y4b0`L9s3ffS z*K<&(-j4T<1k1b+EBOfqvjLYk&abSo%c~m~E?-(_H5epnaZpnQTjrwPjc7?ms65vf z_D*peGmsrAnU7kS#v_!yDJC#etgS3vzIbuv((=l3c?{jES;)7ztF{;;PHmF5i`|~> zWa5bS0UC1J%QC!d-6UFI`B4f4l#@z=RY{&<KPKm$sXj-BAt#p8-3qBPEd$ba@@OGwgQ!4AKfb4q|~;Lg8*f* zS%Wk7aoB@Srl`OX=2`o=sSRPmiyS6D+v4cM5#F2n$a0AZzKmUR5`!OE?7 z9u1RR_g-7M3n6$7<;Gr<;u*EhRXrbik6ITaHX8dV^ek!w8H5lT4W3OYUYFk8gpsk? zjB0pnx9X7TvRPqg7?eh9}h+IB1kVSIf+Keg)!%Lr&Sveb*=)`# z?i%6Q#JzxckGL1ww%gGt_cHAs-ZY~9y8PeFLVk2VA=h%Dh5Klaq#G3*IO*$4_szMk z+^Qesbq?UPP{pHs7q_|J&~{0kpqRxu-hlIYeacdferE6w8iSb?#_ZaSDU9CyebwY9 ztgQzA{}j^AXy{Mw)`2_#Ht~b0>QK~l5|x?m02M{VbY*4OSAeIb?ZobBO2eMeu}(H@ zF17ivC#KoM=Kln9gxgCgdabVXjZP z2@aZ2I<0RmhBU|jD3zfzIPfjVc~y=?1#e_Cbo#L5mv-@&=jLFHP6eC6Q2BuW!EwGk z&H|Ttxq!~W^yrvNoCY|aa-cV%W06R0bg55oB-De_l8rOVyV4$BOi|HRNaFtiDJ{yaic`Tx(6)Bhbr z`9IQN^;$yCudbQT)mw!?g+PTsg+PTsg+PTsg+PTsg+PTsg+PVCn+yS}6VdA})7HgV jjzm>SvgY$VTKs)4Rlku@Kb|T|ECG%S;zv2zoz{N=z-m+@ literal 0 HcmV?d00001 diff --git a/mod4a-rundocker/Dockerfile b/mod4a-rundocker/Dockerfile index caa4522..e7edb64 100644 --- a/mod4a-rundocker/Dockerfile +++ b/mod4a-rundocker/Dockerfile @@ -1,5 +1,5 @@ FROM python:2-slim WORKDIR /app -COPY . . RUN pip install -r requirements.txt +COPY . . ENTRYPOINT exec gunicorn -b :$PORT -w 2 main:app diff --git a/mod4b-rundocker/Dockerfile b/mod4b-rundocker/Dockerfile index 8abd3d0..40b803a 100644 --- a/mod4b-rundocker/Dockerfile +++ b/mod4b-rundocker/Dockerfile @@ -1,5 +1,5 @@ FROM python:3-slim WORKDIR /app -COPY . . RUN pip install -r requirements.txt +COPY . . ENTRYPOINT exec gunicorn -b :$PORT -w 2 main:app From e736cc7744480e408f29ba32c2fc62fa632244bf Mon Sep 17 00:00:00 2001 From: wesley chun Date: Thu, 21 Oct 2021 17:22:00 -0700 Subject: [PATCH 18/56] add latest videos & switch to bundled services public preview --- .README.md.swp | Bin 24576 -> 0 bytes README.md | 15 +++++++-------- mod12b-memcache/README.md | 2 +- mod1b-flask/README.md | 4 ++-- mod7b-gaetasks/README.md | 4 ++-- 5 files changed, 12 insertions(+), 13 deletions(-) delete mode 100644 .README.md.swp diff --git a/.README.md.swp b/.README.md.swp deleted file mode 100644 index 21d57c65d1dec9b88c55842b93dcab79c3596090..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 24576 zcmeHOON=8&8E!}*;SmxNDdHt4W);@9rtP)8o86t2R+^oeeIV`bWM(!Y8AUr~ciC>H zyQ|ZW@j3^%Bp?BTL>%A<96`V#5ho5wB*-B`JT6EPa6@oFfFMFh5s&Y$s(y@TGV9$a zpd@Y0FOS>ZReyc;zyEUY$b%OyvJ=g=h1Xq{^~gs~@$inP-u>6ZmNgb}pDthTMfcf0 z@AstK4kOuaMw0u@zKp`=P|Bev@V}2iK1VmUd@kZ7lC2)^Z>M2P-(mC6jaz+BrcR58 zp$LX?Xcw`R*{{iLk>-GLk_DH)nUOtt>8ITxt5w^?0nluMl`!BakKo`@|hr zx}EkB!r{=|C)quBUwKkm$Sf9uq9Uzz&;UFLJf=Sx%HziP%m zWv>5h>ib*F=X=fj?Ph`5I?d(ktwNwephBQRphBQRphBQRphBQRphBQRphDm+i2z6E z{GUjIpQ6MX{r@cg|H++}^<~hPKrh~5Sw96m`##Hh2IPY71O4{Bmh~IZgYU7dE1>ab*zCGafkR;#wLo9tjr6gbgAgSS+F)w=d!*duZf(f=y(~ zd_GY(jQ9@j6v7b>OC%Ek@8OP~>v?W4WNyHUNe{nKZ)xsPJjkeKw(vzbBo>>0I;)wc8(>khz?j1;onlL z8TLb)KH2odq5MBPjrma6p6k0b7xBPj=1G8$ciJhyFkxdi8DTSU#wQj`-#o9r9jn#2 zK|hV~@Pze8JQ~tgFM6@WC(+-=Q=+`k%`kHN*xduzE>01ddo^d9kEJr<@k|fuTk9;u z@?tiS5%VSF!VU1RuLP!vqZWtQl8h5bz0J;vKt$NXcpGrXBzmjrOcU!URJyq3Ecac{ z<&m3cM3^IXgeOB1)lYFhJcSFPCnxl|csLAW6752e_Zvxum^`5e<4K$dpCuylAy-~2 z9tmtej%oSWoEXn8vUBR_mQlAFqy@{2nfP$qQ5twndXj&xi4oSNM^W+x*aY{xdl!N!vg^umYSjuPK9!WNIx#F68` zOmuuEv753SN`lO!onv*aGkzK;Y}1^a(<|O`dRqj=R)Bx<6{$maGg&A-4<2qgy{n15 zivRQ~h9WUqtKbEY0STEbq`P^X^#oAB1|$%?PzfP+0E(_zB>Udny_enQwf*+;VVNGr zI1TPaf;*E2O+$Mbh--;G6afo8o{;{{GZ+UMz&d(3n5eX~(FkPFXxN?@^8O_A86_6L zL8Ec$^2W+Sqro<&_)`w2kxW830~p-~1&A)v|jsNMlTF^4p|k3|gif2{bRm2+nu|5IYV;lZ4kr@H7^O zVQ~?dC`-z*yoAX?v6D3l5P(;JFbwap@CG*d24^1p!427@*(p2#q8<}A;7~Cp2xZ`q zf!T2#A!Ak>lLqk}?n2!>QCA{3LY!tiC#e&Cgbcs|l4*eCIxe0@@Kop)u`Li>kVYWl zo6!9}L_P}K$CW(0$`xrpuVBHtU}TNopaqj|(At$E!lMNosy|d1#&F$`uFUsPh+_-{ zpj$Ggf@GVvI=3-w082xP23}~V03T*nMg!P^L)fBt0qGS@ia%m&H%4R<;3{RL)R#LKtdHHg2Af3G#VW=sMjJ{BM=`6>W@Y`z9`l)X ztBkPRPqAXsC#|>#s3EH35W8sPl%(x?W{re9OJ^*~|Bv5hS>FRu{-5>7Ir9B2&{fa_pjFUG&`a+}7z=s= z`TsAF`#%DD7_+mp^yv7m05hjq@fD>XqRQUrZ_o>srhQ}qV*m8~B|DUY%m zJerSeGB<=hVD5kcb#0`MBGIlmWz{6@s1%{nWM%xCQHxB6)y{h2noD8+0h1uaF+U+x ztPjj1rbaT&bGKY(C?!cIu%CS%q}isJt4bx^mKkAs!p<8s;hy9eX( zuZgVc1=GGx#jCxgo`D-dAxnywWpk{S@jTByjhY5KG0#@dud-u01wAKG?ryhDCqPx) z(7Z9dFIAw->mWK2SA7&}BJo&f>Zq5Yd{Uk|Q{;T=u9>>s)^i^(&)qR|s~Q^WzcIQW zDBz(ofx?OEo!O}AQxymWEBtQTSD7;9+@L^aMH?tcR{vlVk)Y%w$7pn*eu(_n=u?}W z^&saZYmvsJP$lG5Et_p-m@#cm*rTkX;Ii6+nl%;?f8N>Nk?)7ROS{u%rhnuMyv1P1axlwttVPc=R} z4VgQjY7(lls3bd0mH)pCG5s?ggh&3Lo!fsNegEe`Ht0^!Gas<5dqB5>evCT6_d!=c zXFwkWy@;B?7eQYD-3j^|>I2^aeI0ZO^eXBDe*%3A^Z@8I=oQooUI0A}+5%k#{SLK) z=RqmxqoCVBFQ8`NfgT2Z81x66Blr>MNzfCZzu@e^uRtm2KG40OU*ZP^Pl9$pZP4#; z!9AdFgCyuQ=oOqZcoy^w=xz{U;DJ5{Is~GaiC%VAQExO|VjA_kKhx0Mj1S?KTbbITWi5>B6*lad2BW@!@wGh^y4b0`L9s3ffS z*K<&(-j4T<1k1b+EBOfqvjLYk&abSo%c~m~E?-(_H5epnaZpnQTjrwPjc7?ms65vf z_D*peGmsrAnU7kS#v_!yDJC#etgS3vzIbuv((=l3c?{jES;)7ztF{;;PHmF5i`|~> zWa5bS0UC1J%QC!d-6UFI`B4f4l#@z=RY{&<KPKm$sXj-BAt#p8-3qBPEd$ba@@OGwgQ!4AKfb4q|~;Lg8*f* zS%Wk7aoB@Srl`OX=2`o=sSRPmiyS6D+v4cM5#F2n$a0AZzKmUR5`!OE?7 z9u1RR_g-7M3n6$7<;Gr<;u*EhRXrbik6ITaHX8dV^ek!w8H5lT4W3OYUYFk8gpsk? zjB0pnx9X7TvRPqg7?eh9}h+IB1kVSIf+Keg)!%Lr&Sveb*=)`# z?i%6Q#JzxckGL1ww%gGt_cHAs-ZY~9y8PeFLVk2VA=h%Dh5Klaq#G3*IO*$4_szMk z+^Qesbq?UPP{pHs7q_|J&~{0kpqRxu-hlIYeacdferE6w8iSb?#_ZaSDU9CyebwY9 ztgQzA{}j^AXy{Mw)`2_#Ht~b0>QK~l5|x?m02M{VbY*4OSAeIb?ZobBO2eMeu}(H@ zF17ivC#KoM=Kln9gxgCgdabVXjZP z2@aZ2I<0RmhBU|jD3zfzIPfjVc~y=?1#e_Cbo#L5mv-@&=jLFHP6eC6Q2BuW!EwGk z&H|Ttxq!~W^yrvNoCY|aa-cV%W06R0bg55oB-De_l8rOVyV4$BOi|HRNaFtiDJ{yaic`Tx(6)Bhbr z`9IQN^;$yCudbQT)mw!?g+PTsg+PTsg+PTsg+PTsg+PTsg+PVCn+yS}6VdA})7HgV jjzm>SvgY$VTKs)4Rlku@Kb|T|ECG%S;zv2zoz{N=z-m+@ diff --git a/README.md b/README.md index fb43771..0aa7467 100644 --- a/README.md +++ b/README.md @@ -29,7 +29,7 @@ Each codelab begins with a "START" code base then walks developers through that App Engine, Cloud Functions, and Cloud Run are not free services. While you may not have needed to enable billing in App Engine's early days, [all applications now require an active billing account](https://cloud.google.com/appengine/docs/standard/payment-instrument) backed by a financial instrument (usually a credit card). Don't worry, App Engine (and other GCP products) still have an ["Always Free" tier](https://cloud.google.com/free/docs/gcp-free-tier#free-tier-usage-limits) and as long as you stay within those limits, you won't incur any charges. Also check the App Engine [pricing](https://cloud.google.com/appengine/pricing) and [quotas](https://cloud.google.com/appengine/quotas) pages for more information. -Furthermore, deploying to GCP serverless platforms incur [minor build and storage costs](https://cloud.google.com/appengine/pricing#pricing-for-related-google-cloud-products). [Cloud Build](https://cloud.google.com/build/pricing) has its own free quota as does [Cloud Storage](https://cloud.google.com/storage/pricing#cloud-storage-always-free). For greater transparency, Cloud Build builds your application image which is than sent to the [Cloud Container Registry](https://cloud.google.com/container-registry/pricing) or [Artifact Registry](https://cloud.google.com/artifact-registry/pricing), its successor; storage of that image uses up some of that (Cloud Storage) quota as does network egress when transferring that image to the service you're deploying to. However you may live in region that does not have such a free tier, so be aware of your storage usage to minimize potential costs. (You may look at what storage you're using and how much, including deleting build artifacts via [your Cloud Storage browser](https://console.cloud.google.com/storage/browser).) +Furthermore, deploying to GCP serverless platforms incur [minor build and storage costs](https://cloud.google.com/appengine/pricing#pricing-for-related-google-cloud-products). [Cloud Build](https://cloud.google.com/build/pricing) has its own free quota as does [Cloud Storage](https://cloud.google.com/storage/pricing#cloud-storage-always-free). For greater transparency, Cloud Build builds your application image which is than sent to the [Cloud Container Registry](https://cloud.google.com/container-registry/pricing), or [Artifact Registry](https://cloud.google.com/artifact-registry/pricing), its successor; storage of that image uses up some of that (Cloud Storage) quota as does network egress when transferring that image to the service you're deploying to. However you may live in region that does not have such a free tier, so be aware of your storage usage to minimize potential costs. (You may look at what storage you're using and how much, including deleting build artifacts via [your Cloud Storage browser](https://console.cloud.google.com/storage/browser).) ## Why @@ -83,9 +83,9 @@ Module | Topic | Video | Codelab | START here | FINISH here 3|Migrate to Cloud Datastore| [link](http://twitter.com/googledevs/status/1422966928910393347?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrcloudds_201003&utm_content=-) | [link](http://g.co/codelabs/pae-migrate-datastore) | Module 2 [code](/mod2a-cloudndb) (2.x) & [code](/mod2b-cloudndb) (3.x) | Module 3 [code](/mod3a-datastore) (2.x) & [code](/mod3b-datastore) (3.x) 4|Migrate to Cloud Run with Docker| [link](https://twitter.com/googledevs/status/1428041270702735362?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrcrdckr_sms_201017&utm_content=-)| [link](http://g.co/codelabs/pae-migrate-rundocker) | Module 2 [code](/mod2a-cloudndb) (2.x) & Module 3 [code](/mod3b-datastore) (3.x) | Module 4 [code](/mod4a-rundocker) (2.x) & [code](/mod4b-rundocker) (3.x) 5|Migrate to Cloud Run with Buildpacks| [link](https://twitter.com/googledevs/status/1433113274984271875?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrcrbdpk_sms_201031&utm_content=-) | [link](http://g.co/codelabs/pae-migrate-runbldpks) | Module 2 [code](/mod2b-cloudndb) (3.x) | Module 5 [code](/mod5-runbldpks) (3.x) -6|Migrate to Cloud Firestore (app)| _TBD_ | [link](http://g.co/codelabs/pae-migrate-firestore) | Module 3 [code](/mod3b-datastore) (3.x) | Module 6 [code](/mod6-firestore) (3.x) -7|Add App Engine `taskqueue` push tasks| _TBD_ | [link](http://g.co/codelabs/pae-migrate-gaetasks) | Module 1 [code](/mod1-flask) (2.x) | Module 7 [code](/mod7-gaetasks) (2.x) & [code](/mod7b-gaetasks) (3.x) -8|Migrate to Cloud Tasks| _TBD_ | [link](http://g.co/codelabs/pae-migrate-cloudtasks) | Module 7 [code](/mod7-gaetasks) (2.x) | Module 8 [code](/mod8-cloudtasks) (2.x) +6|Migrate to Cloud Firestore (app)| [link](http://youtu.be/wNs36kukVOQ?list=PL2pQQBHvYcs0PEecTcLD9_VaLvuhK0_VQ?utm_source=youtube&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrcloudfsam_sms_201014&utm_content=info_card) | [link](http://g.co/codelabs/pae-migrate-firestore) | Module 3 [code](/mod3b-datastore) (3.x) | Module 6 [code](/mod6-firestore) (3.x) +7|Add App Engine `taskqueue` push tasks| [link](https://twitter.com/googledevs/status/1443410302113099778?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrgaetasks_sms_201028&utm_content=-) | [link](http://g.co/codelabs/pae-migrate-gaetasks) | Module 1 [code](/mod1-flask) (2.x) | Module 7 [code](/mod7-gaetasks) (2.x) & [code](/mod7b-gaetasks) (3.x) +8|Migrate to Cloud Tasks| [link](https://twitter.com/googledevs/status/1450960021018267656?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrcloudtasks_sms_201112&utm_content=-) | [link](http://g.co/codelabs/pae-migrate-cloudtasks) | Module 7 [code](/mod7-gaetasks) (2.x) | Module 8 [code](/mod8-cloudtasks) (2.x) 9|Migrate to Python 3, Cloud Firestore & Cloud Tasks v2| _TBD_ | _TBD_ | Module 8 [code](/mod8-cloudtasks) (2.x) | Module 9 [code](/mod9-py3fstasks) (3.x) 10|Migrate to Cloud Firestore (data)| _TBD_ | _N/A_ | _N/A_ | _TBD_ 11|Migrate to Cloud Functions| _TBD_ | _TBD_ | Module 2 [code](/mod2b-cloudndb) (3.x) | Module 11 [code](/mod11-functions) (3.x) @@ -196,12 +196,11 @@ If there is a logical codelab to do immediately after completing one, they will - Module 11 - migrate to Cloud Functions - Module 5 - migrate to Cloud Run container with Cloud Buildpacks - Module 4 - migrate to Cloud Run container with Docker - - Module 6 - migrate to Cloud Firestore (app) - [Module 6 codelab](http://g.co/codelabs/pae-migrate-firestore): **Migrate from Cloud Datastore to [Cloud Firestore](http://cloud.google.com/firestore)** (app) - **Highly optional** migration - - Requires new project & Datastore has better write performance (currently) - - If you **must have** Firestore's Firebase features + - Requires new project & Datastore has better write performance (currently) and is a fully-supported product and will get new features, including some Firestore native features + - If you **must have** Firestore's Firebase features now and can't wait - Python 3 only - START: [Module 3 code - Cloud Datastore](/mod3b-datastore) (3.x) - FINISH: [Module 6 code - Cloud Firestore](/mod6-firestore) (3.x) @@ -261,7 +260,7 @@ If your original app users does *not* have a user interface, i.e., mobile backen ## Accessing legacy services in second generation -Some legacy App Engine first generation platform (Python 2, Java 8, PHP 5, and Go 1.11 & older) services are available on a limited basis to second generation runtimes (Python 3, Java 11, PHP 7, and Go 1.12 & newer) as part of an early-access program. There are no videos or codelabs yet, however the Module 1 Flask migration using App Engine `ndb` [Python 2 sample ](/mod1-flask) is available in [Python 3](/mod1b-flask) if you have access. Similarly, Python 3 editions are also available for Modules 7 and 12 which add usage of App Engine `taskqueue` and `memcache`, respectively. See their respective `README` files for more info and to sign-up if interested. +Many legacy App Engine first generation platform (Python 2, Java 8, PHP 5, and Go 1.11 & older) services are available ([as of Sep 2021](https://twitter.com/googledevs/status/1445916786755571712) for second generation runtimes (Python 3, Java 11, PHP 7, and Go 1.12 & newer) in a public preview. There are no videos or codelabs yet, however the Module 1 Flask migration using App Engine `ndb` [Python 2 sample ](/mod1-flask) is available in [Python 3](/mod1b-flask) if you have access. Similarly, Python 3 editions are also available for Modules 7 and 12 which add usage of App Engine `taskqueue` and `memcache`, respectively. ## References diff --git a/mod12b-memcache/README.md b/mod12b-memcache/README.md index 3def656..c66a990 100644 --- a/mod12b-memcache/README.md +++ b/mod12b-memcache/README.md @@ -2,4 +2,4 @@ This repo folder is the corresponding Python 3 code to the Module 12 codelab (TBD). The tutorial STARTs with the Python 2 code in the [Module 1 repo folder](/mod1-flask) and leads developers through adding usage of App Engine's `memcache`, followed by a bonus migration to Python 3, culminating in the code in this folder. -> **LEGACY SERVICES PRIVATE PREVIEW**: Accessing legacy services such as App Engine `ndb` and `memcache` from Python 3 (and next generation App Engine in general) is currently in private preview and requires allowlisted access. Please complete [this form](https://docs.google.com/forms/d/e/1FAIpQLSd1hFLA2UFSYwIMxm9ZI3pwigORZBgjJRH0qrnhtE7nvhhRCQ/viewform) to sign-up for the program. Registering will get you in the queue to access the private preview, its documentation, and the announcements mailing list. For more info on what services are (or are not) available (at this time), see [this thread in the r/AppEngine Subreddit](https://reddit.com/r/AppEngine/comments/o9wr72). +> **LEGACY SERVICES PUBLIC PREVIEW**: Accessing legacy services such as App Engine `ndb` and `memcache` from Python 3 (and next generation App Engine in general) is available in a public preview. See the [Sep 2021 announcement](https://twitter.com/googledevs/status/1445916786755571712) for more information. diff --git a/mod1b-flask/README.md b/mod1b-flask/README.md index 111eded..b721e29 100644 --- a/mod1b-flask/README.md +++ b/mod1b-flask/README.md @@ -1,5 +1,5 @@ # Module 1 - Migrate from `webapp2` to Flask -This repo folder is the corresponding Python 3 code to the [Module 1 codelab](http://g.co/codelabs/pae-migrate-flask). The tutorial STARTs with the Python 2 code in the [Module 0 repo folder](/mod0-baseline) and leads developers through migrating away from App Engine's `webapp2` web framework to Flask, culminating in the code in the [mod1-flask](/mod1-flask) folder. The codelab does **not** currently feature any bonus migration to Python 3, however, if you do so, and add in components from the legacy services private preview program (see sidebar below), it will culminate in the code in *this* (`mod1b-flask`) folder. In the [next (Module 2) codelab](http://g.co/codelabs/pae-migrate-cloudndb), users will migrate (the original Python 2 version of) this app from App Engine `ndb` to Cloud NDB for Datastore access. +This repo folder is the corresponding Python 3 code to the [Module 1 codelab](http://g.co/codelabs/pae-migrate-flask). The tutorial STARTs with the Python 2 code in the [Module 0 repo folder](/mod0-baseline) and leads developers through migrating away from App Engine's `webapp2` web framework to Flask, culminating in the code in the [mod1-flask](/mod1-flask) folder. The codelab does **not** currently feature any bonus migration to Python 3, however doing so requires you to participate in the bundled services public preview program (see sidebar below) and which culminates in the code in *this* (`mod1b-flask`) folder. In the [next (Module 2) codelab](http://g.co/codelabs/pae-migrate-cloudndb), users will migrate (the original Python 2 version of) this app from App Engine `ndb` to Cloud NDB for Datastore access. -> **LEGACY SERVICES PRIVATE PREVIEW**: Accessing legacy services such as App Engine `ndb` from Python 3 (and next generation App Engine in general) is currently in private preview and requires allowlisted access. Please complete [this form](https://docs.google.com/forms/d/e/1FAIpQLSd1hFLA2UFSYwIMxm9ZI3pwigORZBgjJRH0qrnhtE7nvhhRCQ/viewform) to sign-up for the program. Registering will get you in the queue to access the private preview, its documentation, and the announcements mailing list. For more info on what services are (or are not) available (at this time), see [this thread in the r/AppEngine Subreddit](https://reddit.com/r/AppEngine/comments/o9wr72). +> **LEGACY SERVICES PUBLIC PREVIEW**: Accessing legacy services such as App Engine `ndb` from Python 3 (and next generation App Engine in general) is available in a public preview. See the [Sep 2021 announcement](https://twitter.com/googledevs/status/1445916786755571712) for more information. diff --git a/mod7b-gaetasks/README.md b/mod7b-gaetasks/README.md index f065500..fa767aa 100644 --- a/mod7b-gaetasks/README.md +++ b/mod7b-gaetasks/README.md @@ -1,5 +1,5 @@ # Module 7 - Add usage of App Engine `taskqueue` to Flask `ndb` sample app -This repo folder is the corresponding Python 3 code to the [Module 7 codelab](http://g.co/codelabs/pae-migrate-gaetasks). The tutorial STARTs with the Python 2 code in the [Module 1 repo folder](/mod1-flask) and leads developers through adding usage of App Engine's `taskqueue`, culminating in the code in the [mod7-gaetasks](/mod7-gaetasks) folder. The codelab does **not** currently feature any bonus migration to Python 3, however, if you do so, and add in components from the legacy services private preview program (see sidebar below), it will culminate in the code in *this* (`mod7b-gaetasks`) folder. In the [next (Module 8) codelab](http://g.co/codelabs/pae-migrate-cloudtasks), users will migrate (the original Python 2 version of) this app from App Engine `taskqueue` to Cloud Tasks. +This repo folder is the corresponding Python 3 code to the [Module 7 codelab](http://g.co/codelabs/pae-migrate-gaetasks). The tutorial STARTs with the Python 2 code in the [Module 1 repo folder](/mod1-flask) and leads developers through adding usage of App Engine's `taskqueue`, culminating in the code in the [mod7-gaetasks](/mod7-gaetasks) folder. The codelab does **not** currently feature any bonus migration to Python 3, however to do so requires you to participate in the bundled services public preview program (see sidebar below) and which culminates in the code in *this* (`mod7b-gaetasks`) folder. In the [next (Module 8) codelab](http://g.co/codelabs/pae-migrate-cloudtasks), users will migrate (the original Python 2 version of) this app from App Engine `taskqueue` to Cloud Tasks. -> **LEGACY SERVICES PRIVATE PREVIEW**: Accessing legacy services such as App Engine `ndb` and `taskqueue` from Python 3 (and next generation App Engine in general) is currently in private preview and requires allowlisted access. Please complete [this form](https://docs.google.com/forms/d/e/1FAIpQLSd1hFLA2UFSYwIMxm9ZI3pwigORZBgjJRH0qrnhtE7nvhhRCQ/viewform) to sign-up for the program. Registering will get you in the queue to access the private preview, its documentation, and the announcements mailing list. For more info on what services are (or are not) available (at this time), see [this thread in the r/AppEngine Subreddit](https://reddit.com/r/AppEngine/comments/o9wr72). +> **LEGACY SERVICES PUBLIC PREVIEW**: Accessing legacy services such as App Engine `ndb` and `taskqueue` from Python 3 (and next generation App Engine in general) is available in a public preview. See the [Sep 2021 announcement](https://twitter.com/googledevs/status/1445916786755571712) for more information. From def314285f0b3f9e6c49bc4f9067ed82436d806e Mon Sep 17 00:00:00 2001 From: wesley chun Date: Mon, 29 Nov 2021 18:52:37 -0800 Subject: [PATCH 19/56] minor fixes/updates, upgrade to Python 3.9 --- mod1-flask/main.py | 2 +- mod12-memcache/main.py | 11 +++++------ mod12b-memcache/app.yaml | 2 +- mod1b-flask/app.yaml | 2 +- mod2b-cloudndb/app.yaml | 2 +- mod3b-datastore/app.yaml | 2 +- mod6-firestore/app.yaml | 2 +- mod7b-gaetasks/app.yaml | 2 +- mod9-py3fstasks/app.yaml | 2 +- mod9-py3fstasks/requirements.txt | 4 ++-- 10 files changed, 15 insertions(+), 16 deletions(-) diff --git a/mod1-flask/main.py b/mod1-flask/main.py index ecf425f..aac3082 100644 --- a/mod1-flask/main.py +++ b/mod1-flask/main.py @@ -1,4 +1,4 @@ -# Copyright 2021 Google LLC +# Copyright 2020 Google LLC # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/mod12-memcache/main.py b/mod12-memcache/main.py index f0d887d..ff2db41 100644 --- a/mod12-memcache/main.py +++ b/mod12-memcache/main.py @@ -30,8 +30,8 @@ def store_visit(remote_addr, user_agent): def fetch_visits(limit): 'get most recent visits' - return [v.to_dict() for v in Visit.query().order( - -Visit.timestamp).fetch(limit)] + return (v.to_dict() for v in Visit.query().order( + -Visit.timestamp).fetch(limit)) @app.route('/') def root(): @@ -41,11 +41,10 @@ def root(): visitor = '{}: {}'.format(ip_addr, usr_agt) visits = memcache.get('visits') - # register visit & run DB query if cache empty or new visitor + # if cache empty or new visitor, register visit & run DB query if not visits or visits[0]['visitor'] != visitor: store_visit(ip_addr, usr_agt) - visits = fetch_visits(10) - memcache.set('visits', visits, HOUR) # must set() not add() here + visits = list(fetch_visits(10)) + memcache.set('visits', visits, HOUR) # set() not add() - # send context to template renderer return render_template('index.html', visits=visits) diff --git a/mod12b-memcache/app.yaml b/mod12b-memcache/app.yaml index 288ce87..4c77d5d 100644 --- a/mod12b-memcache/app.yaml +++ b/mod12b-memcache/app.yaml @@ -12,5 +12,5 @@ # See the License for the specific language governing permissions and # limitations under the License. -runtime: python38 +runtime: python39 app_engine_apis: true diff --git a/mod1b-flask/app.yaml b/mod1b-flask/app.yaml index 288ce87..4c77d5d 100644 --- a/mod1b-flask/app.yaml +++ b/mod1b-flask/app.yaml @@ -12,5 +12,5 @@ # See the License for the specific language governing permissions and # limitations under the License. -runtime: python38 +runtime: python39 app_engine_apis: true diff --git a/mod2b-cloudndb/app.yaml b/mod2b-cloudndb/app.yaml index a4ce819..a926609 100644 --- a/mod2b-cloudndb/app.yaml +++ b/mod2b-cloudndb/app.yaml @@ -12,4 +12,4 @@ # See the License for the specific language governing permissions and # limitations under the License. -runtime: python38 +runtime: python39 diff --git a/mod3b-datastore/app.yaml b/mod3b-datastore/app.yaml index a4ce819..a926609 100644 --- a/mod3b-datastore/app.yaml +++ b/mod3b-datastore/app.yaml @@ -12,4 +12,4 @@ # See the License for the specific language governing permissions and # limitations under the License. -runtime: python38 +runtime: python39 diff --git a/mod6-firestore/app.yaml b/mod6-firestore/app.yaml index 48dfc91..a3d0c92 100644 --- a/mod6-firestore/app.yaml +++ b/mod6-firestore/app.yaml @@ -12,7 +12,7 @@ # See the License for the specific language governing permissions and # limitations under the License. -runtime: python38 +runtime: python39 handlers: - url: /.* diff --git a/mod7b-gaetasks/app.yaml b/mod7b-gaetasks/app.yaml index 288ce87..4c77d5d 100644 --- a/mod7b-gaetasks/app.yaml +++ b/mod7b-gaetasks/app.yaml @@ -12,5 +12,5 @@ # See the License for the specific language governing permissions and # limitations under the License. -runtime: python38 +runtime: python39 app_engine_apis: true diff --git a/mod9-py3fstasks/app.yaml b/mod9-py3fstasks/app.yaml index a4ce819..a926609 100644 --- a/mod9-py3fstasks/app.yaml +++ b/mod9-py3fstasks/app.yaml @@ -12,4 +12,4 @@ # See the License for the specific language governing permissions and # limitations under the License. -runtime: python38 +runtime: python39 diff --git a/mod9-py3fstasks/requirements.txt b/mod9-py3fstasks/requirements.txt index ca6d2f7..98ea30b 100644 --- a/mod9-py3fstasks/requirements.txt +++ b/mod9-py3fstasks/requirements.txt @@ -1,3 +1,3 @@ flask==1.1.2 -google-cloud-firestore==2.2.0 -google-cloud-tasks==2.5.1 +google-cloud-firestore==2.3.4 +google-cloud-tasks==2.7.0 From 133c4077d1813d11bda12bbc76e75f34e8baeb02 Mon Sep 17 00:00:00 2001 From: wesley chun Date: Thu, 16 Dec 2021 13:46:05 -0800 Subject: [PATCH 20/56] add Memorystore/Mod13, update .gcloudignore & pkg versions --- mod0-baseline/.gcloudignore | 70 +++++++++++++++++++--- mod1-flask/.gcloudignore | 70 +++++++++++++++++++--- mod1-flask/appengine_config.py | 2 +- mod1-flask/requirements.txt | 2 +- mod11-functions/.gcloudignore | 70 +++++++++++++++++++--- mod11-functions/requirements.txt | 4 +- mod12-memcache/.gcloudignore | 70 +++++++++++++++++++--- mod12-memcache/main.py | 4 +- mod12-memcache/requirements.txt | 2 +- mod12b-memcache/.gcloudignore | 70 +++++++++++++++++++--- mod12b-memcache/main.py | 11 ++-- mod12b-memcache/requirements.txt | 2 +- mod13a-memorystore/.gcloudignore | 80 +++++++++++++++++++++++++ mod13a-memorystore/README.md | 3 + mod13a-memorystore/app.yaml | 28 +++++++++ mod13a-memorystore/appengine_config.py | 20 +++++++ mod13a-memorystore/main.py | 56 +++++++++++++++++ mod13a-memorystore/requirements.txt | 2 + mod13a-memorystore/templates/index.html | 16 +++++ mod13b-memorystore/.gcloudignore | 80 +++++++++++++++++++++++++ mod13b-memorystore/README.md | 5 ++ mod13b-memorystore/app.yaml | 22 +++++++ mod13b-memorystore/main.py | 59 ++++++++++++++++++ mod13b-memorystore/requirements.txt | 3 + mod13b-memorystore/templates/index.html | 16 +++++ mod1b-flask/.gcloudignore | 70 +++++++++++++++++++--- mod1b-flask/requirements.txt | 2 +- mod2a-cloudndb/.gcloudignore | 70 +++++++++++++++++++--- mod2a-cloudndb/requirements.txt | 4 +- mod2b-cloudndb/.gcloudignore | 70 +++++++++++++++++++--- mod2b-cloudndb/requirements.txt | 4 +- mod3a-datastore/.gcloudignore | 70 +++++++++++++++++++--- mod3a-datastore/requirements.txt | 2 +- mod3b-datastore/.gcloudignore | 70 +++++++++++++++++++--- mod3b-datastore/requirements.txt | 2 +- mod4a-rundocker/.gcloudignore | 70 +++++++++++++++++++--- mod4a-rundocker/requirements.txt | 4 +- mod4b-rundocker/.gcloudignore | 70 +++++++++++++++++++--- mod4b-rundocker/requirements.txt | 6 +- mod5-runbldpks/.gcloudignore | 70 +++++++++++++++++++--- mod5-runbldpks/requirements.txt | 6 +- mod6-firestore/.gcloudignore | 70 +++++++++++++++++++--- mod6-firestore/requirements.txt | 4 +- mod7-gaetasks/.gcloudignore | 70 +++++++++++++++++++--- mod7-gaetasks/requirements.txt | 2 +- mod7b-gaetasks/.gcloudignore | 70 +++++++++++++++++++--- mod7b-gaetasks/requirements.txt | 2 +- mod8-cloudtasks/.gcloudignore | 70 +++++++++++++++++++--- mod8-cloudtasks/requirements.txt | 6 +- mod9-py3fstasks/.gcloudignore | 70 +++++++++++++++++++--- mod9-py3fstasks/requirements.txt | 4 +- 51 files changed, 1561 insertions(+), 164 deletions(-) create mode 100644 mod13a-memorystore/.gcloudignore create mode 100644 mod13a-memorystore/README.md create mode 100644 mod13a-memorystore/app.yaml create mode 100644 mod13a-memorystore/appengine_config.py create mode 100644 mod13a-memorystore/main.py create mode 100644 mod13a-memorystore/requirements.txt create mode 100644 mod13a-memorystore/templates/index.html create mode 100644 mod13b-memorystore/.gcloudignore create mode 100644 mod13b-memorystore/README.md create mode 100644 mod13b-memorystore/app.yaml create mode 100644 mod13b-memorystore/main.py create mode 100644 mod13b-memorystore/requirements.txt create mode 100644 mod13b-memorystore/templates/index.html diff --git a/mod0-baseline/.gcloudignore b/mod0-baseline/.gcloudignore index bcf97a8..b4dbc7e 100644 --- a/mod0-baseline/.gcloudignore +++ b/mod0-baseline/.gcloudignore @@ -8,17 +8,73 @@ # .gcloudignore -# Ignore source code control maintenance files -.git +# Source code control files +.git/ .gitignore .hgignore .hg/ -# Python files -*.pyc -*.pyo +# README/text files +LICENSE +*.md + +# Tests/results (not in .gitignore) +noxfile.py +test_translate.py +pylintrc +pylintrc.test + +# most of .gitignore (except `lib`) +# +# Python +*.py[cod] __pycache__/ /setup.cfg -# no need to upload README -README.md +# C extensions +*.so + +# Packages +*.egg +*.egg-info +dist +build +eggs +.eggs +parts +bin +var +sdist +develop-eggs +.installed.cfg +lib64 +*.tgz + +# Installer logs +pip-log.txt + +# Tests/results +.nox/ +.pytest_cache/ +.cache +.pytype +.coverage +coverage.xml +*sponge_log.xml +system_tests/local_test_setup + +# Mac +.DS_Store + +# IDEs/editors +*.sw[op] +*~ +.vscode +.idea + +# Built documentation +docs/_build +docs.metadata + +# Virtual environment +env/ diff --git a/mod1-flask/.gcloudignore b/mod1-flask/.gcloudignore index bcf97a8..b4dbc7e 100644 --- a/mod1-flask/.gcloudignore +++ b/mod1-flask/.gcloudignore @@ -8,17 +8,73 @@ # .gcloudignore -# Ignore source code control maintenance files -.git +# Source code control files +.git/ .gitignore .hgignore .hg/ -# Python files -*.pyc -*.pyo +# README/text files +LICENSE +*.md + +# Tests/results (not in .gitignore) +noxfile.py +test_translate.py +pylintrc +pylintrc.test + +# most of .gitignore (except `lib`) +# +# Python +*.py[cod] __pycache__/ /setup.cfg -# no need to upload README -README.md +# C extensions +*.so + +# Packages +*.egg +*.egg-info +dist +build +eggs +.eggs +parts +bin +var +sdist +develop-eggs +.installed.cfg +lib64 +*.tgz + +# Installer logs +pip-log.txt + +# Tests/results +.nox/ +.pytest_cache/ +.cache +.pytype +.coverage +coverage.xml +*sponge_log.xml +system_tests/local_test_setup + +# Mac +.DS_Store + +# IDEs/editors +*.sw[op] +*~ +.vscode +.idea + +# Built documentation +docs/_build +docs.metadata + +# Virtual environment +env/ diff --git a/mod1-flask/appengine_config.py b/mod1-flask/appengine_config.py index 9760ebb..0ca8634 100644 --- a/mod1-flask/appengine_config.py +++ b/mod1-flask/appengine_config.py @@ -1,4 +1,4 @@ -# Copyright 2021 Google LLC +# Copyright 2020 Google LLC # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/mod1-flask/requirements.txt b/mod1-flask/requirements.txt index 5c508e5..7e10602 100644 --- a/mod1-flask/requirements.txt +++ b/mod1-flask/requirements.txt @@ -1 +1 @@ -flask==1.1.2 +flask diff --git a/mod11-functions/.gcloudignore b/mod11-functions/.gcloudignore index bcf97a8..b4dbc7e 100644 --- a/mod11-functions/.gcloudignore +++ b/mod11-functions/.gcloudignore @@ -8,17 +8,73 @@ # .gcloudignore -# Ignore source code control maintenance files -.git +# Source code control files +.git/ .gitignore .hgignore .hg/ -# Python files -*.pyc -*.pyo +# README/text files +LICENSE +*.md + +# Tests/results (not in .gitignore) +noxfile.py +test_translate.py +pylintrc +pylintrc.test + +# most of .gitignore (except `lib`) +# +# Python +*.py[cod] __pycache__/ /setup.cfg -# no need to upload README -README.md +# C extensions +*.so + +# Packages +*.egg +*.egg-info +dist +build +eggs +.eggs +parts +bin +var +sdist +develop-eggs +.installed.cfg +lib64 +*.tgz + +# Installer logs +pip-log.txt + +# Tests/results +.nox/ +.pytest_cache/ +.cache +.pytype +.coverage +coverage.xml +*sponge_log.xml +system_tests/local_test_setup + +# Mac +.DS_Store + +# IDEs/editors +*.sw[op] +*~ +.vscode +.idea + +# Built documentation +docs/_build +docs.metadata + +# Virtual environment +env/ diff --git a/mod11-functions/requirements.txt b/mod11-functions/requirements.txt index c5fbb74..81ef820 100644 --- a/mod11-functions/requirements.txt +++ b/mod11-functions/requirements.txt @@ -1,2 +1,2 @@ -flask==1.1.2 -google-cloud-ndb==1.9.0 +flask +google-cloud-ndb==1.11.1 diff --git a/mod12-memcache/.gcloudignore b/mod12-memcache/.gcloudignore index bcf97a8..b4dbc7e 100644 --- a/mod12-memcache/.gcloudignore +++ b/mod12-memcache/.gcloudignore @@ -8,17 +8,73 @@ # .gcloudignore -# Ignore source code control maintenance files -.git +# Source code control files +.git/ .gitignore .hgignore .hg/ -# Python files -*.pyc -*.pyo +# README/text files +LICENSE +*.md + +# Tests/results (not in .gitignore) +noxfile.py +test_translate.py +pylintrc +pylintrc.test + +# most of .gitignore (except `lib`) +# +# Python +*.py[cod] __pycache__/ /setup.cfg -# no need to upload README -README.md +# C extensions +*.so + +# Packages +*.egg +*.egg-info +dist +build +eggs +.eggs +parts +bin +var +sdist +develop-eggs +.installed.cfg +lib64 +*.tgz + +# Installer logs +pip-log.txt + +# Tests/results +.nox/ +.pytest_cache/ +.cache +.pytype +.coverage +coverage.xml +*sponge_log.xml +system_tests/local_test_setup + +# Mac +.DS_Store + +# IDEs/editors +*.sw[op] +*~ +.vscode +.idea + +# Built documentation +docs/_build +docs.metadata + +# Virtual environment +env/ diff --git a/mod12-memcache/main.py b/mod12-memcache/main.py index ff2db41..0278ec9 100644 --- a/mod12-memcache/main.py +++ b/mod12-memcache/main.py @@ -1,4 +1,4 @@ -# Copyright 2020 Google LLC +# Copyright 2021 Google LLC # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -41,7 +41,7 @@ def root(): visitor = '{}: {}'.format(ip_addr, usr_agt) visits = memcache.get('visits') - # if cache empty or new visitor, register visit & run DB query + # register visit & run DB query if cache empty or new visitor if not visits or visits[0]['visitor'] != visitor: store_visit(ip_addr, usr_agt) visits = list(fetch_visits(10)) diff --git a/mod12-memcache/requirements.txt b/mod12-memcache/requirements.txt index 5c508e5..7e10602 100644 --- a/mod12-memcache/requirements.txt +++ b/mod12-memcache/requirements.txt @@ -1 +1 @@ -flask==1.1.2 +flask diff --git a/mod12b-memcache/.gcloudignore b/mod12b-memcache/.gcloudignore index bcf97a8..b4dbc7e 100644 --- a/mod12b-memcache/.gcloudignore +++ b/mod12b-memcache/.gcloudignore @@ -8,17 +8,73 @@ # .gcloudignore -# Ignore source code control maintenance files -.git +# Source code control files +.git/ .gitignore .hgignore .hg/ -# Python files -*.pyc -*.pyo +# README/text files +LICENSE +*.md + +# Tests/results (not in .gitignore) +noxfile.py +test_translate.py +pylintrc +pylintrc.test + +# most of .gitignore (except `lib`) +# +# Python +*.py[cod] __pycache__/ /setup.cfg -# no need to upload README -README.md +# C extensions +*.so + +# Packages +*.egg +*.egg-info +dist +build +eggs +.eggs +parts +bin +var +sdist +develop-eggs +.installed.cfg +lib64 +*.tgz + +# Installer logs +pip-log.txt + +# Tests/results +.nox/ +.pytest_cache/ +.cache +.pytype +.coverage +coverage.xml +*sponge_log.xml +system_tests/local_test_setup + +# Mac +.DS_Store + +# IDEs/editors +*.sw[op] +*~ +.vscode +.idea + +# Built documentation +docs/_build +docs.metadata + +# Virtual environment +env/ diff --git a/mod12b-memcache/main.py b/mod12b-memcache/main.py index 47fd8e3..8ae9d7d 100644 --- a/mod12b-memcache/main.py +++ b/mod12b-memcache/main.py @@ -1,4 +1,4 @@ -# Copyright 2020 Google LLC +# Copyright 2021 Google LLC # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -31,8 +31,8 @@ def store_visit(remote_addr, user_agent): def fetch_visits(limit): 'get most recent visits' - return [v.to_dict() for v in Visit.query().order( - -Visit.timestamp).fetch(limit)] + return (v.to_dict() for v in Visit.query().order( + -Visit.timestamp).fetch(limit)) @app.route('/') def root(): @@ -45,8 +45,7 @@ def root(): # register visit & run DB query if cache empty or new visitor if not visits or visits[0]['visitor'] != visitor: store_visit(ip_addr, usr_agt) - visits = fetch_visits(10) - memcache.set('visits', visits, HOUR) # must set() not add() here + visits = list(fetch_visits(10)) + memcache.set('visits', visits, HOUR) # set() not add() - # send context to template renderer return render_template('index.html', visits=visits) diff --git a/mod12b-memcache/requirements.txt b/mod12b-memcache/requirements.txt index c661cf7..a2e7c7f 100644 --- a/mod12b-memcache/requirements.txt +++ b/mod12b-memcache/requirements.txt @@ -1,2 +1,2 @@ -flask==1.1.2 +flask appengine-python-standard diff --git a/mod13a-memorystore/.gcloudignore b/mod13a-memorystore/.gcloudignore new file mode 100644 index 0000000..b4dbc7e --- /dev/null +++ b/mod13a-memorystore/.gcloudignore @@ -0,0 +1,80 @@ +# This file specifies files that are *not* uploaded to Google Cloud Platform +# using gcloud. It follows the same syntax as .gitignore, with the addition of +# "#!include" directives (which insert the entries of the given .gitignore-style +# file at that point). +# +# For more information, run: +# $ gcloud topic gcloudignore +# +.gcloudignore + +# Source code control files +.git/ +.gitignore +.hgignore +.hg/ + +# README/text files +LICENSE +*.md + +# Tests/results (not in .gitignore) +noxfile.py +test_translate.py +pylintrc +pylintrc.test + +# most of .gitignore (except `lib`) +# +# Python +*.py[cod] +__pycache__/ +/setup.cfg + +# C extensions +*.so + +# Packages +*.egg +*.egg-info +dist +build +eggs +.eggs +parts +bin +var +sdist +develop-eggs +.installed.cfg +lib64 +*.tgz + +# Installer logs +pip-log.txt + +# Tests/results +.nox/ +.pytest_cache/ +.cache +.pytype +.coverage +coverage.xml +*sponge_log.xml +system_tests/local_test_setup + +# Mac +.DS_Store + +# IDEs/editors +*.sw[op] +*~ +.vscode +.idea + +# Built documentation +docs/_build +docs.metadata + +# Virtual environment +env/ diff --git a/mod13a-memorystore/README.md b/mod13a-memorystore/README.md new file mode 100644 index 0000000..f4447f9 --- /dev/null +++ b/mod13a-memorystore/README.md @@ -0,0 +1,3 @@ +# Module 12 - Add usage of App Engine `memcache` to Flask `ndb` sample app + +This repo folder is the corresponding Python 2 code to the Module 12 codelab (TBD). The tutorial STARTs with the Python 2 code in the [Module 1 repo folder](/mod1-flask) and leads developers through adding usage of App Engine's `memcache`, culminating in the code in this folder. diff --git a/mod13a-memorystore/app.yaml b/mod13a-memorystore/app.yaml new file mode 100644 index 0000000..c683da5 --- /dev/null +++ b/mod13a-memorystore/app.yaml @@ -0,0 +1,28 @@ +# Copyright 2021 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +runtime: python27 +threadsafe: yes +api_version: 1 + +handlers: +- url: /.* + script: main.app + +env_variables: + REDIS_HOST: 'YOUR_REDIS_HOST' + REDIS_PORT: 'YOUR_REDIS_PORT' + +vpc_access_connector: + name: projects/PROJECT_ID/locations/REGION/connectors/CONNECTOR diff --git a/mod13a-memorystore/appengine_config.py b/mod13a-memorystore/appengine_config.py new file mode 100644 index 0000000..9760ebb --- /dev/null +++ b/mod13a-memorystore/appengine_config.py @@ -0,0 +1,20 @@ +# Copyright 2021 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from google.appengine.ext import vendor + +# Set PATH to your libraries folder. +PATH = 'lib' +# Add libraries installed in the PATH folder. +vendor.add(PATH) diff --git a/mod13a-memorystore/main.py b/mod13a-memorystore/main.py new file mode 100644 index 0000000..3384d71 --- /dev/null +++ b/mod13a-memorystore/main.py @@ -0,0 +1,56 @@ +# Copyright 2021 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import os +import pickle +from flask import Flask, render_template, request +from google.appengine.ext import ndb +import redis + +app = Flask(__name__) +HOUR = 3600 +REDIS_HOST = os.environ.get('REDIS_HOST', 'localhost') +REDIS_PORT = os.environ.get('REDIS_PORT', '6379') +REDIS = redis.Redis(host=REDIS_HOST, port=REDIS_PORT) + +class Visit(ndb.Model): + 'Visit entity registers visitor IP address & timestamp' + visitor = ndb.StringProperty() + timestamp = ndb.DateTimeProperty(auto_now_add=True) + +def store_visit(remote_addr, user_agent): + 'create new Visit entity in Datastore' + Visit(visitor='{}: {}'.format(remote_addr, user_agent)).put() + +def fetch_visits(limit): + 'get most recent visits' + return (v.to_dict() for v in Visit.query().order( + -Visit.timestamp).fetch(limit)) + +@app.route('/') +def root(): + 'main application (GET) handler' + # check for (hour-)cached visits + ip_addr, usr_agt = request.remote_addr, request.user_agent + visitor = '{}: {}'.format(ip_addr, usr_agt) + rsp = REDIS.get('visits') + visits = pickle.loads(rsp) if rsp else None + + # register visit & run DB query if cache empty or new visitor + if not visits or visits[0]['visitor'] != visitor: + store_visit(ip_addr, usr_agt) + visits = list(fetch_visits(10)) + REDIS.set('visits', pickle.dumps(visits), ex=HOUR) # set() not add() + + return render_template('index.html', visits=visits) diff --git a/mod13a-memorystore/requirements.txt b/mod13a-memorystore/requirements.txt new file mode 100644 index 0000000..1a5dc97 --- /dev/null +++ b/mod13a-memorystore/requirements.txt @@ -0,0 +1,2 @@ +flask +redis diff --git a/mod13a-memorystore/templates/index.html b/mod13a-memorystore/templates/index.html new file mode 100644 index 0000000..e140206 --- /dev/null +++ b/mod13a-memorystore/templates/index.html @@ -0,0 +1,16 @@ + + + +VisitMe Example + + +

VisitMe example

+

Last 10 visits

+
    +{% for visit in visits %} +
  • {{ visit.timestamp.ctime() }} from {{ visit.visitor }}
  • +{% endfor %} +
+ + + diff --git a/mod13b-memorystore/.gcloudignore b/mod13b-memorystore/.gcloudignore new file mode 100644 index 0000000..b4dbc7e --- /dev/null +++ b/mod13b-memorystore/.gcloudignore @@ -0,0 +1,80 @@ +# This file specifies files that are *not* uploaded to Google Cloud Platform +# using gcloud. It follows the same syntax as .gitignore, with the addition of +# "#!include" directives (which insert the entries of the given .gitignore-style +# file at that point). +# +# For more information, run: +# $ gcloud topic gcloudignore +# +.gcloudignore + +# Source code control files +.git/ +.gitignore +.hgignore +.hg/ + +# README/text files +LICENSE +*.md + +# Tests/results (not in .gitignore) +noxfile.py +test_translate.py +pylintrc +pylintrc.test + +# most of .gitignore (except `lib`) +# +# Python +*.py[cod] +__pycache__/ +/setup.cfg + +# C extensions +*.so + +# Packages +*.egg +*.egg-info +dist +build +eggs +.eggs +parts +bin +var +sdist +develop-eggs +.installed.cfg +lib64 +*.tgz + +# Installer logs +pip-log.txt + +# Tests/results +.nox/ +.pytest_cache/ +.cache +.pytype +.coverage +coverage.xml +*sponge_log.xml +system_tests/local_test_setup + +# Mac +.DS_Store + +# IDEs/editors +*.sw[op] +*~ +.vscode +.idea + +# Built documentation +docs/_build +docs.metadata + +# Virtual environment +env/ diff --git a/mod13b-memorystore/README.md b/mod13b-memorystore/README.md new file mode 100644 index 0000000..c66a990 --- /dev/null +++ b/mod13b-memorystore/README.md @@ -0,0 +1,5 @@ +# Module 12 - Add usage of App Engine `memcache` to Flask `ndb` sample app + +This repo folder is the corresponding Python 3 code to the Module 12 codelab (TBD). The tutorial STARTs with the Python 2 code in the [Module 1 repo folder](/mod1-flask) and leads developers through adding usage of App Engine's `memcache`, followed by a bonus migration to Python 3, culminating in the code in this folder. + +> **LEGACY SERVICES PUBLIC PREVIEW**: Accessing legacy services such as App Engine `ndb` and `memcache` from Python 3 (and next generation App Engine in general) is available in a public preview. See the [Sep 2021 announcement](https://twitter.com/googledevs/status/1445916786755571712) for more information. diff --git a/mod13b-memorystore/app.yaml b/mod13b-memorystore/app.yaml new file mode 100644 index 0000000..aa3960c --- /dev/null +++ b/mod13b-memorystore/app.yaml @@ -0,0 +1,22 @@ +# Copyright 2021 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +runtime: python39 + +env_variables: + REDIS_HOST: 'YOUR_REDIS_HOST' + REDIS_PORT: 'YOUR_REDIS_PORT' + +vpc_access_connector: + name: projects/PROJECT_ID/locations/REGION/connectors/CONNECTOR diff --git a/mod13b-memorystore/main.py b/mod13b-memorystore/main.py new file mode 100644 index 0000000..6923fae --- /dev/null +++ b/mod13b-memorystore/main.py @@ -0,0 +1,59 @@ +# Copyright 2021 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import os +import pickle +from flask import Flask, render_template, request +from google.cloud import ndb +import redis + +app = Flask(__name__) +ds_client = ndb.Client() +HOUR = 3600 +REDIS_HOST = os.environ.get('REDIS_HOST', 'localhost') +REDIS_PORT = os.environ.get('REDIS_PORT', '6379') +REDIS = redis.Redis(host=REDIS_HOST, port=REDIS_PORT) + +class Visit(ndb.Model): + 'Visit entity registers visitor IP address & timestamp' + visitor = ndb.StringProperty() + timestamp = ndb.DateTimeProperty(auto_now_add=True) + +def store_visit(remote_addr, user_agent): + 'create new Visit entity in Datastore' + with ds_client.context(): + Visit(visitor='{}: {}'.format(remote_addr, user_agent)).put() + +def fetch_visits(limit): + 'get most recent visits' + with ds_client.context(): + return (v.to_dict() for v in Visit.query().order( + -Visit.timestamp).fetch(limit)) + +@app.route('/') +def root(): + 'main application (GET) handler' + # check for (hour-)cached visits + ip_addr, usr_agt = request.remote_addr, request.user_agent + visitor = '{}: {}'.format(ip_addr, usr_agt) + rsp = REDIS.get('visits') + visits = pickle.loads(rsp) if rsp else None + + # register visit & run DB query if cache empty or new visitor + if not visits or visits[0]['visitor'] != visitor: + store_visit(ip_addr, usr_agt) + visits = list(fetch_visits(10)) + REDIS.set('visits', pickle.dumps(visits), ex=HOUR) # set() not add() + + return render_template('index.html', visits=visits) diff --git a/mod13b-memorystore/requirements.txt b/mod13b-memorystore/requirements.txt new file mode 100644 index 0000000..1bdb2fd --- /dev/null +++ b/mod13b-memorystore/requirements.txt @@ -0,0 +1,3 @@ +flask +redis +google-cloud-ndb diff --git a/mod13b-memorystore/templates/index.html b/mod13b-memorystore/templates/index.html new file mode 100644 index 0000000..e140206 --- /dev/null +++ b/mod13b-memorystore/templates/index.html @@ -0,0 +1,16 @@ + + + +VisitMe Example + + +

VisitMe example

+

Last 10 visits

+
    +{% for visit in visits %} +
  • {{ visit.timestamp.ctime() }} from {{ visit.visitor }}
  • +{% endfor %} +
+ + + diff --git a/mod1b-flask/.gcloudignore b/mod1b-flask/.gcloudignore index bcf97a8..b4dbc7e 100644 --- a/mod1b-flask/.gcloudignore +++ b/mod1b-flask/.gcloudignore @@ -8,17 +8,73 @@ # .gcloudignore -# Ignore source code control maintenance files -.git +# Source code control files +.git/ .gitignore .hgignore .hg/ -# Python files -*.pyc -*.pyo +# README/text files +LICENSE +*.md + +# Tests/results (not in .gitignore) +noxfile.py +test_translate.py +pylintrc +pylintrc.test + +# most of .gitignore (except `lib`) +# +# Python +*.py[cod] __pycache__/ /setup.cfg -# no need to upload README -README.md +# C extensions +*.so + +# Packages +*.egg +*.egg-info +dist +build +eggs +.eggs +parts +bin +var +sdist +develop-eggs +.installed.cfg +lib64 +*.tgz + +# Installer logs +pip-log.txt + +# Tests/results +.nox/ +.pytest_cache/ +.cache +.pytype +.coverage +coverage.xml +*sponge_log.xml +system_tests/local_test_setup + +# Mac +.DS_Store + +# IDEs/editors +*.sw[op] +*~ +.vscode +.idea + +# Built documentation +docs/_build +docs.metadata + +# Virtual environment +env/ diff --git a/mod1b-flask/requirements.txt b/mod1b-flask/requirements.txt index c661cf7..a2e7c7f 100644 --- a/mod1b-flask/requirements.txt +++ b/mod1b-flask/requirements.txt @@ -1,2 +1,2 @@ -flask==1.1.2 +flask appengine-python-standard diff --git a/mod2a-cloudndb/.gcloudignore b/mod2a-cloudndb/.gcloudignore index bcf97a8..b4dbc7e 100644 --- a/mod2a-cloudndb/.gcloudignore +++ b/mod2a-cloudndb/.gcloudignore @@ -8,17 +8,73 @@ # .gcloudignore -# Ignore source code control maintenance files -.git +# Source code control files +.git/ .gitignore .hgignore .hg/ -# Python files -*.pyc -*.pyo +# README/text files +LICENSE +*.md + +# Tests/results (not in .gitignore) +noxfile.py +test_translate.py +pylintrc +pylintrc.test + +# most of .gitignore (except `lib`) +# +# Python +*.py[cod] __pycache__/ /setup.cfg -# no need to upload README -README.md +# C extensions +*.so + +# Packages +*.egg +*.egg-info +dist +build +eggs +.eggs +parts +bin +var +sdist +develop-eggs +.installed.cfg +lib64 +*.tgz + +# Installer logs +pip-log.txt + +# Tests/results +.nox/ +.pytest_cache/ +.cache +.pytype +.coverage +coverage.xml +*sponge_log.xml +system_tests/local_test_setup + +# Mac +.DS_Store + +# IDEs/editors +*.sw[op] +*~ +.vscode +.idea + +# Built documentation +docs/_build +docs.metadata + +# Virtual environment +env/ diff --git a/mod2a-cloudndb/requirements.txt b/mod2a-cloudndb/requirements.txt index c5fbb74..84932a7 100644 --- a/mod2a-cloudndb/requirements.txt +++ b/mod2a-cloudndb/requirements.txt @@ -1,2 +1,2 @@ -flask==1.1.2 -google-cloud-ndb==1.9.0 +flask +google-cloud-ndb diff --git a/mod2b-cloudndb/.gcloudignore b/mod2b-cloudndb/.gcloudignore index bcf97a8..b4dbc7e 100644 --- a/mod2b-cloudndb/.gcloudignore +++ b/mod2b-cloudndb/.gcloudignore @@ -8,17 +8,73 @@ # .gcloudignore -# Ignore source code control maintenance files -.git +# Source code control files +.git/ .gitignore .hgignore .hg/ -# Python files -*.pyc -*.pyo +# README/text files +LICENSE +*.md + +# Tests/results (not in .gitignore) +noxfile.py +test_translate.py +pylintrc +pylintrc.test + +# most of .gitignore (except `lib`) +# +# Python +*.py[cod] __pycache__/ /setup.cfg -# no need to upload README -README.md +# C extensions +*.so + +# Packages +*.egg +*.egg-info +dist +build +eggs +.eggs +parts +bin +var +sdist +develop-eggs +.installed.cfg +lib64 +*.tgz + +# Installer logs +pip-log.txt + +# Tests/results +.nox/ +.pytest_cache/ +.cache +.pytype +.coverage +coverage.xml +*sponge_log.xml +system_tests/local_test_setup + +# Mac +.DS_Store + +# IDEs/editors +*.sw[op] +*~ +.vscode +.idea + +# Built documentation +docs/_build +docs.metadata + +# Virtual environment +env/ diff --git a/mod2b-cloudndb/requirements.txt b/mod2b-cloudndb/requirements.txt index c5fbb74..81ef820 100644 --- a/mod2b-cloudndb/requirements.txt +++ b/mod2b-cloudndb/requirements.txt @@ -1,2 +1,2 @@ -flask==1.1.2 -google-cloud-ndb==1.9.0 +flask +google-cloud-ndb==1.11.1 diff --git a/mod3a-datastore/.gcloudignore b/mod3a-datastore/.gcloudignore index bcf97a8..b4dbc7e 100644 --- a/mod3a-datastore/.gcloudignore +++ b/mod3a-datastore/.gcloudignore @@ -8,17 +8,73 @@ # .gcloudignore -# Ignore source code control maintenance files -.git +# Source code control files +.git/ .gitignore .hgignore .hg/ -# Python files -*.pyc -*.pyo +# README/text files +LICENSE +*.md + +# Tests/results (not in .gitignore) +noxfile.py +test_translate.py +pylintrc +pylintrc.test + +# most of .gitignore (except `lib`) +# +# Python +*.py[cod] __pycache__/ /setup.cfg -# no need to upload README -README.md +# C extensions +*.so + +# Packages +*.egg +*.egg-info +dist +build +eggs +.eggs +parts +bin +var +sdist +develop-eggs +.installed.cfg +lib64 +*.tgz + +# Installer logs +pip-log.txt + +# Tests/results +.nox/ +.pytest_cache/ +.cache +.pytype +.coverage +coverage.xml +*sponge_log.xml +system_tests/local_test_setup + +# Mac +.DS_Store + +# IDEs/editors +*.sw[op] +*~ +.vscode +.idea + +# Built documentation +docs/_build +docs.metadata + +# Virtual environment +env/ diff --git a/mod3a-datastore/requirements.txt b/mod3a-datastore/requirements.txt index 7092ef7..78086d0 100644 --- a/mod3a-datastore/requirements.txt +++ b/mod3a-datastore/requirements.txt @@ -1,2 +1,2 @@ -flask==1.1.2 +flask google-cloud-datastore==1.15.3 diff --git a/mod3b-datastore/.gcloudignore b/mod3b-datastore/.gcloudignore index bcf97a8..b4dbc7e 100644 --- a/mod3b-datastore/.gcloudignore +++ b/mod3b-datastore/.gcloudignore @@ -8,17 +8,73 @@ # .gcloudignore -# Ignore source code control maintenance files -.git +# Source code control files +.git/ .gitignore .hgignore .hg/ -# Python files -*.pyc -*.pyo +# README/text files +LICENSE +*.md + +# Tests/results (not in .gitignore) +noxfile.py +test_translate.py +pylintrc +pylintrc.test + +# most of .gitignore (except `lib`) +# +# Python +*.py[cod] __pycache__/ /setup.cfg -# no need to upload README -README.md +# C extensions +*.so + +# Packages +*.egg +*.egg-info +dist +build +eggs +.eggs +parts +bin +var +sdist +develop-eggs +.installed.cfg +lib64 +*.tgz + +# Installer logs +pip-log.txt + +# Tests/results +.nox/ +.pytest_cache/ +.cache +.pytype +.coverage +coverage.xml +*sponge_log.xml +system_tests/local_test_setup + +# Mac +.DS_Store + +# IDEs/editors +*.sw[op] +*~ +.vscode +.idea + +# Built documentation +docs/_build +docs.metadata + +# Virtual environment +env/ diff --git a/mod3b-datastore/requirements.txt b/mod3b-datastore/requirements.txt index f174e5c..3718741 100644 --- a/mod3b-datastore/requirements.txt +++ b/mod3b-datastore/requirements.txt @@ -1,2 +1,2 @@ flask==1.1.2 -google-cloud-datastore==2.1.3 +google-cloud-datastore==2.3.0 diff --git a/mod4a-rundocker/.gcloudignore b/mod4a-rundocker/.gcloudignore index bcf97a8..b4dbc7e 100644 --- a/mod4a-rundocker/.gcloudignore +++ b/mod4a-rundocker/.gcloudignore @@ -8,17 +8,73 @@ # .gcloudignore -# Ignore source code control maintenance files -.git +# Source code control files +.git/ .gitignore .hgignore .hg/ -# Python files -*.pyc -*.pyo +# README/text files +LICENSE +*.md + +# Tests/results (not in .gitignore) +noxfile.py +test_translate.py +pylintrc +pylintrc.test + +# most of .gitignore (except `lib`) +# +# Python +*.py[cod] __pycache__/ /setup.cfg -# no need to upload README -README.md +# C extensions +*.so + +# Packages +*.egg +*.egg-info +dist +build +eggs +.eggs +parts +bin +var +sdist +develop-eggs +.installed.cfg +lib64 +*.tgz + +# Installer logs +pip-log.txt + +# Tests/results +.nox/ +.pytest_cache/ +.cache +.pytype +.coverage +coverage.xml +*sponge_log.xml +system_tests/local_test_setup + +# Mac +.DS_Store + +# IDEs/editors +*.sw[op] +*~ +.vscode +.idea + +# Built documentation +docs/_build +docs.metadata + +# Virtual environment +env/ diff --git a/mod4a-rundocker/requirements.txt b/mod4a-rundocker/requirements.txt index 7bdc163..ba543a5 100644 --- a/mod4a-rundocker/requirements.txt +++ b/mod4a-rundocker/requirements.txt @@ -1,3 +1,3 @@ gunicorn==19.10.0 -flask==1.1.2 -google-cloud-ndb==1.9.0 +flask +google-cloud-ndb==1.11.1 diff --git a/mod4b-rundocker/.gcloudignore b/mod4b-rundocker/.gcloudignore index bcf97a8..b4dbc7e 100644 --- a/mod4b-rundocker/.gcloudignore +++ b/mod4b-rundocker/.gcloudignore @@ -8,17 +8,73 @@ # .gcloudignore -# Ignore source code control maintenance files -.git +# Source code control files +.git/ .gitignore .hgignore .hg/ -# Python files -*.pyc -*.pyo +# README/text files +LICENSE +*.md + +# Tests/results (not in .gitignore) +noxfile.py +test_translate.py +pylintrc +pylintrc.test + +# most of .gitignore (except `lib`) +# +# Python +*.py[cod] __pycache__/ /setup.cfg -# no need to upload README -README.md +# C extensions +*.so + +# Packages +*.egg +*.egg-info +dist +build +eggs +.eggs +parts +bin +var +sdist +develop-eggs +.installed.cfg +lib64 +*.tgz + +# Installer logs +pip-log.txt + +# Tests/results +.nox/ +.pytest_cache/ +.cache +.pytype +.coverage +coverage.xml +*sponge_log.xml +system_tests/local_test_setup + +# Mac +.DS_Store + +# IDEs/editors +*.sw[op] +*~ +.vscode +.idea + +# Built documentation +docs/_build +docs.metadata + +# Virtual environment +env/ diff --git a/mod4b-rundocker/requirements.txt b/mod4b-rundocker/requirements.txt index 3f9026e..5780c7e 100644 --- a/mod4b-rundocker/requirements.txt +++ b/mod4b-rundocker/requirements.txt @@ -1,3 +1,3 @@ -gunicorn==20.1.0 -flask==1.1.2 -google-cloud-datastore==2.1.3 +gunicorn +flask +google-cloud-datastore==2.3.0 diff --git a/mod5-runbldpks/.gcloudignore b/mod5-runbldpks/.gcloudignore index bcf97a8..b4dbc7e 100644 --- a/mod5-runbldpks/.gcloudignore +++ b/mod5-runbldpks/.gcloudignore @@ -8,17 +8,73 @@ # .gcloudignore -# Ignore source code control maintenance files -.git +# Source code control files +.git/ .gitignore .hgignore .hg/ -# Python files -*.pyc -*.pyo +# README/text files +LICENSE +*.md + +# Tests/results (not in .gitignore) +noxfile.py +test_translate.py +pylintrc +pylintrc.test + +# most of .gitignore (except `lib`) +# +# Python +*.py[cod] __pycache__/ /setup.cfg -# no need to upload README -README.md +# C extensions +*.so + +# Packages +*.egg +*.egg-info +dist +build +eggs +.eggs +parts +bin +var +sdist +develop-eggs +.installed.cfg +lib64 +*.tgz + +# Installer logs +pip-log.txt + +# Tests/results +.nox/ +.pytest_cache/ +.cache +.pytype +.coverage +coverage.xml +*sponge_log.xml +system_tests/local_test_setup + +# Mac +.DS_Store + +# IDEs/editors +*.sw[op] +*~ +.vscode +.idea + +# Built documentation +docs/_build +docs.metadata + +# Virtual environment +env/ diff --git a/mod5-runbldpks/requirements.txt b/mod5-runbldpks/requirements.txt index 3a4be91..e612636 100644 --- a/mod5-runbldpks/requirements.txt +++ b/mod5-runbldpks/requirements.txt @@ -1,3 +1,3 @@ -gunicorn==20.1.0 -flask==1.1.2 -google-cloud-ndb==1.9.0 +gunicor +flask +google-cloud-ndb diff --git a/mod6-firestore/.gcloudignore b/mod6-firestore/.gcloudignore index bcf97a8..b4dbc7e 100644 --- a/mod6-firestore/.gcloudignore +++ b/mod6-firestore/.gcloudignore @@ -8,17 +8,73 @@ # .gcloudignore -# Ignore source code control maintenance files -.git +# Source code control files +.git/ .gitignore .hgignore .hg/ -# Python files -*.pyc -*.pyo +# README/text files +LICENSE +*.md + +# Tests/results (not in .gitignore) +noxfile.py +test_translate.py +pylintrc +pylintrc.test + +# most of .gitignore (except `lib`) +# +# Python +*.py[cod] __pycache__/ /setup.cfg -# no need to upload README -README.md +# C extensions +*.so + +# Packages +*.egg +*.egg-info +dist +build +eggs +.eggs +parts +bin +var +sdist +develop-eggs +.installed.cfg +lib64 +*.tgz + +# Installer logs +pip-log.txt + +# Tests/results +.nox/ +.pytest_cache/ +.cache +.pytype +.coverage +coverage.xml +*sponge_log.xml +system_tests/local_test_setup + +# Mac +.DS_Store + +# IDEs/editors +*.sw[op] +*~ +.vscode +.idea + +# Built documentation +docs/_build +docs.metadata + +# Virtual environment +env/ diff --git a/mod6-firestore/requirements.txt b/mod6-firestore/requirements.txt index 9602ece..b8191d9 100644 --- a/mod6-firestore/requirements.txt +++ b/mod6-firestore/requirements.txt @@ -1,2 +1,2 @@ -flask==1.1.2 -google-cloud-firestore==2.1.3 +flask +google-cloud-firestore==2.3.4 diff --git a/mod7-gaetasks/.gcloudignore b/mod7-gaetasks/.gcloudignore index bcf97a8..b4dbc7e 100644 --- a/mod7-gaetasks/.gcloudignore +++ b/mod7-gaetasks/.gcloudignore @@ -8,17 +8,73 @@ # .gcloudignore -# Ignore source code control maintenance files -.git +# Source code control files +.git/ .gitignore .hgignore .hg/ -# Python files -*.pyc -*.pyo +# README/text files +LICENSE +*.md + +# Tests/results (not in .gitignore) +noxfile.py +test_translate.py +pylintrc +pylintrc.test + +# most of .gitignore (except `lib`) +# +# Python +*.py[cod] __pycache__/ /setup.cfg -# no need to upload README -README.md +# C extensions +*.so + +# Packages +*.egg +*.egg-info +dist +build +eggs +.eggs +parts +bin +var +sdist +develop-eggs +.installed.cfg +lib64 +*.tgz + +# Installer logs +pip-log.txt + +# Tests/results +.nox/ +.pytest_cache/ +.cache +.pytype +.coverage +coverage.xml +*sponge_log.xml +system_tests/local_test_setup + +# Mac +.DS_Store + +# IDEs/editors +*.sw[op] +*~ +.vscode +.idea + +# Built documentation +docs/_build +docs.metadata + +# Virtual environment +env/ diff --git a/mod7-gaetasks/requirements.txt b/mod7-gaetasks/requirements.txt index 5c508e5..7e10602 100644 --- a/mod7-gaetasks/requirements.txt +++ b/mod7-gaetasks/requirements.txt @@ -1 +1 @@ -flask==1.1.2 +flask diff --git a/mod7b-gaetasks/.gcloudignore b/mod7b-gaetasks/.gcloudignore index bcf97a8..b4dbc7e 100644 --- a/mod7b-gaetasks/.gcloudignore +++ b/mod7b-gaetasks/.gcloudignore @@ -8,17 +8,73 @@ # .gcloudignore -# Ignore source code control maintenance files -.git +# Source code control files +.git/ .gitignore .hgignore .hg/ -# Python files -*.pyc -*.pyo +# README/text files +LICENSE +*.md + +# Tests/results (not in .gitignore) +noxfile.py +test_translate.py +pylintrc +pylintrc.test + +# most of .gitignore (except `lib`) +# +# Python +*.py[cod] __pycache__/ /setup.cfg -# no need to upload README -README.md +# C extensions +*.so + +# Packages +*.egg +*.egg-info +dist +build +eggs +.eggs +parts +bin +var +sdist +develop-eggs +.installed.cfg +lib64 +*.tgz + +# Installer logs +pip-log.txt + +# Tests/results +.nox/ +.pytest_cache/ +.cache +.pytype +.coverage +coverage.xml +*sponge_log.xml +system_tests/local_test_setup + +# Mac +.DS_Store + +# IDEs/editors +*.sw[op] +*~ +.vscode +.idea + +# Built documentation +docs/_build +docs.metadata + +# Virtual environment +env/ diff --git a/mod7b-gaetasks/requirements.txt b/mod7b-gaetasks/requirements.txt index c661cf7..a2e7c7f 100644 --- a/mod7b-gaetasks/requirements.txt +++ b/mod7b-gaetasks/requirements.txt @@ -1,2 +1,2 @@ -flask==1.1.2 +flask appengine-python-standard diff --git a/mod8-cloudtasks/.gcloudignore b/mod8-cloudtasks/.gcloudignore index bcf97a8..b4dbc7e 100644 --- a/mod8-cloudtasks/.gcloudignore +++ b/mod8-cloudtasks/.gcloudignore @@ -8,17 +8,73 @@ # .gcloudignore -# Ignore source code control maintenance files -.git +# Source code control files +.git/ .gitignore .hgignore .hg/ -# Python files -*.pyc -*.pyo +# README/text files +LICENSE +*.md + +# Tests/results (not in .gitignore) +noxfile.py +test_translate.py +pylintrc +pylintrc.test + +# most of .gitignore (except `lib`) +# +# Python +*.py[cod] __pycache__/ /setup.cfg -# no need to upload README -README.md +# C extensions +*.so + +# Packages +*.egg +*.egg-info +dist +build +eggs +.eggs +parts +bin +var +sdist +develop-eggs +.installed.cfg +lib64 +*.tgz + +# Installer logs +pip-log.txt + +# Tests/results +.nox/ +.pytest_cache/ +.cache +.pytype +.coverage +coverage.xml +*sponge_log.xml +system_tests/local_test_setup + +# Mac +.DS_Store + +# IDEs/editors +*.sw[op] +*~ +.vscode +.idea + +# Built documentation +docs/_build +docs.metadata + +# Virtual environment +env/ diff --git a/mod8-cloudtasks/requirements.txt b/mod8-cloudtasks/requirements.txt index 6241dea..67c24bf 100644 --- a/mod8-cloudtasks/requirements.txt +++ b/mod8-cloudtasks/requirements.txt @@ -1,3 +1,3 @@ -flask==1.1.2 -google-cloud-ndb==1.7.1 -google-cloud-tasks==1.5.0 +flask +google-cloud-ndb +google-cloud-tasks diff --git a/mod9-py3fstasks/.gcloudignore b/mod9-py3fstasks/.gcloudignore index bcf97a8..b4dbc7e 100644 --- a/mod9-py3fstasks/.gcloudignore +++ b/mod9-py3fstasks/.gcloudignore @@ -8,17 +8,73 @@ # .gcloudignore -# Ignore source code control maintenance files -.git +# Source code control files +.git/ .gitignore .hgignore .hg/ -# Python files -*.pyc -*.pyo +# README/text files +LICENSE +*.md + +# Tests/results (not in .gitignore) +noxfile.py +test_translate.py +pylintrc +pylintrc.test + +# most of .gitignore (except `lib`) +# +# Python +*.py[cod] __pycache__/ /setup.cfg -# no need to upload README -README.md +# C extensions +*.so + +# Packages +*.egg +*.egg-info +dist +build +eggs +.eggs +parts +bin +var +sdist +develop-eggs +.installed.cfg +lib64 +*.tgz + +# Installer logs +pip-log.txt + +# Tests/results +.nox/ +.pytest_cache/ +.cache +.pytype +.coverage +coverage.xml +*sponge_log.xml +system_tests/local_test_setup + +# Mac +.DS_Store + +# IDEs/editors +*.sw[op] +*~ +.vscode +.idea + +# Built documentation +docs/_build +docs.metadata + +# Virtual environment +env/ diff --git a/mod9-py3fstasks/requirements.txt b/mod9-py3fstasks/requirements.txt index 98ea30b..9f80c69 100644 --- a/mod9-py3fstasks/requirements.txt +++ b/mod9-py3fstasks/requirements.txt @@ -1,3 +1,3 @@ -flask==1.1.2 +flask google-cloud-firestore==2.3.4 -google-cloud-tasks==2.7.0 +google-cloud-tasks==2.7.1 From baa34642228cee88492993a82395e638806599b6 Mon Sep 17 00:00:00 2001 From: wesley chun Date: Thu, 16 Dec 2021 14:01:38 -0800 Subject: [PATCH 21/56] add Module 13/Cloud Memorystore and other tweaks --- README.md | 39 ++++++--------------------------------- 1 file changed, 6 insertions(+), 33 deletions(-) diff --git a/README.md b/README.md index 0aa7467..2992db4 100644 --- a/README.md +++ b/README.md @@ -83,13 +83,14 @@ Module | Topic | Video | Codelab | START here | FINISH here 3|Migrate to Cloud Datastore| [link](http://twitter.com/googledevs/status/1422966928910393347?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrcloudds_201003&utm_content=-) | [link](http://g.co/codelabs/pae-migrate-datastore) | Module 2 [code](/mod2a-cloudndb) (2.x) & [code](/mod2b-cloudndb) (3.x) | Module 3 [code](/mod3a-datastore) (2.x) & [code](/mod3b-datastore) (3.x) 4|Migrate to Cloud Run with Docker| [link](https://twitter.com/googledevs/status/1428041270702735362?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrcrdckr_sms_201017&utm_content=-)| [link](http://g.co/codelabs/pae-migrate-rundocker) | Module 2 [code](/mod2a-cloudndb) (2.x) & Module 3 [code](/mod3b-datastore) (3.x) | Module 4 [code](/mod4a-rundocker) (2.x) & [code](/mod4b-rundocker) (3.x) 5|Migrate to Cloud Run with Buildpacks| [link](https://twitter.com/googledevs/status/1433113274984271875?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrcrbdpk_sms_201031&utm_content=-) | [link](http://g.co/codelabs/pae-migrate-runbldpks) | Module 2 [code](/mod2b-cloudndb) (3.x) | Module 5 [code](/mod5-runbldpks) (3.x) -6|Migrate to Cloud Firestore (app)| [link](http://youtu.be/wNs36kukVOQ?list=PL2pQQBHvYcs0PEecTcLD9_VaLvuhK0_VQ?utm_source=youtube&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrcloudfsam_sms_201014&utm_content=info_card) | [link](http://g.co/codelabs/pae-migrate-firestore) | Module 3 [code](/mod3b-datastore) (3.x) | Module 6 [code](/mod6-firestore) (3.x) +6|Migrate to Cloud Firestore| _N/A_ | _N/A_ | Module 3 [code](/mod3b-datastore) (3.x) | _no work required; Datastore upgrade automatic_ 7|Add App Engine `taskqueue` push tasks| [link](https://twitter.com/googledevs/status/1443410302113099778?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrgaetasks_sms_201028&utm_content=-) | [link](http://g.co/codelabs/pae-migrate-gaetasks) | Module 1 [code](/mod1-flask) (2.x) | Module 7 [code](/mod7-gaetasks) (2.x) & [code](/mod7b-gaetasks) (3.x) 8|Migrate to Cloud Tasks| [link](https://twitter.com/googledevs/status/1450960021018267656?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrcloudtasks_sms_201112&utm_content=-) | [link](http://g.co/codelabs/pae-migrate-cloudtasks) | Module 7 [code](/mod7-gaetasks) (2.x) | Module 8 [code](/mod8-cloudtasks) (2.x) -9|Migrate to Python 3, Cloud Firestore & Cloud Tasks v2| _TBD_ | _TBD_ | Module 8 [code](/mod8-cloudtasks) (2.x) | Module 9 [code](/mod9-py3fstasks) (3.x) -10|Migrate to Cloud Firestore (data)| _TBD_ | _N/A_ | _N/A_ | _TBD_ +9|Migrate to Python 3, Cloud Datastore & Cloud Tasks v2| _TBD_ | _TBD_ | Module 8 [code](/mod8-cloudtasks) (2.x) | _TBD_ +10|Migrate Datastore/Firestore data to another project| _TBD_ | _N/A_ | _N/A_ | _TBD_ 11|Migrate to Cloud Functions| _TBD_ | _TBD_ | Module 2 [code](/mod2b-cloudndb) (3.x) | Module 11 [code](/mod11-functions) (3.x) 12|Add App Engine `memcache`| _TBD_ | _TBD_ | Module 1 [code](/mod1-flask) (2.x) | Module 12 [code](/mod12-memcache) (2.x) & [code](/mod12b-memcache) (3.x) +13|Migrate to Cloud Memorystore| _TBD_ | _TBD_ | Module 12 [code](/mod12-memcache) (2.x) & [code](/mod12b-memcache) (3.x) | Module 13 [code](/mod13a-memorystore) (2.x) & [code](/mod13b-memorystore) (3.x) ### Table of contents @@ -139,14 +140,14 @@ If there is a logical codelab to do immediately after completing one, they will - FINISH: [Module 8 code - Cloud Tasks](/mod8-cloudtasks) (2.x) - NEXT: Module 9 - migrate to Python 3 and Cloud Datastore -- **Module 9 codelab** (TBD): **Migrate a Python 2 Cloud NDB & Cloud Tasks (v1) app to a Python 3 Cloud Firestore & Cloud Tasks (v2) app** +- **Module 9 codelab** (TBD): **Migrate a Python 2 Cloud NDB & Cloud Tasks (v1) app to a Python 3 Cloud Datastore & Cloud Tasks (v2) app** - **Optional** migrations - Migrating to Python 3 is not required but recommended as Python 2 has been sunset - Migrating to Cloud Firestore is *very* optional as Cloud NDB works on 3.x and most importantly, Cloud Firestore requires a completely new GCP project - Python 2 - START: [Module 8 code - Cloud Tasks](/mod8-cloudtasks) (2.x) - Python 3 - - FINISH: [Module 9 code - Cloud Firestore & Tasks](/mod9-py3fstasks) (3.x) + - FINISH: _TBD_ - RECOMMENDED: - Module 11 - migrate to Cloud Functions - Module 5 - migrate to Cloud Run container with Cloud Buildpacks @@ -197,34 +198,6 @@ If there is a logical codelab to do immediately after completing one, they will - Module 5 - migrate to Cloud Run container with Cloud Buildpacks - Module 4 - migrate to Cloud Run container with Docker -- [Module 6 codelab](http://g.co/codelabs/pae-migrate-firestore): **Migrate from Cloud Datastore to [Cloud Firestore](http://cloud.google.com/firestore)** (app) - - **Highly optional** migration - - Requires new project & Datastore has better write performance (currently) and is a fully-supported product and will get new features, including some Firestore native features - - If you **must have** Firestore's Firebase features now and can't wait - - Python 3 only - - START: [Module 3 code - Cloud Datastore](/mod3b-datastore) (3.x) - - FINISH: [Module 6 code - Cloud Firestore](/mod6-firestore) (3.x) - - NEXT: - - Module 10 - migrate to Cloud Firestore (data) - - RECOMMENDED: - - Module 7 - add App Engine (push) tasks - - OTHER OPTIONS (in somewhat priority order): - - Module 11 - migrate to Cloud Functions - - Module 5 - migrate to Cloud Run container with Cloud Buildpacks - - Module 4 - migrate to Cloud Run container with Docker - -- **Module 10 codelab** (TBD): **Migrate from Cloud Datastore to [Cloud Firestore](http://cloud.google.com/firestore)** (data) - - **Highly optional** migration - - Requires new project & Datastore has better write performance (currently) - - If you **must have** Firestore's Firebase features - - Python 3 only - - RECOMMENDED: - - Module 7 - add App Engine (push) tasks - - OTHER OPTIONS (in somewhat priority order): - - Module 11 - migrate to Cloud Functions - - Module 5 - migrate to Cloud Run container with Cloud Buildpacks - - Module 4 - migrate to Cloud Run container with Docker - - **Module 11 codelab** (TBD): **Migrate from App Engine to Cloud Functions** - **Optional** migration - Recommende for small apps or for breaking up large apps into multiple microservices From d55e7f4e6c0232a8267e238b160f1a9253bd3245 Mon Sep 17 00:00:00 2001 From: wesley chun Date: Thu, 16 Dec 2021 18:32:41 -0800 Subject: [PATCH 22/56] Memorystore migration cleanup --- README.md | 2 +- mod13a-memorystore/README.md | 4 ++-- mod13a-memorystore/main.py | 13 ++++++++----- mod13a-memorystore/requirements.txt | 1 + mod13b-memorystore/README.md | 9 ++++++--- mod13b-memorystore/main.py | 26 +++++++++++++------------- mod13b-memorystore/requirements.txt | 2 +- mod2a-cloudndb/README.md | 4 ++-- mod2b-cloudndb/README.md | 4 ++-- mod8-cloudtasks/README.md | 2 +- 10 files changed, 37 insertions(+), 30 deletions(-) diff --git a/README.md b/README.md index 2992db4..62e25a3 100644 --- a/README.md +++ b/README.md @@ -83,7 +83,7 @@ Module | Topic | Video | Codelab | START here | FINISH here 3|Migrate to Cloud Datastore| [link](http://twitter.com/googledevs/status/1422966928910393347?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrcloudds_201003&utm_content=-) | [link](http://g.co/codelabs/pae-migrate-datastore) | Module 2 [code](/mod2a-cloudndb) (2.x) & [code](/mod2b-cloudndb) (3.x) | Module 3 [code](/mod3a-datastore) (2.x) & [code](/mod3b-datastore) (3.x) 4|Migrate to Cloud Run with Docker| [link](https://twitter.com/googledevs/status/1428041270702735362?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrcrdckr_sms_201017&utm_content=-)| [link](http://g.co/codelabs/pae-migrate-rundocker) | Module 2 [code](/mod2a-cloudndb) (2.x) & Module 3 [code](/mod3b-datastore) (3.x) | Module 4 [code](/mod4a-rundocker) (2.x) & [code](/mod4b-rundocker) (3.x) 5|Migrate to Cloud Run with Buildpacks| [link](https://twitter.com/googledevs/status/1433113274984271875?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrcrbdpk_sms_201031&utm_content=-) | [link](http://g.co/codelabs/pae-migrate-runbldpks) | Module 2 [code](/mod2b-cloudndb) (3.x) | Module 5 [code](/mod5-runbldpks) (3.x) -6|Migrate to Cloud Firestore| _N/A_ | _N/A_ | Module 3 [code](/mod3b-datastore) (3.x) | _no work required; Datastore upgrade automatic_ +6|Migrate to Cloud Firestore| _N/A_ | _N/A_ | Module 3 [code](/mod3b-datastore) (3.x) | _no work required; [Datastore upgrade automatic](https://cloud.google.com/datastore/docs/upgrade-to-firestore)_ 7|Add App Engine `taskqueue` push tasks| [link](https://twitter.com/googledevs/status/1443410302113099778?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrgaetasks_sms_201028&utm_content=-) | [link](http://g.co/codelabs/pae-migrate-gaetasks) | Module 1 [code](/mod1-flask) (2.x) | Module 7 [code](/mod7-gaetasks) (2.x) & [code](/mod7b-gaetasks) (3.x) 8|Migrate to Cloud Tasks| [link](https://twitter.com/googledevs/status/1450960021018267656?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrcloudtasks_sms_201112&utm_content=-) | [link](http://g.co/codelabs/pae-migrate-cloudtasks) | Module 7 [code](/mod7-gaetasks) (2.x) | Module 8 [code](/mod8-cloudtasks) (2.x) 9|Migrate to Python 3, Cloud Datastore & Cloud Tasks v2| _TBD_ | _TBD_ | Module 8 [code](/mod8-cloudtasks) (2.x) | _TBD_ diff --git a/mod13a-memorystore/README.md b/mod13a-memorystore/README.md index f4447f9..126d96e 100644 --- a/mod13a-memorystore/README.md +++ b/mod13a-memorystore/README.md @@ -1,3 +1,3 @@ -# Module 12 - Add usage of App Engine `memcache` to Flask `ndb` sample app +# Module 13 - Migrate from App Engine `memcache` to Cloud Memorystore -This repo folder is the corresponding Python 2 code to the Module 12 codelab (TBD). The tutorial STARTs with the Python 2 code in the [Module 1 repo folder](/mod1-flask) and leads developers through adding usage of App Engine's `memcache`, culminating in the code in this folder. +This repo folder is the corresponding Python 2 code to the Module 13 codelab (TBD). The tutorial STARTs with the Python 2 code in the [Module 12 repo folder](/mod12-memcache) and leads developers through a migration to Cloud Memorystore, culminating in the code in this (`mod13a-memorystore`) folder. Also included is a migration from App Engine `ndb` to Google Cloud NDB, mirroring the content covered in [Module 2](http://g.co/codelabs/pae-migrate-cloudndb). diff --git a/mod13a-memorystore/main.py b/mod13a-memorystore/main.py index 3384d71..d3fc5de 100644 --- a/mod13a-memorystore/main.py +++ b/mod13a-memorystore/main.py @@ -15,10 +15,11 @@ import os import pickle from flask import Flask, render_template, request -from google.appengine.ext import ndb +from google.cloud import ndb import redis app = Flask(__name__) +ds_client = ndb.Client() HOUR = 3600 REDIS_HOST = os.environ.get('REDIS_HOST', 'localhost') REDIS_PORT = os.environ.get('REDIS_PORT', '6379') @@ -31,12 +32,14 @@ class Visit(ndb.Model): def store_visit(remote_addr, user_agent): 'create new Visit entity in Datastore' - Visit(visitor='{}: {}'.format(remote_addr, user_agent)).put() + with ds_client.context(): + Visit(visitor='{}: {}'.format(remote_addr, user_agent)).put() def fetch_visits(limit): 'get most recent visits' - return (v.to_dict() for v in Visit.query().order( - -Visit.timestamp).fetch(limit)) + with ds_client.context(): + return (v.to_dict() for v in Visit.query().order( + -Visit.timestamp).fetch(limit)) @app.route('/') def root(): @@ -51,6 +54,6 @@ def root(): if not visits or visits[0]['visitor'] != visitor: store_visit(ip_addr, usr_agt) visits = list(fetch_visits(10)) - REDIS.set('visits', pickle.dumps(visits), ex=HOUR) # set() not add() + REDIS.set('visits', pickle.dumps(visits), ex=HOUR) return render_template('index.html', visits=visits) diff --git a/mod13a-memorystore/requirements.txt b/mod13a-memorystore/requirements.txt index 1a5dc97..1bdb2fd 100644 --- a/mod13a-memorystore/requirements.txt +++ b/mod13a-memorystore/requirements.txt @@ -1,2 +1,3 @@ flask redis +google-cloud-ndb diff --git a/mod13b-memorystore/README.md b/mod13b-memorystore/README.md index c66a990..eb622f3 100644 --- a/mod13b-memorystore/README.md +++ b/mod13b-memorystore/README.md @@ -1,5 +1,8 @@ -# Module 12 - Add usage of App Engine `memcache` to Flask `ndb` sample app +# Module 13 - Migrate from App Engine `memcache` to Cloud Memorystore -This repo folder is the corresponding Python 3 code to the Module 12 codelab (TBD). The tutorial STARTs with the Python 2 code in the [Module 1 repo folder](/mod1-flask) and leads developers through adding usage of App Engine's `memcache`, followed by a bonus migration to Python 3, culminating in the code in this folder. +This repo folder is the corresponding Python 3 code to a pair of different migrations related to the Module 12 and 13 codelabs. The tutorial either... -> **LEGACY SERVICES PUBLIC PREVIEW**: Accessing legacy services such as App Engine `ndb` and `memcache` from Python 3 (and next generation App Engine in general) is available in a public preview. See the [Sep 2021 announcement](https://twitter.com/googledevs/status/1445916786755571712) for more information. +- STARTs with the _Python 3_ code in the [Module 12b repo folder](/mod12b-memcache) and leads developers through migrations from App Engine `memcache` to Cloud Memorystore and App Engine `ndb` to Cloud Datastore (skipping Cloud NDB); **or**, +- STARTs with the _Python 2_ code in the [Module 13a repo folder](/mod13a-memorystore) and leads developers through migrations from Cloud NDB to Cloud Datastore, mirroring the content covered in [Module 3](http://g.co/codelabs/pae-migrate-datastore), and also a BONUS migration from Python 2 to 3. + +Either way, both set of migrations culminate in the code in this (`mod13b-memorystore`) folder. diff --git a/mod13b-memorystore/main.py b/mod13b-memorystore/main.py index 6923fae..9455eb2 100644 --- a/mod13b-memorystore/main.py +++ b/mod13b-memorystore/main.py @@ -12,34 +12,34 @@ # See the License for the specific language governing permissions and # limitations under the License. +from datetime import datetime import os import pickle from flask import Flask, render_template, request -from google.cloud import ndb +from google.cloud import datastore import redis app = Flask(__name__) -ds_client = ndb.Client() +ds_client = datastore.Client() HOUR = 3600 REDIS_HOST = os.environ.get('REDIS_HOST', 'localhost') REDIS_PORT = os.environ.get('REDIS_PORT', '6379') REDIS = redis.Redis(host=REDIS_HOST, port=REDIS_PORT) -class Visit(ndb.Model): - 'Visit entity registers visitor IP address & timestamp' - visitor = ndb.StringProperty() - timestamp = ndb.DateTimeProperty(auto_now_add=True) - def store_visit(remote_addr, user_agent): 'create new Visit entity in Datastore' - with ds_client.context(): - Visit(visitor='{}: {}'.format(remote_addr, user_agent)).put() + entity = datastore.Entity(key=ds_client.key('Visit')) + entity.update({ + 'timestamp': datetime.now(), + 'visitor': '{}: {}'.format(remote_addr, user_agent), + }) + ds_client.put(entity) def fetch_visits(limit): 'get most recent visits' - with ds_client.context(): - return (v.to_dict() for v in Visit.query().order( - -Visit.timestamp).fetch(limit)) + query = ds_client.query(kind='Visit') + query.order = ['-timestamp'] + return query.fetch(limit=limit) @app.route('/') def root(): @@ -54,6 +54,6 @@ def root(): if not visits or visits[0]['visitor'] != visitor: store_visit(ip_addr, usr_agt) visits = list(fetch_visits(10)) - REDIS.set('visits', pickle.dumps(visits), ex=HOUR) # set() not add() + REDIS.set('visits', pickle.dumps(visits), ex=HOUR) return render_template('index.html', visits=visits) diff --git a/mod13b-memorystore/requirements.txt b/mod13b-memorystore/requirements.txt index 1bdb2fd..2197d83 100644 --- a/mod13b-memorystore/requirements.txt +++ b/mod13b-memorystore/requirements.txt @@ -1,3 +1,3 @@ flask redis -google-cloud-ndb +google-cloud-datastore diff --git a/mod2a-cloudndb/README.md b/mod2a-cloudndb/README.md index a94baf5..9d63ae1 100644 --- a/mod2a-cloudndb/README.md +++ b/mod2a-cloudndb/README.md @@ -1,3 +1,3 @@ -# Module 2 - Migrate from App Engine `ndb` to Google Cloud NDB +# Module 2 - Migrate from App Engine `ndb` to Cloud NDB -This repo folder is the corresponding Python 2 code to the [Module 2 codelab](http://g.co/codelabs/pae-migrate-cloudndb). The tutorial STARTs with the Python 2 code in the [Module 1 repo folder](/mod1-flask) and leads developers through migrating away from App Engine's `ndb` to Cloud NDB to access Datastore, culminating in the code in this folder. +This repo folder is the corresponding Python 2 code to the [Module 2 codelab](http://g.co/codelabs/pae-migrate-cloudndb). The tutorial STARTs with the Python 2 code in the [Module 1 repo folder](/mod1-flask) and leads developers through migrating away from App Engine's `ndb` to Cloud NDB to access Datastore, culminating in the code in this (`mod2a-cloudndb`) folder. diff --git a/mod2b-cloudndb/README.md b/mod2b-cloudndb/README.md index a67f223..19d56c1 100644 --- a/mod2b-cloudndb/README.md +++ b/mod2b-cloudndb/README.md @@ -1,3 +1,3 @@ -# Module 2 - Migrate from App Engine `ndb` to Google Cloud NDB +# Module 2 - Migrate from App Engine `ndb` to Cloud NDB -This repo folder is the corresponding Python 3 code to the [Module 2 codelab](http://g.co/codelabs/pae-migrate-cloudndb). The tutorial STARTs with the Python 2 code in the [Module 1 repo folder](/mod1-flask) and leads developers through migrating away from App Engine's `ndb` to Cloud NDB to access Datastore culminating in the code in the [mod2a-cloudndb](/mod2a-cloudndb) folder. That is followed by a BONUS migration to Python 3, thus the code in *this* (`mod2b-cloudndb`) folder. +This repo folder is the corresponding Python 3 code to the [Module 2 codelab](http://g.co/codelabs/pae-migrate-cloudndb). The tutorial STARTs with the Python 2 code in the [Module 1 repo folder](/mod1-flask) and leads developers through migrating away from App Engine's `ndb` to Cloud NDB to access Datastore culminating in the code in the [mod2a-cloudndb](/mod2a-cloudndb) folder. That is followed by a BONUS migration to Python 3, culminating in the code in *this* (`mod2b-cloudndb`) folder. diff --git a/mod8-cloudtasks/README.md b/mod8-cloudtasks/README.md index 10c0351..789d37c 100644 --- a/mod8-cloudtasks/README.md +++ b/mod8-cloudtasks/README.md @@ -1,3 +1,3 @@ -# Module 8 - Migrate from App Engine `taskqueue` to Google Cloud Tasks +# Module 8 - Migrate from App Engine `taskqueue` to Cloud Tasks This repo folder is the corresponding Python 2 code to the [Module 8 codelab](http://g.co/codelabs/pae-migrate-cloudtasks). The tutorial STARTs with the Python 2 code in the [Module 7 repo folder](/mod7-gaetasks) and leads developers through migrating from `taskqueue` to Cloud Tasks, culminating in the code in this folder. From 6a9a560cbd51cbdda58a1746efb7d51f7acb3f44 Mon Sep 17 00:00:00 2001 From: wesley chun Date: Fri, 7 Jan 2022 18:14:03 -0800 Subject: [PATCH 23/56] update links & switch mod13b to Cloud NDB --- README.md | 21 ++++++++++----------- mod13b-memorystore/main.py | 24 ++++++++++++------------ mod13b-memorystore/requirements.txt | 2 +- mod9-py3fstasks/main.py | 19 ++++++++++--------- 4 files changed, 33 insertions(+), 33 deletions(-) diff --git a/README.md b/README.md index 62e25a3..13927aa 100644 --- a/README.md +++ b/README.md @@ -88,8 +88,8 @@ Module | Topic | Video | Codelab | START here | FINISH here 8|Migrate to Cloud Tasks| [link](https://twitter.com/googledevs/status/1450960021018267656?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrcloudtasks_sms_201112&utm_content=-) | [link](http://g.co/codelabs/pae-migrate-cloudtasks) | Module 7 [code](/mod7-gaetasks) (2.x) | Module 8 [code](/mod8-cloudtasks) (2.x) 9|Migrate to Python 3, Cloud Datastore & Cloud Tasks v2| _TBD_ | _TBD_ | Module 8 [code](/mod8-cloudtasks) (2.x) | _TBD_ 10|Migrate Datastore/Firestore data to another project| _TBD_ | _N/A_ | _N/A_ | _TBD_ -11|Migrate to Cloud Functions| _TBD_ | _TBD_ | Module 2 [code](/mod2b-cloudndb) (3.x) | Module 11 [code](/mod11-functions) (3.x) -12|Add App Engine `memcache`| _TBD_ | _TBD_ | Module 1 [code](/mod1-flask) (2.x) | Module 12 [code](/mod12-memcache) (2.x) & [code](/mod12b-memcache) (3.x) +11|Migrate to Cloud Functions| _TBD_ | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-11-functions?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrcloudfuncs_sms_202006&utm_content=-) | Module 2 [code](/mod2b-cloudndb) (3.x) | Module 11 [code](/mod11-functions) (3.x) +12|Add App Engine `memcache`| _TBD_ | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-12-memcache?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrcloudfuncs_sms_202006&utm_content=-) | Module 1 [code](/mod1-flask) (2.x) | Module 12 [code](/mod12-memcache) (2.x) & [code](/mod12b-memcache) (3.x) 13|Migrate to Cloud Memorystore| _TBD_ | _TBD_ | Module 12 [code](/mod12-memcache) (2.x) & [code](/mod12b-memcache) (3.x) | Module 13 [code](/mod13a-memorystore) (2.x) & [code](/mod13b-memorystore) (3.x) @@ -98,7 +98,7 @@ Module | Topic | Video | Codelab | START here | FINISH here If there is a logical codelab to do immediately after completing one, they will be designated as NEXT. Other recommended codelabs will be listed as RECOMMENDED, and the more optional ones will be labeled as OTHERS (and usually in some kind of priority order). -- [Module 1 codelab](http://g.co/codelabs/pae-migrate-flask): **Migrate from `webapp2` to [Flask](https://flask.palletsprojects.com)** +- [Module 1 codelab](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-1-flask?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrwa2flsk_201008&utm_content=-): **Migrate from `webapp2` to [Flask](https://flask.palletsprojects.com)** - **Required** migration (can also pick your own framework) - `webapp2` does not do routing thus unsupported by App Engine (even though a [3.x port exists](https://github.com/fili/webapp2-gae-python37)) - Python 2 only @@ -107,8 +107,7 @@ If there is a logical codelab to do immediately after completing one, they will - NEXT: - Module 2 - migrate to Cloud NDB - -- [Module 2 codelab](http://g.co/codelabs/pae-migrate-cloudndb): **Migrate from App Engine `ndb` to [Cloud NDB](https://googleapis.dev/python/python-ndb/latest)** +- [Module 2 codelab](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-2-cloudndb?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrcloudndb_201021&utm_content=-): **Migrate from App Engine `ndb` to [Cloud NDB](https://googleapis.dev/python/python-ndb/latest)** - **Required** migration - Migration to Cloud NDB which is supported by Python 3 and the next-gen platform. - Python 2 @@ -124,14 +123,14 @@ If there is a logical codelab to do immediately after completing one, they will - Module 4 - migrate to Cloud Run container with Docker - Module 3 - migrate to Cloud Datastore -- [Module 7 codelab](http://g.co/codelabs/pae-migrate-gaetasks): **Add App Engine (push) Task Queues to App Engine `ndb` Flask app** +- [Module 7 codelab](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-7-gaetasks?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrgaetasks_sms_201028&utm_content=-): **Add App Engine (push) Task Queues to App Engine `ndb` Flask app** - **Not a migration**: add GAE Task Queues to prepare for migration to Cloud Tasks - Python 2 - START: [Module 1 code - Framework](/mod1-flask) (2.x) - FINISH: [Module 7 code - GAE Task Queues](/mod7-gaetasks) (2.x) - NEXT: Module 8 - migrate App Engine push tasks to Cloud Tasks -- [Module 8 codelab](http://g.co/codelabs/pae-migrate-cloudtasks): **Migrate from App Engine (push) Task Queues to [Cloud Tasks](http://cloud.google.com/tasks) v1** +- [Module 8 codelab](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-8-cloudtasks?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrcloudtasks_sms_201112&utm_content=-): **Migrate from App Engine (push) Task Queues to [Cloud Tasks](http://cloud.google.com/tasks) v1** - **Required** migration - Migration to Cloud Tasks which is supported by Python 3 and the next-gen platform. - Note this is only push tasks... pull tasks will be handled in a different codelab. @@ -153,7 +152,7 @@ If there is a logical codelab to do immediately after completing one, they will - Module 5 - migrate to Cloud Run container with Cloud Buildpacks - Module 4 - migrate to Cloud Run container with Docker -- [Module 4 codelab](http://g.co/codelabs/pae-migrate-rundocker): **Migrate from App Engine to Cloud Run with Docker** +- [Module 4 codelab](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-4-rundocker?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrcrdckr_sms_201017&utm_content=-): **Migrate from App Engine to Cloud Run with Docker** - **Optional** migration - "Containerize" your app (migrate your app to a container) with Docker - Python 2 @@ -168,7 +167,7 @@ If there is a logical codelab to do immediately after completing one, they will - Module 7 - add App Engine (push) tasks - Module 11 - migrate to Cloud Functions -- [Module 5 codelab](http://g.co/codelabs/pae-migrate-runbldpks): **Migrate from App Engine to Cloud Run with Cloud Buildpacks** +- [Module 5 codelab](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-5-runbldpks?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrcrbdpk_sms_201031&utm_content=-): **Migrate from App Engine to Cloud Run with Cloud Buildpacks** - **Optional** migration - "Containerize" your app (migrate your app to a container) with... - [Cloud Buildpacks]() which lets you containerize your app without `Dockerfile`s @@ -181,7 +180,7 @@ If there is a logical codelab to do immediately after completing one, they will - Module 7 - add App Engine (push) tasks - Module 11 - migrate to Cloud Functions -- [Module 3 codelab](http://g.co/codelabs/pae-migrate-datastore): **Migrate from Cloud NDB to [Cloud Datastore](http://cloud.google.com/datastore)** +- [Module 3 codelab](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-3-datastore?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrcloudds_201003&utm_content=-): **Migrate from Cloud NDB to [Cloud Datastore](http://cloud.google.com/datastore)** - **Optional** migration - Recommended only if using Cloud Datastore elsewhere (GAE *or* non-App Engine) apps - Helps w/code consistency & reusability, reduces maintenance costs @@ -198,7 +197,7 @@ If there is a logical codelab to do immediately after completing one, they will - Module 5 - migrate to Cloud Run container with Cloud Buildpacks - Module 4 - migrate to Cloud Run container with Docker -- **Module 11 codelab** (TBD): **Migrate from App Engine to Cloud Functions** +- [Module 11 codelab](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-11-functions?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrcloudfuncs_sms_202006&utm_content=-): **Migrate from App Engine to Cloud Functions** - **Optional** migration - Recommende for small apps or for breaking up large apps into multiple microservices - Python 3 only diff --git a/mod13b-memorystore/main.py b/mod13b-memorystore/main.py index 9455eb2..d3fc5de 100644 --- a/mod13b-memorystore/main.py +++ b/mod13b-memorystore/main.py @@ -12,34 +12,34 @@ # See the License for the specific language governing permissions and # limitations under the License. -from datetime import datetime import os import pickle from flask import Flask, render_template, request -from google.cloud import datastore +from google.cloud import ndb import redis app = Flask(__name__) -ds_client = datastore.Client() +ds_client = ndb.Client() HOUR = 3600 REDIS_HOST = os.environ.get('REDIS_HOST', 'localhost') REDIS_PORT = os.environ.get('REDIS_PORT', '6379') REDIS = redis.Redis(host=REDIS_HOST, port=REDIS_PORT) +class Visit(ndb.Model): + 'Visit entity registers visitor IP address & timestamp' + visitor = ndb.StringProperty() + timestamp = ndb.DateTimeProperty(auto_now_add=True) + def store_visit(remote_addr, user_agent): 'create new Visit entity in Datastore' - entity = datastore.Entity(key=ds_client.key('Visit')) - entity.update({ - 'timestamp': datetime.now(), - 'visitor': '{}: {}'.format(remote_addr, user_agent), - }) - ds_client.put(entity) + with ds_client.context(): + Visit(visitor='{}: {}'.format(remote_addr, user_agent)).put() def fetch_visits(limit): 'get most recent visits' - query = ds_client.query(kind='Visit') - query.order = ['-timestamp'] - return query.fetch(limit=limit) + with ds_client.context(): + return (v.to_dict() for v in Visit.query().order( + -Visit.timestamp).fetch(limit)) @app.route('/') def root(): diff --git a/mod13b-memorystore/requirements.txt b/mod13b-memorystore/requirements.txt index 2197d83..1bdb2fd 100644 --- a/mod13b-memorystore/requirements.txt +++ b/mod13b-memorystore/requirements.txt @@ -1,3 +1,3 @@ flask redis -google-cloud-datastore +google-cloud-ndb diff --git a/mod9-py3fstasks/main.py b/mod9-py3fstasks/main.py index 0f7b12c..9a7ce95 100644 --- a/mod9-py3fstasks/main.py +++ b/mod9-py3fstasks/main.py @@ -17,10 +17,10 @@ import time from flask import Flask, render_template, request import google.auth -from google.cloud import firestore, tasks +from google.cloud import datastore, tasks app = Flask(__name__) -fs_client = firestore.Client() +ds_client = datastore.Client() ts_client = tasks.CloudTasksClient() _, PROJECT_ID = google.auth.default() @@ -30,12 +30,13 @@ PATH_PREFIX = QUEUE_PATH.rsplit('/', 2)[0] def store_visit(remote_addr, user_agent): - 'create new Visit document in Firestore' - doc_ref = fs_client.collection('Visit') - doc_ref.add({ + 'create new Visit entity in Datastore' + entity = datastore.Entity(key=ds_client.key('Visit')) + entity.update({ 'timestamp': datetime.now(), 'visitor': '{}: {}'.format(remote_addr, user_agent), }) + ds_client.put(entity) def _create_queue_if(): 'app-internal function creating default queue if it does not exist' @@ -49,9 +50,9 @@ def _create_queue_if(): def fetch_visits(limit): 'get most recent visits & add task to delete older visits' - visits_ref = fs_client.collection('Visit') - visits = list(v.to_dict() for v in visits_ref.order_by('timestamp', - direction=firestore.Query.DESCENDING).limit(limit).stream()) + query = ds_client.query(kind='Visit') + query.order = ['-timestamp'] + visits = query.fetch(limit=limit) oldest = time.mktime(visits[-1]['timestamp'].timetuple()) oldest_str = time.ctime(oldest) print('Delete entities older than %s' % oldest_str) @@ -78,7 +79,7 @@ def _delete_docs(visits): def trim(): '(push) task queue handler to delete oldest visits' oldest = float(request.get_json().get('oldest')) - query = fs_client.collection('Visit') + query = ds_client.collection('Visit') visits = query.where('timestamp', '<', datetime.fromtimestamp(oldest)).stream() dlist = ', '.join(str(v_id) for v_id in _delete_docs(visits)) From 21d54859621b666616b9d2d9ae3f2fc657f5a316 Mon Sep 17 00:00:00 2001 From: wesley chun Date: Tue, 18 Jan 2022 19:23:34 -0800 Subject: [PATCH 24/56] optimize queries, mod9 switch to DS --- README.md | 6 +++--- mod0-baseline/main.py | 3 +-- mod1-flask/main.py | 3 +-- mod11-functions/main.py | 3 +-- mod12-memcache/main.py | 5 ++--- mod12b-memcache/main.py | 5 ++--- mod13a-memorystore/app.yaml | 6 ++++++ mod13a-memorystore/appengine_config.py | 3 +++ mod13a-memorystore/main.py | 5 ++--- mod13b-memorystore/main.py | 5 ++--- mod1b-flask/main.py | 3 +-- mod2a-cloudndb/main.py | 3 +-- mod2b-cloudndb/main.py | 3 +-- mod2b-cloudndb/requirements.txt | 2 +- mod4a-rundocker/main.py | 3 +-- mod4a-rundocker/requirements.txt | 2 +- mod5-runbldpks/main.py | 3 +-- mod5-runbldpks/requirements.txt | 2 +- mod7-gaetasks/main.py | 2 +- mod7b-gaetasks/main.py | 2 +- mod8-cloudtasks/main.py | 2 +- mod9-py3fstasks/main.py | 23 ++++++++++------------- mod9-py3fstasks/requirements.txt | 4 ++-- 23 files changed, 46 insertions(+), 52 deletions(-) diff --git a/README.md b/README.md index 13927aa..7ebd2da 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,7 @@ # Modernizing Google Cloud serverless compute applications ### To the latest Cloud services and serverless platforms -This is the corresponding repository to the [Serverless Migration Station](https://bit.ly/3xk2Swi) video series whose goal is to help users on a Google Cloud serverless compute platform modernize to newer Cloud products or other serverless compute platforms. Each modernization migration aims to feature a video, codelab (self-paced, hands-on tutorial), and code samples. The content initially focuses on App Engine and Google's earliest Cloud users. +This is the corresponding repository to the [Serverless Migration Station](https://bit.ly/3xk2Swi) video series whose goal is to help users on a Google Cloud serverless compute platform modernize to newer Cloud products or other serverless compute platforms. Each modernization migration aims to feature a video, codelab (self-paced, hands-on tutorial), and code samples. The content initially focuses on App Engine and Google's earliest Cloud users. Read more about the [codelabs in this announcement](https://developers.googleblog.com/2021/03/modernizing-your-google-app-engine-applications.html?utm_source=ext&utm_medium=partner&utm_campaign=CDR_wes_aap-serverless_modernizegae_codelabsannounce_201031&utm_content=-) as well as [this one introducing the video series](https://developers.googleblog.com/2021/06/introducing-serverless-migration.html?utm_source=blog&utm_medium=partner&utm_campaign=CDR_wes_aap-serverless_smsintro_201023). [Google App Engine](https://cloud.google.com/appengine) (Standard) has undergone significant changes between the legacy and next generation platforms. To address this, we've created a set of codelabs (free, online, self-paced, hands-on tutorials) and corresponding videos (when available) to show developers how to perform individual migrations they can apply to modernize their apps for the latest runtimes, with this repo managing the samples from those codelabs. Codelab content typically falls into one of these three topics: @@ -142,7 +142,7 @@ If there is a logical codelab to do immediately after completing one, they will - **Module 9 codelab** (TBD): **Migrate a Python 2 Cloud NDB & Cloud Tasks (v1) app to a Python 3 Cloud Datastore & Cloud Tasks (v2) app** - **Optional** migrations - Migrating to Python 3 is not required but recommended as Python 2 has been sunset - - Migrating to Cloud Firestore is *very* optional as Cloud NDB works on 3.x and most importantly, Cloud Firestore requires a completely new GCP project + - Migrating to Cloud Datastore is optional as Cloud NDB works on 3.x - Python 2 - START: [Module 8 code - Cloud Tasks](/mod8-cloudtasks) (2.x) - Python 3 @@ -202,7 +202,7 @@ If there is a logical codelab to do immediately after completing one, they will - Recommende for small apps or for breaking up large apps into multiple microservices - Python 3 only - START: [Module 2 code - Cloud NDB](/mod2b-cloudndb) (3.x) - - FINISH: [Module 11 code - Cloud Firestore](/mod11-functions) (3.x) + - FINISH: [Module 11 code - Cloud Functions](/mod11-functions) (3.x) - RECOMMENDED: - Module 7 - add App Engine (push) tasks - OTHER OPTIONS (in somewhat priority order): diff --git a/mod0-baseline/main.py b/mod0-baseline/main.py index 29ef681..42afde4 100644 --- a/mod0-baseline/main.py +++ b/mod0-baseline/main.py @@ -28,8 +28,7 @@ def store_visit(remote_addr, user_agent): def fetch_visits(limit): 'get most recent visits' - return (v.to_dict() for v in Visit.query().order( - -Visit.timestamp).fetch(limit)) + return Visit.query().order(-Visit.timestamp).fetch(limit) class MainHandler(webapp2.RequestHandler): 'main application (GET) handler' diff --git a/mod1-flask/main.py b/mod1-flask/main.py index aac3082..14f5f89 100644 --- a/mod1-flask/main.py +++ b/mod1-flask/main.py @@ -28,8 +28,7 @@ def store_visit(remote_addr, user_agent): def fetch_visits(limit): 'get most recent visits' - return (v.to_dict() for v in Visit.query().order( - -Visit.timestamp).fetch(limit)) + return Visit.query().order(-Visit.timestamp).fetch(limit) @app.route('/') def root(): diff --git a/mod11-functions/main.py b/mod11-functions/main.py index 6c0acc6..a71655f 100644 --- a/mod11-functions/main.py +++ b/mod11-functions/main.py @@ -30,8 +30,7 @@ def store_visit(remote_addr, user_agent): def fetch_visits(limit): 'get most recent visits' with ds_client.context(): - return (v.to_dict() for v in Visit.query().order( - -Visit.timestamp).fetch(limit)) + return Visit.query().order(-Visit.timestamp).fetch(limit) def visitme(request): 'main application (GET) handler' diff --git a/mod12-memcache/main.py b/mod12-memcache/main.py index 0278ec9..d183b12 100644 --- a/mod12-memcache/main.py +++ b/mod12-memcache/main.py @@ -30,8 +30,7 @@ def store_visit(remote_addr, user_agent): def fetch_visits(limit): 'get most recent visits' - return (v.to_dict() for v in Visit.query().order( - -Visit.timestamp).fetch(limit)) + return Visit.query().order(-Visit.timestamp).fetch(limit) @app.route('/') def root(): @@ -42,7 +41,7 @@ def root(): visits = memcache.get('visits') # register visit & run DB query if cache empty or new visitor - if not visits or visits[0]['visitor'] != visitor: + if not visits or visits[0].visitor != visitor: store_visit(ip_addr, usr_agt) visits = list(fetch_visits(10)) memcache.set('visits', visits, HOUR) # set() not add() diff --git a/mod12b-memcache/main.py b/mod12b-memcache/main.py index 8ae9d7d..ae7924b 100644 --- a/mod12b-memcache/main.py +++ b/mod12b-memcache/main.py @@ -31,8 +31,7 @@ def store_visit(remote_addr, user_agent): def fetch_visits(limit): 'get most recent visits' - return (v.to_dict() for v in Visit.query().order( - -Visit.timestamp).fetch(limit)) + return Visit.query().order(-Visit.timestamp).fetch(limit) @app.route('/') def root(): @@ -43,7 +42,7 @@ def root(): visits = memcache.get('visits') # register visit & run DB query if cache empty or new visitor - if not visits or visits[0]['visitor'] != visitor: + if not visits or visits[0].visitor != visitor: store_visit(ip_addr, usr_agt) visits = list(fetch_visits(10)) memcache.set('visits', visits, HOUR) # set() not add() diff --git a/mod13a-memorystore/app.yaml b/mod13a-memorystore/app.yaml index c683da5..db67a62 100644 --- a/mod13a-memorystore/app.yaml +++ b/mod13a-memorystore/app.yaml @@ -20,6 +20,12 @@ handlers: - url: /.* script: main.app +libraries: +- name: grpcio + version: 1.0.0 +- name: setuptools + version: 36.6.0 + env_variables: REDIS_HOST: 'YOUR_REDIS_HOST' REDIS_PORT: 'YOUR_REDIS_PORT' diff --git a/mod13a-memorystore/appengine_config.py b/mod13a-memorystore/appengine_config.py index 9760ebb..2a41fb4 100644 --- a/mod13a-memorystore/appengine_config.py +++ b/mod13a-memorystore/appengine_config.py @@ -12,9 +12,12 @@ # See the License for the specific language governing permissions and # limitations under the License. +import pkg_resources from google.appengine.ext import vendor # Set PATH to your libraries folder. PATH = 'lib' # Add libraries installed in the PATH folder. vendor.add(PATH) +# Add libraries to pkg_resources working set to find the distribution. +pkg_resources.working_set.add_entry(PATH) diff --git a/mod13a-memorystore/main.py b/mod13a-memorystore/main.py index d3fc5de..086c286 100644 --- a/mod13a-memorystore/main.py +++ b/mod13a-memorystore/main.py @@ -38,8 +38,7 @@ def store_visit(remote_addr, user_agent): def fetch_visits(limit): 'get most recent visits' with ds_client.context(): - return (v.to_dict() for v in Visit.query().order( - -Visit.timestamp).fetch(limit)) + return Visit.query().order(-Visit.timestamp).fetch(limit) @app.route('/') def root(): @@ -51,7 +50,7 @@ def root(): visits = pickle.loads(rsp) if rsp else None # register visit & run DB query if cache empty or new visitor - if not visits or visits[0]['visitor'] != visitor: + if not visits or visits[0].visitor != visitor: store_visit(ip_addr, usr_agt) visits = list(fetch_visits(10)) REDIS.set('visits', pickle.dumps(visits), ex=HOUR) diff --git a/mod13b-memorystore/main.py b/mod13b-memorystore/main.py index d3fc5de..086c286 100644 --- a/mod13b-memorystore/main.py +++ b/mod13b-memorystore/main.py @@ -38,8 +38,7 @@ def store_visit(remote_addr, user_agent): def fetch_visits(limit): 'get most recent visits' with ds_client.context(): - return (v.to_dict() for v in Visit.query().order( - -Visit.timestamp).fetch(limit)) + return Visit.query().order(-Visit.timestamp).fetch(limit) @app.route('/') def root(): @@ -51,7 +50,7 @@ def root(): visits = pickle.loads(rsp) if rsp else None # register visit & run DB query if cache empty or new visitor - if not visits or visits[0]['visitor'] != visitor: + if not visits or visits[0].visitor != visitor: store_visit(ip_addr, usr_agt) visits = list(fetch_visits(10)) REDIS.set('visits', pickle.dumps(visits), ex=HOUR) diff --git a/mod1b-flask/main.py b/mod1b-flask/main.py index bfc4c80..e3880c7 100644 --- a/mod1b-flask/main.py +++ b/mod1b-flask/main.py @@ -30,8 +30,7 @@ def store_visit(remote_addr, user_agent): def fetch_visits(limit): 'get most recent visits' - return (v.to_dict() for v in Visit.query().order( - -Visit.timestamp).fetch(limit)) + return Visit.query().order(-Visit.timestamp).fetch(limit) @app.route('/') def root(): diff --git a/mod2a-cloudndb/main.py b/mod2a-cloudndb/main.py index 4c3015a..36834ef 100644 --- a/mod2a-cloudndb/main.py +++ b/mod2a-cloudndb/main.py @@ -31,8 +31,7 @@ def store_visit(remote_addr, user_agent): def fetch_visits(limit): 'get most recent visits' with ds_client.context(): - return (v.to_dict() for v in Visit.query().order( - -Visit.timestamp).fetch(limit)) + return Visit.query().order(-Visit.timestamp).fetch(limit) @app.route('/') def root(): diff --git a/mod2b-cloudndb/main.py b/mod2b-cloudndb/main.py index 4c3015a..36834ef 100644 --- a/mod2b-cloudndb/main.py +++ b/mod2b-cloudndb/main.py @@ -31,8 +31,7 @@ def store_visit(remote_addr, user_agent): def fetch_visits(limit): 'get most recent visits' with ds_client.context(): - return (v.to_dict() for v in Visit.query().order( - -Visit.timestamp).fetch(limit)) + return Visit.query().order(-Visit.timestamp).fetch(limit) @app.route('/') def root(): diff --git a/mod2b-cloudndb/requirements.txt b/mod2b-cloudndb/requirements.txt index 81ef820..84932a7 100644 --- a/mod2b-cloudndb/requirements.txt +++ b/mod2b-cloudndb/requirements.txt @@ -1,2 +1,2 @@ flask -google-cloud-ndb==1.11.1 +google-cloud-ndb diff --git a/mod4a-rundocker/main.py b/mod4a-rundocker/main.py index 4c3015a..36834ef 100644 --- a/mod4a-rundocker/main.py +++ b/mod4a-rundocker/main.py @@ -31,8 +31,7 @@ def store_visit(remote_addr, user_agent): def fetch_visits(limit): 'get most recent visits' with ds_client.context(): - return (v.to_dict() for v in Visit.query().order( - -Visit.timestamp).fetch(limit)) + return Visit.query().order(-Visit.timestamp).fetch(limit) @app.route('/') def root(): diff --git a/mod4a-rundocker/requirements.txt b/mod4a-rundocker/requirements.txt index ba543a5..7890c49 100644 --- a/mod4a-rundocker/requirements.txt +++ b/mod4a-rundocker/requirements.txt @@ -1,3 +1,3 @@ -gunicorn==19.10.0 +gunicorn flask google-cloud-ndb==1.11.1 diff --git a/mod5-runbldpks/main.py b/mod5-runbldpks/main.py index 4c3015a..36834ef 100644 --- a/mod5-runbldpks/main.py +++ b/mod5-runbldpks/main.py @@ -31,8 +31,7 @@ def store_visit(remote_addr, user_agent): def fetch_visits(limit): 'get most recent visits' with ds_client.context(): - return (v.to_dict() for v in Visit.query().order( - -Visit.timestamp).fetch(limit)) + return Visit.query().order(-Visit.timestamp).fetch(limit) @app.route('/') def root(): diff --git a/mod5-runbldpks/requirements.txt b/mod5-runbldpks/requirements.txt index e612636..60bbd95 100644 --- a/mod5-runbldpks/requirements.txt +++ b/mod5-runbldpks/requirements.txt @@ -1,3 +1,3 @@ -gunicor +gunicorn flask google-cloud-ndb diff --git a/mod7-gaetasks/main.py b/mod7-gaetasks/main.py index 49e2b08..b329961 100644 --- a/mod7-gaetasks/main.py +++ b/mod7-gaetasks/main.py @@ -37,7 +37,7 @@ def fetch_visits(limit): oldest_str = time.ctime(oldest) logging.info('Delete entities older than %s' % oldest_str) taskqueue.add(url='/trim', params={'oldest': oldest}) - return (v.to_dict() for v in data), oldest_str + return data, oldest_str @app.route('/trim', methods=['POST']) def trim(): diff --git a/mod7b-gaetasks/main.py b/mod7b-gaetasks/main.py index 35a1188..af356de 100644 --- a/mod7b-gaetasks/main.py +++ b/mod7b-gaetasks/main.py @@ -38,7 +38,7 @@ def fetch_visits(limit): oldest_str = time.ctime(oldest) logging.info('Delete entities older than %s' % oldest_str) taskqueue.add(url='/trim', params={'oldest': oldest}) - return (v.to_dict() for v in data), oldest_str + return data, oldest_str @app.route('/trim', methods=['POST']) def trim(): diff --git a/mod8-cloudtasks/main.py b/mod8-cloudtasks/main.py index 10a1085..15361e9 100644 --- a/mod8-cloudtasks/main.py +++ b/mod8-cloudtasks/main.py @@ -56,7 +56,7 @@ def fetch_visits(limit): } } ts_client.create_task(parent=QUEUE_PATH, task=task) - return (v.to_dict() for v in data), oldest_str + return data, oldest_str @app.route('/trim', methods=['POST']) def trim(): diff --git a/mod9-py3fstasks/main.py b/mod9-py3fstasks/main.py index 9a7ce95..f444830 100644 --- a/mod9-py3fstasks/main.py +++ b/mod9-py3fstasks/main.py @@ -52,7 +52,7 @@ def fetch_visits(limit): 'get most recent visits & add task to delete older visits' query = ds_client.query(kind='Visit') query.order = ['-timestamp'] - visits = query.fetch(limit=limit) + visits = list(query.fetch(limit=limit)) oldest = time.mktime(visits[-1]['timestamp'].timetuple()) oldest_str = time.ctime(oldest) print('Delete entities older than %s' % oldest_str) @@ -69,22 +69,19 @@ def fetch_visits(limit): ts_client.create_task(parent=QUEUE_PATH, task=task) return visits, oldest_str -def _delete_docs(visits): - 'app-internal generator deleting old FS visit documents' - for visit in visits: - visit.reference.delete() - yield visit.id - @app.route('/trim', methods=['POST']) def trim(): '(push) task queue handler to delete oldest visits' oldest = float(request.get_json().get('oldest')) - query = ds_client.collection('Visit') - visits = query.where('timestamp', '<', - datetime.fromtimestamp(oldest)).stream() - dlist = ', '.join(str(v_id) for v_id in _delete_docs(visits)) - if dlist: - print('Deleting %d entities: %s' % (dlist.count(',')+1, dlist)) + query = ds_client.query(kind='Visit') + query.add_filter('timestamp', '<', datetime.fromtimestamp(oldest)) + query.keys_only() + keys = list(visit.key for visit in query.fetch()) + nkeys = len(keys) + if nkeys: + print('Deleting %d entities: %s' % ( + nkeys, ', '.join(str(k.id) for k in keys))) + ds_client.delete_multi(keys) else: print('No entities older than: %s' % time.ctime(oldest)) return '' # need to return SOME string w/200 diff --git a/mod9-py3fstasks/requirements.txt b/mod9-py3fstasks/requirements.txt index 9f80c69..be6794c 100644 --- a/mod9-py3fstasks/requirements.txt +++ b/mod9-py3fstasks/requirements.txt @@ -1,3 +1,3 @@ flask -google-cloud-firestore==2.3.4 -google-cloud-tasks==2.7.1 +google-cloud-datastore +google-cloud-tasks From 47a42bed5d85edb4cc636dbc016941e3ba4b8bd6 Mon Sep 17 00:00:00 2001 From: wesley chun Date: Tue, 18 Jan 2022 19:35:25 -0800 Subject: [PATCH 25/56] mod9 switch to DS --- .../.gcloudignore | 0 mod9-py3dstasks/README.md | 11 ++++++ {mod9-py3fstasks => mod9-py3dstasks}/app.yaml | 0 {mod9-py3fstasks => mod9-py3dstasks}/main.py | 0 .../requirements.txt | 0 .../templates/index.html | 0 mod9-py3fstasks/README.md | 34 ------------------- 7 files changed, 11 insertions(+), 34 deletions(-) rename {mod9-py3fstasks => mod9-py3dstasks}/.gcloudignore (100%) create mode 100644 mod9-py3dstasks/README.md rename {mod9-py3fstasks => mod9-py3dstasks}/app.yaml (100%) rename {mod9-py3fstasks => mod9-py3dstasks}/main.py (100%) rename {mod9-py3fstasks => mod9-py3dstasks}/requirements.txt (100%) rename {mod9-py3fstasks => mod9-py3dstasks}/templates/index.html (100%) delete mode 100644 mod9-py3fstasks/README.md diff --git a/mod9-py3fstasks/.gcloudignore b/mod9-py3dstasks/.gcloudignore similarity index 100% rename from mod9-py3fstasks/.gcloudignore rename to mod9-py3dstasks/.gcloudignore diff --git a/mod9-py3dstasks/README.md b/mod9-py3dstasks/README.md new file mode 100644 index 0000000..92aa7f0 --- /dev/null +++ b/mod9-py3dstasks/README.md @@ -0,0 +1,11 @@ +# Module 9 - Migrate from Python 2 to 3 and Cloud NDB to Cloud Datastore + +This repo folder is the corresponding Python 3 code to the [Module 9 codelab](http://g.co/codelabs/pae-migrate-py3dstasks). The tutorial STARTs with the Python 2 code in the [Module 8 repo folder](/mod7-cloudtasks) and leads developers through migrating from Python 2 to 3, Cloud NDB to Cloud Datastore plus any changes from Cloud Tasks v1 to v2, culminating in the code in this folder. One major addition to look for here vs. Module 8 is that App Engine `taskqueue` creates a `default` push queue while Cloud Tasks does not, so that now has to be done in code. + +**NOTE: Backport to Python 2**: When migrating this app to Python 3, we added a Python 3 dependency: the `print()` function. If for any reason you need to get back on Python 2 App Engine, you would have to: + + 1. Decide on your logging strategy. The Python 2 App Engine runtime now allows writing to `stdout`, so you don't have to revert back to `logging.info()` (or preferred logging level), however writing to `stdout` defaults to `logging.error()`. If that is acceptable and to continue with `print()`, add this import (above all others) at top of `main.py`: + + from __future__ import print_function + + 2. Revert back to your Python 2 configuration files. For this app, it would be the Module 8 [`app.yaml`](/blob/master/mod8-cloudtasks/app.yaml) and [`appengine_config.py`](/blob/master/mod8-cloudtasks/appengine_config.py) files. diff --git a/mod9-py3fstasks/app.yaml b/mod9-py3dstasks/app.yaml similarity index 100% rename from mod9-py3fstasks/app.yaml rename to mod9-py3dstasks/app.yaml diff --git a/mod9-py3fstasks/main.py b/mod9-py3dstasks/main.py similarity index 100% rename from mod9-py3fstasks/main.py rename to mod9-py3dstasks/main.py diff --git a/mod9-py3fstasks/requirements.txt b/mod9-py3dstasks/requirements.txt similarity index 100% rename from mod9-py3fstasks/requirements.txt rename to mod9-py3dstasks/requirements.txt diff --git a/mod9-py3fstasks/templates/index.html b/mod9-py3dstasks/templates/index.html similarity index 100% rename from mod9-py3fstasks/templates/index.html rename to mod9-py3dstasks/templates/index.html diff --git a/mod9-py3fstasks/README.md b/mod9-py3fstasks/README.md deleted file mode 100644 index cbbd97f..0000000 --- a/mod9-py3fstasks/README.md +++ /dev/null @@ -1,34 +0,0 @@ -# Module 9 - Migrate from Python 2 to 3 and Cloud NDB to Cloud Firestore - -This repo folder is the corresponding Python 3 code to the [Module 9 codelab](http://g.co/codelabs/pae-migrate-py3fstasks). The tutorial STARTs with the Python 2 code in the [Module 8 repo folder](/mod7-cloudtasks) and leads developers through migrating from Python 2 to 3, Cloud NDB to Cloud Firestore (skipping over a Cloud Datstore migration) plus any changes from Cloud Tasks v1 to v2, culminating in the code in this folder. One major addition to look for here vs. Module 8 is that App Engine `taskqueue` creates a `default` push queue while Cloud Tasks does not, so that now has to be done in code. - -**NOTE: Batch delete**: The deletion process in this app is "one-at-a-time." If your app requires deletion of more than a few documents, consider switching to the batch model. In this case, you would replace `_delete_docs()` with: - - def _delete_docs(visits): - 'app-internal generator deleting old FS visit documents' - batch = fs_client.batch() - for visit in visits: - batch.delete(visit.reference) - yield visit.id - batch.commit() - -**NOTE: Backport to Python 2**: When migrating this app to Python 3, we added a Python 3 dependency: the `print()` function. If for any reason you need to get back on Python 2 App Engine, you would have to: - - 1. Decide on your logging strategy. The Python 2 App Engine runtime now allows writing to `stdout`, so you don't have to revert back to `logging.info()` (or preferred logging level), however writing to `stdout` defaults to `logging.error()`. If that is acceptable and to continue with `print()`, add a `__future__.print_function` import above the others so the top of `main.py` looks like this: - - from __future__ import print_function - from datetime import datetime - import json - import time - from flask import Flask, render_template, request - import google.auth - from google.cloud import firestore, tasks - - 2. Revert back to your Python 2 configuration files. For this app, it would be Module 8's [`app.yaml`](https://github.com/googlecodelabs/migrate-python2-appengine/blob/master/mod8-cloudtasks/app.yaml) and [`appengine_config.py`](https://github.com/googlecodelabs/migrate-python2-appengine/blob/master/mod8-cloudtasks/appengine_config.py) files. - - 3. Revert all package versions from `requirements.txt` to get the latest/last(?) package versions for Python 2 as well as run `pip install -t lib -r requirements.txt` again. Here is what ours looks like: - - flask==1.1.2 - google-cloud-firestore==1.9.0 - google-cloud-tasks==1.5.0 - From dafd8a0fdf95af1545b25ce9be5b26fbd0120b29 Mon Sep 17 00:00:00 2001 From: wesley chun Date: Wed, 26 Jan 2022 18:39:00 -0800 Subject: [PATCH 26/56] README updates; add pending move --- README.md | 25 ++++++++++++++++++++++--- mod13b-memorystore/README.md | 11 ++++++----- mod9-py3dstasks/README.md | 2 +- 3 files changed, 29 insertions(+), 9 deletions(-) diff --git a/README.md b/README.md index 7ebd2da..5d58960 100644 --- a/README.md +++ b/README.md @@ -1,3 +1,8 @@ +| :boom: ALERT!! | +|:---------------------------| +| This repo will soon be relocating to [GoogleCloudPlatform](https://github.com/GoogleCloudPlatform) as we better organize these code samples! Stay tuned as more info is coming soon. | + + # Modernizing Google Cloud serverless compute applications ### To the latest Cloud services and serverless platforms @@ -57,7 +62,9 @@ These are the challenges developers are facing, so the purpose of this content i > **NOTE:** App Engine ([Flexible](https://cloud.google.com/appengine/docs/flexible/python/runtime?hl=en#interpreter)) is a next-gen service but is not within the scope of these tutorials. Curious developers can compare App Engine [Standard vs. Flexible](https://cloud.google.com/appengine/docs/the-appengine-environments) to find out more. Also, many of the Flexible use cases can now be handled by [Cloud Run](http://cloud.run). -## Progression (START and FINISH) +## Progression (what order to do things) + +### "START" and "FINISH" repo folders All codelabs begin with code in a START repo folder and end with code in a FINISH folder, implementing a single migration. Upon completion, users should confirm their code (for the most part) matches what's in the FINISH folder. The baseline migration sample app (Module 0; link below) is a barebones Python 2.7 App Engine app that uses the `webapp2` web framework plus the `ndb` Datastore library. @@ -65,7 +72,19 @@ All codelabs begin with code in a START repo folder and end with code in a FINIS 1. Next, STARTing with the _Module 1_ application code (yours or ours), _Module 2_ migrates from `ndb` to Cloud NDB, ending with code matching the (Module 2) FINISH repo folder. There's also has a bonus migration to Python 3, resulting in another FINISH repo folder, this one deployed on the next-generation platform. 1. _Your_ Python 2 apps may be using other built-in services like Task Queues or Memcache, so additional migration modules follow, some more optional than others, and not all are available yet (keep checking back here for updates). -Beyond Module 2, with some exceptions, **there is no specific order** of what migrations modules to tackle next. It depends on your needs (and your applications'). +### The order of migrations + +Beyond Module 2, with some exceptions, **there is no specific order** of what migrations modules to tackle next. It depends on your needs (and your applications'). However, there are related migrations where one or more modules must be completed beforehand. This table attempts to put an order on module subsets. + +Topic | Module ordering | Description +--- | --- | --- +Baseline | 0 ⇒ 1 | Not a migration but a description of the baseline application (review this material before doing any migrations) +Web framework | 1 ⇒ _everything else_ | Current App Engine runtimes do not come with a web framework, so this must be the first migration performed. All migrations below can be performed after this one. +Datastore | 2 [⇒ 3 [⇒ 6]] | Moving off App Engine `ndb` makes your apps more portable, so the **Module 2** Cloud NDB migration is _recommended_. **Module 3:** Migrating to Cloud Datastore (Firestore in Datastore mode) is _optional_ and only recommended if you have other code using Cloud Datastore. **Module 6**: Migrating to Cloud Firestore (Native mode) is generally _not recommended_ unless you must have the Firebase features it has, and those features will eventually be integrated into Cloud Datastore. +(Push) Task Queues | [7 ⇒] 8 [⇒ 9] | Moving off App Engine `taskqueue` makes your apps more portable, so the **Module 8** Cloud Tasks migration is _recommended_ for those using push tasks. Those unfamiliar with push tasks should do **Module 7** first to add push tasks to the sample app. **Module 9:** Migrating to Cloud Datastore (Firestore in Datastore mode), Cloud Tasks (v2), and Python 3 is _optional_ and only recommended if you have other code using Cloud Datastore and considering upgrading to Python 3. +Memcache | [12 ⇒] 13 | Moving off App Engine `memcache` makes your apps more portable, so the **Module 13** Cloud Memorystore (for Redis) migration is _recommended_ for those using `memcache`. Those unfamiliar with `memcache` should do **Module 12** first to add its usage to the sample app. +Cloud Functions | 11 | Cloud Functions does not support Python 2, so after the Module 1 migration, you need to upgrade your app to Python 3 before attempting this migration, recommended if you have a very small App Engine app, or it has only one function/feature. +Cloud Run | 4 or 5 | **Module 4** covers migrating to Cloud Run with Docker. Those unfamiliar with containers or do not wish to create/maintain a `Dockerfile` should do **Module 5**. Those doing **Module 4** will get additional information about Cloud Run in **Module 5** not covered in **Module 4**. ## Migration modules @@ -86,7 +105,7 @@ Module | Topic | Video | Codelab | START here | FINISH here 6|Migrate to Cloud Firestore| _N/A_ | _N/A_ | Module 3 [code](/mod3b-datastore) (3.x) | _no work required; [Datastore upgrade automatic](https://cloud.google.com/datastore/docs/upgrade-to-firestore)_ 7|Add App Engine `taskqueue` push tasks| [link](https://twitter.com/googledevs/status/1443410302113099778?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrgaetasks_sms_201028&utm_content=-) | [link](http://g.co/codelabs/pae-migrate-gaetasks) | Module 1 [code](/mod1-flask) (2.x) | Module 7 [code](/mod7-gaetasks) (2.x) & [code](/mod7b-gaetasks) (3.x) 8|Migrate to Cloud Tasks| [link](https://twitter.com/googledevs/status/1450960021018267656?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrcloudtasks_sms_201112&utm_content=-) | [link](http://g.co/codelabs/pae-migrate-cloudtasks) | Module 7 [code](/mod7-gaetasks) (2.x) | Module 8 [code](/mod8-cloudtasks) (2.x) -9|Migrate to Python 3, Cloud Datastore & Cloud Tasks v2| _TBD_ | _TBD_ | Module 8 [code](/mod8-cloudtasks) (2.x) | _TBD_ +9|Migrate to Python 3, Cloud Datastore & Cloud Tasks v2| _TBD_ | _TBD_ | Module 8 [code](/mod8-cloudtasks) (2.x) | Module 9 [code](/mod9-py3dstasks) 10|Migrate Datastore/Firestore data to another project| _TBD_ | _N/A_ | _N/A_ | _TBD_ 11|Migrate to Cloud Functions| _TBD_ | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-11-functions?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrcloudfuncs_sms_202006&utm_content=-) | Module 2 [code](/mod2b-cloudndb) (3.x) | Module 11 [code](/mod11-functions) (3.x) 12|Add App Engine `memcache`| _TBD_ | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-12-memcache?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrcloudfuncs_sms_202006&utm_content=-) | Module 1 [code](/mod1-flask) (2.x) | Module 12 [code](/mod12-memcache) (2.x) & [code](/mod12b-memcache) (3.x) diff --git a/mod13b-memorystore/README.md b/mod13b-memorystore/README.md index eb622f3..3d04f68 100644 --- a/mod13b-memorystore/README.md +++ b/mod13b-memorystore/README.md @@ -1,8 +1,9 @@ # Module 13 - Migrate from App Engine `memcache` to Cloud Memorystore -This repo folder is the corresponding Python 3 code to a pair of different migrations related to the Module 12 and 13 codelabs. The tutorial either... +This repo folder is the corresponding Python 3 version of the Module 13 app. -- STARTs with the _Python 3_ code in the [Module 12b repo folder](/mod12b-memcache) and leads developers through migrations from App Engine `memcache` to Cloud Memorystore and App Engine `ndb` to Cloud Datastore (skipping Cloud NDB); **or**, -- STARTs with the _Python 2_ code in the [Module 13a repo folder](/mod13a-memorystore) and leads developers through migrations from Cloud NDB to Cloud Datastore, mirroring the content covered in [Module 3](http://g.co/codelabs/pae-migrate-datastore), and also a BONUS migration from Python 2 to 3. - -Either way, both set of migrations culminate in the code in this (`mod13b-memorystore`) folder. +- All files in this folder are identical to the _Python 2_ code in the [Module 13a repo folder](/mod13a-memorystore) **except**: + 1. `app.yaml` was modified for the Python 3 runtime. + 1. `appengine_config.py` is unused and thus deleted. +- An optional migration from Cloud NDB to Cloud Datastore can be achieved via the content covered in [Module 3](http://g.co/codelabs/pae-migrate-datastore). +- The _Python 3_ version of the Module 12 app ([Module 12b repo folder](/mod12b-memcache)) features additional code to support those App Engine legacy ("bundled") services (like `memcache`). Because the app in this folder does not use such services (moved to Cloud Memorystore), that extra support does not appear, so the code here should not be considered a direct migration of that app to Cloud Memorystore (and Cloud NDB), unlike the Python 2 equivalents (Modules [12a](/mod12-memcache) and [13a](/mod13a-memorystore)) which can. diff --git a/mod9-py3dstasks/README.md b/mod9-py3dstasks/README.md index 92aa7f0..1146d79 100644 --- a/mod9-py3dstasks/README.md +++ b/mod9-py3dstasks/README.md @@ -1,6 +1,6 @@ # Module 9 - Migrate from Python 2 to 3 and Cloud NDB to Cloud Datastore -This repo folder is the corresponding Python 3 code to the [Module 9 codelab](http://g.co/codelabs/pae-migrate-py3dstasks). The tutorial STARTs with the Python 2 code in the [Module 8 repo folder](/mod7-cloudtasks) and leads developers through migrating from Python 2 to 3, Cloud NDB to Cloud Datastore plus any changes from Cloud Tasks v1 to v2, culminating in the code in this folder. One major addition to look for here vs. Module 8 is that App Engine `taskqueue` creates a `default` push queue while Cloud Tasks does not, so that now has to be done in code. +This repo folder is the corresponding Python 3 code to the Module 9 codelab (_TBD_). The tutorial STARTs with the Python 2 code in the [Module 8 repo folder](/mod7-cloudtasks) and leads developers through migrating from Python 2 to 3, Cloud NDB to Cloud Datastore plus any changes from Cloud Tasks v1 to v2, culminating in the code in this folder. One major addition to look for here vs. Module 8 is that App Engine `taskqueue` creates a `default` push queue while Cloud Tasks does not, so that now has to be done in code. **NOTE: Backport to Python 2**: When migrating this app to Python 3, we added a Python 3 dependency: the `print()` function. If for any reason you need to get back on Python 2 App Engine, you would have to: From 1c48980b8fa70e76d7ab442d454016b295bcb553 Mon Sep 17 00:00:00 2001 From: wesley chun Date: Wed, 2 Feb 2022 01:02:07 -0800 Subject: [PATCH 27/56] add region tags; Redis port: int --- mod0-baseline/main.py | 2 ++ mod13b-memorystore/main.py | 2 +- mod1b-flask/main.py | 2 ++ 3 files changed, 5 insertions(+), 1 deletion(-) diff --git a/mod0-baseline/main.py b/mod0-baseline/main.py index 42afde4..57e261c 100644 --- a/mod0-baseline/main.py +++ b/mod0-baseline/main.py @@ -12,6 +12,7 @@ # See the License for the specific language governing permissions and # limitations under the License. +# [START mod0_baseline] import os import webapp2 from google.appengine.ext import ndb @@ -41,3 +42,4 @@ def get(self): app = webapp2.WSGIApplication([ ('/', MainHandler), ], debug=True) +# [END mod0_baseline] diff --git a/mod13b-memorystore/main.py b/mod13b-memorystore/main.py index 086c286..fb567e8 100644 --- a/mod13b-memorystore/main.py +++ b/mod13b-memorystore/main.py @@ -22,7 +22,7 @@ ds_client = ndb.Client() HOUR = 3600 REDIS_HOST = os.environ.get('REDIS_HOST', 'localhost') -REDIS_PORT = os.environ.get('REDIS_PORT', '6379') +REDIS_PORT = os.environ.get('REDIS_PORT', 6379) REDIS = redis.Redis(host=REDIS_HOST, port=REDIS_PORT) class Visit(ndb.Model): diff --git a/mod1b-flask/main.py b/mod1b-flask/main.py index e3880c7..8d34768 100644 --- a/mod1b-flask/main.py +++ b/mod1b-flask/main.py @@ -12,6 +12,7 @@ # See the License for the specific language governing permissions and # limitations under the License. +# [START mod1b_flask] from flask import Flask, render_template, request from google.appengine.api import wrap_wsgi_app from google.appengine.ext import ndb @@ -38,3 +39,4 @@ def root(): store_visit(request.remote_addr, request.user_agent) visits = fetch_visits(10) return render_template('index.html', visits=visits) +# [END mod1b_flask] From 027c0f03463e3188669ca367f96a6e99025e20f9 Mon Sep 17 00:00:00 2001 From: wesley chun Date: Wed, 9 Mar 2022 00:43:37 -0800 Subject: [PATCH 28/56] add Blobstore & GCS samples --- README.md | 9 ++- mod13b-memorystore/main.py | 2 +- mod15-blobstore/.gcloudignore | 80 +++++++++++++++++++ mod15-blobstore/README.md | 9 +++ mod15-blobstore/app.yaml | 25 ++++++ mod15-blobstore/main-gcs.py | 83 ++++++++++++++++++++ mod15-blobstore/main.py | 79 +++++++++++++++++++ mod15-blobstore/templates/index.html | 37 +++++++++ mod16a-cloudstorage/.gcloudignore | 80 +++++++++++++++++++ mod16a-cloudstorage/README.md | 9 +++ mod16a-cloudstorage/app.yaml | 29 +++++++ mod16a-cloudstorage/appengine_config.py | 23 ++++++ mod16a-cloudstorage/main-migrate.py | 97 ++++++++++++++++++++++++ mod16a-cloudstorage/main.py | 88 +++++++++++++++++++++ mod16a-cloudstorage/requirements.txt | 3 + mod16a-cloudstorage/templates/index.html | 37 +++++++++ 16 files changed, 687 insertions(+), 3 deletions(-) create mode 100644 mod15-blobstore/.gcloudignore create mode 100644 mod15-blobstore/README.md create mode 100644 mod15-blobstore/app.yaml create mode 100644 mod15-blobstore/main-gcs.py create mode 100644 mod15-blobstore/main.py create mode 100644 mod15-blobstore/templates/index.html create mode 100644 mod16a-cloudstorage/.gcloudignore create mode 100644 mod16a-cloudstorage/README.md create mode 100644 mod16a-cloudstorage/app.yaml create mode 100644 mod16a-cloudstorage/appengine_config.py create mode 100644 mod16a-cloudstorage/main-migrate.py create mode 100644 mod16a-cloudstorage/main.py create mode 100644 mod16a-cloudstorage/requirements.txt create mode 100644 mod16a-cloudstorage/templates/index.html diff --git a/README.md b/README.md index 5d58960..1743651 100644 --- a/README.md +++ b/README.md @@ -85,6 +85,8 @@ Datastore | 2 [⇒ 3 [⇒ 6]] | Moving off App Engine `ndb` makes your app Memcache | [12 ⇒] 13 | Moving off App Engine `memcache` makes your apps more portable, so the **Module 13** Cloud Memorystore (for Redis) migration is _recommended_ for those using `memcache`. Those unfamiliar with `memcache` should do **Module 12** first to add its usage to the sample app. Cloud Functions | 11 | Cloud Functions does not support Python 2, so after the Module 1 migration, you need to upgrade your app to Python 3 before attempting this migration, recommended if you have a very small App Engine app, or it has only one function/feature. Cloud Run | 4 or 5 | **Module 4** covers migrating to Cloud Run with Docker. Those unfamiliar with containers or do not wish to create/maintain a `Dockerfile` should do **Module 5**. Those doing **Module 4** will get additional information about Cloud Run in **Module 5** not covered in **Module 4**. +Blobstore | [15 ⇒] 16 | Moving off App Engine `blobstore` makes your apps more portable, so the **Module 16** Cloud Storage migration is _recommended_ for those using `blobstore`. Those unfamiliar with `blobstore` should do **Module 15** first to add its usage to the sample app. +general migration | [6 ⇒] 10 [⇒ 14] | This series is more generic and not targeting a specific feature migration, but rather if you need to migrate your App Engine apps from one running project to another. It starts with **Module 6** if you need to migrate your code, say from Datastore to Firestore. **Module 10** is if you need to migrate your data from one project to another, and finally, **Module 14** is after you're done migrating your app, your data, or both, and need to migrate a running service on one GCP project to another. ## Migration modules @@ -108,8 +110,11 @@ Module | Topic | Video | Codelab | START here | FINISH here 9|Migrate to Python 3, Cloud Datastore & Cloud Tasks v2| _TBD_ | _TBD_ | Module 8 [code](/mod8-cloudtasks) (2.x) | Module 9 [code](/mod9-py3dstasks) 10|Migrate Datastore/Firestore data to another project| _TBD_ | _N/A_ | _N/A_ | _TBD_ 11|Migrate to Cloud Functions| _TBD_ | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-11-functions?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrcloudfuncs_sms_202006&utm_content=-) | Module 2 [code](/mod2b-cloudndb) (3.x) | Module 11 [code](/mod11-functions) (3.x) -12|Add App Engine `memcache`| _TBD_ | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-12-memcache?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrcloudfuncs_sms_202006&utm_content=-) | Module 1 [code](/mod1-flask) (2.x) | Module 12 [code](/mod12-memcache) (2.x) & [code](/mod12b-memcache) (3.x) +12|Add App Engine `memcache`| _TBD_ | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-12-memcache?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrmemcache_sms_202006&utm_content=-) | Module 1 [code](/mod1-flask) (2.x) | Module 12 [code](/mod12-memcache) (2.x) & [code](/mod12b-memcache) (3.x) 13|Migrate to Cloud Memorystore| _TBD_ | _TBD_ | Module 12 [code](/mod12-memcache) (2.x) & [code](/mod12b-memcache) (3.x) | Module 13 [code](/mod13a-memorystore) (2.x) & [code](/mod13b-memorystore) (3.x) +14|Migrate service between projects| _TBD_ | _TBD_ | _TBD_ | _TBD_ +15|Add App Engine `blobstore`| _TBD_ | _TBD_ | Module 0 [code](/mod0-baseline) (2.x) | Module 15 [code](/mod15-blobstore) (2.x) +16|Migrate to Cloud Storage| _TBD_ | _TBD_ | Module 15 [code](/mod15-blobstore) (2.x) | Module 16 [code](/mod16a-cloudstorage) (2.x) ### Table of contents @@ -218,7 +223,7 @@ If there is a logical codelab to do immediately after completing one, they will - [Module 11 codelab](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-11-functions?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrcloudfuncs_sms_202006&utm_content=-): **Migrate from App Engine to Cloud Functions** - **Optional** migration - - Recommende for small apps or for breaking up large apps into multiple microservices + - Recommended for small apps or for breaking up large apps into multiple microservices - Python 3 only - START: [Module 2 code - Cloud NDB](/mod2b-cloudndb) (3.x) - FINISH: [Module 11 code - Cloud Functions](/mod11-functions) (3.x) diff --git a/mod13b-memorystore/main.py b/mod13b-memorystore/main.py index fb567e8..086c286 100644 --- a/mod13b-memorystore/main.py +++ b/mod13b-memorystore/main.py @@ -22,7 +22,7 @@ ds_client = ndb.Client() HOUR = 3600 REDIS_HOST = os.environ.get('REDIS_HOST', 'localhost') -REDIS_PORT = os.environ.get('REDIS_PORT', 6379) +REDIS_PORT = os.environ.get('REDIS_PORT', '6379') REDIS = redis.Redis(host=REDIS_HOST, port=REDIS_PORT) class Visit(ndb.Model): diff --git a/mod15-blobstore/.gcloudignore b/mod15-blobstore/.gcloudignore new file mode 100644 index 0000000..b4dbc7e --- /dev/null +++ b/mod15-blobstore/.gcloudignore @@ -0,0 +1,80 @@ +# This file specifies files that are *not* uploaded to Google Cloud Platform +# using gcloud. It follows the same syntax as .gitignore, with the addition of +# "#!include" directives (which insert the entries of the given .gitignore-style +# file at that point). +# +# For more information, run: +# $ gcloud topic gcloudignore +# +.gcloudignore + +# Source code control files +.git/ +.gitignore +.hgignore +.hg/ + +# README/text files +LICENSE +*.md + +# Tests/results (not in .gitignore) +noxfile.py +test_translate.py +pylintrc +pylintrc.test + +# most of .gitignore (except `lib`) +# +# Python +*.py[cod] +__pycache__/ +/setup.cfg + +# C extensions +*.so + +# Packages +*.egg +*.egg-info +dist +build +eggs +.eggs +parts +bin +var +sdist +develop-eggs +.installed.cfg +lib64 +*.tgz + +# Installer logs +pip-log.txt + +# Tests/results +.nox/ +.pytest_cache/ +.cache +.pytype +.coverage +coverage.xml +*sponge_log.xml +system_tests/local_test_setup + +# Mac +.DS_Store + +# IDEs/editors +*.sw[op] +*~ +.vscode +.idea + +# Built documentation +docs/_build +docs.metadata + +# Virtual environment +env/ diff --git a/mod15-blobstore/README.md b/mod15-blobstore/README.md new file mode 100644 index 0000000..b167eaf --- /dev/null +++ b/mod15-blobstore/README.md @@ -0,0 +1,9 @@ +# Module 15 - Add usage of App Engine `blobstore` to baseline sample app (Module 0) + +This repo folder is the corresponding code to the [Module 14 codelab](http://g.co/codelabs/pae-migrate-blobstore). The tutorial STARTs with the Python 2 code in the [Module 0 repo folder](/mod0-baseline) and leads developers through adding use of App Engine `blobstore`. Unlike other sample apps, this does not use the default Django templating system, but instead, uses Jinja2, which is supported in `webapp2_extras`. + +Blobstore evolved into [Google Cloud Storage](https://cloud.google.com/storage), and all blobs/files created using the Blobstore API go into the default Cloud Storage bucket for your Cloud project, which is the project's ID, meaning it's your `appspot` domain name, e.g., for project `my-project`, your default bucket would be `my-project.appspot.com`. It is programmatically accessible via `google.appengine.api.app_identity.get_default_gcs_bucket_name()`. + +The primary application file [`main.py`](main.py) writes files directly to the default bucket. If you want to customize the GCS location where App Engine writes files, see the alternative [`main-gcs.py`](main-gcs.py) file. It uses `google.appengine.api.app_identity.get_default_gcs_bucket_name()` along with the `gs_bucket_name` parameter when calling `google.appengine.ext.blobstore.create_upload_url()`. + +Unlike some of the other migrations, Blobstore usage depends on the `webapp` and `webapp2`, so this migration must start at Module 0 rather than Module 1. One update however, is that this sample does use the [Jinja2 templating system](https://jinja.palletsprojects.com) rather than the default Django templates used in Module 0. It is supported in `webapp2` via the `webapp2_extras` package. diff --git a/mod15-blobstore/app.yaml b/mod15-blobstore/app.yaml new file mode 100644 index 0000000..09e8291 --- /dev/null +++ b/mod15-blobstore/app.yaml @@ -0,0 +1,25 @@ +# Copyright 2022 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +runtime: python27 +threadsafe: yes +api_version: 1 + +handlers: +- url: /.* + script: main.app + +libraries: +- name: jinja2 + version: latest diff --git a/mod15-blobstore/main-gcs.py b/mod15-blobstore/main-gcs.py new file mode 100644 index 0000000..922a38e --- /dev/null +++ b/mod15-blobstore/main-gcs.py @@ -0,0 +1,83 @@ +# Copyright 2022 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import webapp2 +from webapp2_extras import jinja2 +from google.appengine.api import app_identity +from google.appengine.ext import blobstore, ndb +from google.appengine.ext.webapp import blobstore_handlers + +BUCKET = app_identity.get_default_gcs_bucket_name() + + +class BaseHandler(webapp2.RequestHandler): + 'Derived request handler mixing-in Jinja2 support' + @webapp2.cached_property + def jinja2(self): + return jinja2.get_jinja2(app=self.app) + + def render_response(self, _template, **context): + self.response.write(self.jinja2.render_template(_template, **context)) + + +class Visit(ndb.Model): + 'Visit entity registers visitor IP address & timestamp' + visitor = ndb.StringProperty() + timestamp = ndb.DateTimeProperty(auto_now_add=True) + file_blob = ndb.BlobKeyProperty() + + +def store_visit(remote_addr, user_agent, upload_key): + 'create new Visit entity in Datastore' + Visit(visitor='{}: {}'.format(remote_addr, user_agent), + file_blob=upload_key).put() + + +def fetch_visits(limit): + 'get most recent visits' + return Visit.query().order(-Visit.timestamp).fetch(limit) + + +class UploadHandler(blobstore_handlers.BlobstoreUploadHandler): + 'Upload blob (POST) handler' + def post(self): + uploads = self.get_uploads() + blob_id = uploads[0].key() if uploads else None + store_visit(self.request.remote_addr, self.request.user_agent, blob_id) + self.redirect('/', code=307) + + +class ViewBlobHandler(blobstore_handlers.BlobstoreDownloadHandler): + 'view uploaded blob (GET) handler' + def get(self, blob_key): + self.send_blob(blob_key) if blobstore.get(blob_key) else self.error(404) + + +class MainHandler(BaseHandler): + 'main application (GET/POST) handler' + def get(self): + self.render_response('index.html', + upload_url=blobstore.create_upload_url('/upload', + gs_bucket_name=BUCKET)) + + def post(self): + visits = fetch_visits(10) + self.render_response('index.html', visits=visits) + + +app = webapp2.WSGIApplication([ + ('/', MainHandler), + ('/upload', UploadHandler), + ('/view/([^/]+)?', ViewBlobHandler), +], debug=True) diff --git a/mod15-blobstore/main.py b/mod15-blobstore/main.py new file mode 100644 index 0000000..f390a18 --- /dev/null +++ b/mod15-blobstore/main.py @@ -0,0 +1,79 @@ +# Copyright 2022 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import webapp2 +from webapp2_extras import jinja2 +from google.appengine.ext import blobstore, ndb +from google.appengine.ext.webapp import blobstore_handlers + + +class BaseHandler(webapp2.RequestHandler): + 'Derived request handler mixing-in Jinja2 support' + @webapp2.cached_property + def jinja2(self): + return jinja2.get_jinja2(app=self.app) + + def render_response(self, _template, **context): + self.response.write(self.jinja2.render_template(_template, **context)) + + +class Visit(ndb.Model): + 'Visit entity registers visitor IP address & timestamp' + visitor = ndb.StringProperty() + timestamp = ndb.DateTimeProperty(auto_now_add=True) + file_blob = ndb.BlobKeyProperty() + + +def store_visit(remote_addr, user_agent, upload_key): + 'create new Visit entity in Datastore' + Visit(visitor='{}: {}'.format(remote_addr, user_agent), + file_blob=upload_key).put() + + +def fetch_visits(limit): + 'get most recent visits' + return Visit.query().order(-Visit.timestamp).fetch(limit) + + +class UploadHandler(blobstore_handlers.BlobstoreUploadHandler): + 'Upload blob (POST) handler' + def post(self): + uploads = self.get_uploads() + blob_id = uploads[0].key() if uploads else None + store_visit(self.request.remote_addr, self.request.user_agent, blob_id) + self.redirect('/', code=307) + + +class ViewBlobHandler(blobstore_handlers.BlobstoreDownloadHandler): + 'view uploaded blob (GET) handler' + def get(self, blob_key): + self.send_blob(blob_key) if blobstore.get(blob_key) else self.error(404) + + +class MainHandler(BaseHandler): + 'main application (GET/POST) handler' + def get(self): + self.render_response('index.html', + upload_url=blobstore.create_upload_url('https://github.com/upload')) + + def post(self): + visits = fetch_visits(10) + self.render_response('index.html', visits=visits) + + +app = webapp2.WSGIApplication([ + ('/', MainHandler), + ('/upload', UploadHandler), + ('/view/([^/]+)?', ViewBlobHandler), +], debug=True) diff --git a/mod15-blobstore/templates/index.html b/mod15-blobstore/templates/index.html new file mode 100644 index 0000000..6f6be5f --- /dev/null +++ b/mod15-blobstore/templates/index.html @@ -0,0 +1,37 @@ + + + +VisitMe Example + + +

VisitMe example

+{% if upload_url %} + +

Welcome... upload a file? (optional)

+
+

+ +
+ +{% else %} + +

Last 10 visits

+
    +{% for visit in visits %} +
  • {{ visit.timestamp.ctime() }} + + {% if visit.file_blob %} + (view) + {% else %} + (none) + {% endif %} + + from {{ visit.visitor }} +
  • +{% endfor %} +
+ +{% endif %} + + + diff --git a/mod16a-cloudstorage/.gcloudignore b/mod16a-cloudstorage/.gcloudignore new file mode 100644 index 0000000..b4dbc7e --- /dev/null +++ b/mod16a-cloudstorage/.gcloudignore @@ -0,0 +1,80 @@ +# This file specifies files that are *not* uploaded to Google Cloud Platform +# using gcloud. It follows the same syntax as .gitignore, with the addition of +# "#!include" directives (which insert the entries of the given .gitignore-style +# file at that point). +# +# For more information, run: +# $ gcloud topic gcloudignore +# +.gcloudignore + +# Source code control files +.git/ +.gitignore +.hgignore +.hg/ + +# README/text files +LICENSE +*.md + +# Tests/results (not in .gitignore) +noxfile.py +test_translate.py +pylintrc +pylintrc.test + +# most of .gitignore (except `lib`) +# +# Python +*.py[cod] +__pycache__/ +/setup.cfg + +# C extensions +*.so + +# Packages +*.egg +*.egg-info +dist +build +eggs +.eggs +parts +bin +var +sdist +develop-eggs +.installed.cfg +lib64 +*.tgz + +# Installer logs +pip-log.txt + +# Tests/results +.nox/ +.pytest_cache/ +.cache +.pytype +.coverage +coverage.xml +*sponge_log.xml +system_tests/local_test_setup + +# Mac +.DS_Store + +# IDEs/editors +*.sw[op] +*~ +.vscode +.idea + +# Built documentation +docs/_build +docs.metadata + +# Virtual environment +env/ diff --git a/mod16a-cloudstorage/README.md b/mod16a-cloudstorage/README.md new file mode 100644 index 0000000..35f4eca --- /dev/null +++ b/mod16a-cloudstorage/README.md @@ -0,0 +1,9 @@ +# Module 16 - Migrate from App Engine `blobstore` to Cloud Storage + +This repo folder is the corresponding code to the [Module 16 codelab](http://g.co/codelabs/pae-migrate-blobstore). The tutorial STARTs with the Python 2 code in the [Module 15 repo folder](/mod15-blobstore) and leads developers through a set of migrations, culminating in the code in this folder. In addition to migrating to Cloud Storage, a few others are done to get from Modules 15 to 16... here is the complete list: + +1. Migrate from App Engine `webapp2` to Flask +1. Migrate from App Engine `ndb` to Cloud NDB +1. Migrate from App Engine `blobstore` to Cloud Storage + +The reason why the web framework requires migration is because `blobstore` has dependencies on `webapp` and `webapp2`, so we could not start directly from a Flask app. diff --git a/mod16a-cloudstorage/app.yaml b/mod16a-cloudstorage/app.yaml new file mode 100644 index 0000000..676a40a --- /dev/null +++ b/mod16a-cloudstorage/app.yaml @@ -0,0 +1,29 @@ +# Copyright 2022 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +runtime: python27 +threadsafe: yes +api_version: 1 + +handlers: +- url: /.* + script: main.app + +libraries: +- name: grpcio + version: latest +- name: setuptools + version: latest +- name: ssl + version: latest diff --git a/mod16a-cloudstorage/appengine_config.py b/mod16a-cloudstorage/appengine_config.py new file mode 100644 index 0000000..2a41fb4 --- /dev/null +++ b/mod16a-cloudstorage/appengine_config.py @@ -0,0 +1,23 @@ +# Copyright 2021 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import pkg_resources +from google.appengine.ext import vendor + +# Set PATH to your libraries folder. +PATH = 'lib' +# Add libraries installed in the PATH folder. +vendor.add(PATH) +# Add libraries to pkg_resources working set to find the distribution. +pkg_resources.working_set.add_entry(PATH) diff --git a/mod16a-cloudstorage/main-migrate.py b/mod16a-cloudstorage/main-migrate.py new file mode 100644 index 0000000..e280cca --- /dev/null +++ b/mod16a-cloudstorage/main-migrate.py @@ -0,0 +1,97 @@ +# Copyright 2022 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import io +import logging + +from flask import (Flask, abort, redirect, render_template, + request, send_file, url_for) +from werkzeug.utils import secure_filename + +import google.auth +from google.cloud import ndb, storage, exceptions + +app = Flask(__name__) +ds_client = ndb.Client() +gcs_client = storage.Client() +_, PROJECT_ID = google.auth.default() +BUCKET = '%s.appspot.com' % PROJECT_ID + + +def store_blob(fname, media): + blob = gcs_client.bucket(BUCKET).blob(fname) + blob.upload_from_file(media, content_type=media.content_type) + + +class Visit(ndb.Model): + 'Visit entity registers visitor IP address & timestamp' + visitor = ndb.StringProperty() + timestamp = ndb.DateTimeProperty(auto_now_add=True) + file_blob = ndb.BlobKeyProperty() # backwards-compatibility + file_gcs = ndb.StringProperty() + + +def store_visit(remote_addr, user_agent, upload_key): + 'create new Visit entity in Datastore' + with ds_client.context(): + Visit(visitor='{}: {}'.format(remote_addr, user_agent), + file_gcs=upload_key).put() + + +def fetch_visits(limit): + 'get most recent visits' + with ds_client.context(): + return Visit.query().order(-Visit.timestamp).fetch(limit) + + +@app.route('/upload', methods=['POST']) +def upload(): + 'Upload blob (POST) handler' + fname = None + uploaded_file = request.files.get('file', None) + if uploaded_file: + fname = secure_filename(uploaded_file.filename) + store_blob(fname, uploaded_file) + store_visit(request.remote_addr, request.user_agent, fname) + return redirect(url_for('root'), code=307) + + +@app.route('/view/') +def view(fname): + 'view uploaded blob (GET) handler' + blob = gcs_client.bucket(BUCKET).blob(fname) + try: + media = blob.download_as_bytes() + except exceptions.NotFound as e: + abort(404) + return send_file(io.BytesIO(media), mimetype=blob.content_type) + + +def etl_visits(visits): + return [{ + 'visitor': v.visitor, + 'timestamp': v.timestamp, + 'file_blob': v.file_gcs if hasattr(v, 'file_gcs') and v.file_gcs else v.file_blob + } for v in visits] + + +@app.route('/', methods=['GET', 'POST']) +def root(): + 'main application (GET/POST) handler' + context = {} + if request.method == 'GET': + context['upload_url'] = url_for('upload') + else: + context['visits'] = etl_visits(fetch_visits(10)) + return render_template('index.html', **context) diff --git a/mod16a-cloudstorage/main.py b/mod16a-cloudstorage/main.py new file mode 100644 index 0000000..fd4dc54 --- /dev/null +++ b/mod16a-cloudstorage/main.py @@ -0,0 +1,88 @@ +# Copyright 2022 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import io +import logging + +from flask import (Flask, abort, redirect, render_template, + request, send_file, url_for) +from werkzeug.utils import secure_filename + +import google.auth +from google.cloud import ndb, storage, exceptions + +app = Flask(__name__) +ds_client = ndb.Client() +gcs_client = storage.Client() +_, PROJECT_ID = google.auth.default() +BUCKET = '%s.appspot.com' % PROJECT_ID + + +def store_blob(fname, media): + blob = gcs_client.bucket(BUCKET).blob(fname) + blob.upload_from_file(media, content_type=media.content_type) + + +class Visit(ndb.Model): + 'Visit entity registers visitor IP address & timestamp' + visitor = ndb.StringProperty() + timestamp = ndb.DateTimeProperty(auto_now_add=True) + file_blob = ndb.StringProperty() + + +def store_visit(remote_addr, user_agent, upload_key): + 'create new Visit entity in Datastore' + with ds_client.context(): + Visit(visitor='{}: {}'.format(remote_addr, user_agent), + file_blob=upload_key).put() + + +def fetch_visits(limit): + 'get most recent visits' + with ds_client.context(): + return Visit.query().order(-Visit.timestamp).fetch(limit) + + +@app.route('/upload', methods=['POST']) +def upload(): + 'Upload blob (POST) handler' + fname = None + uploaded_file = request.files.get('file', None) + if uploaded_file: + fname = secure_filename(uploaded_file.filename) + store_blob(fname, uploaded_file) + store_visit(request.remote_addr, request.user_agent, fname) + return redirect(url_for('root'), code=307) + + +@app.route('/view/') +def view(fname): + 'view uploaded blob (GET) handler' + blob = gcs_client.bucket(BUCKET).blob(fname) + try: + media = blob.download_as_bytes() + except exceptions.NotFound as e: + abort(404) + return send_file(io.BytesIO(media), mimetype=blob.content_type) + + +@app.route('/', methods=['GET', 'POST']) +def root(): + 'main application (GET/POST) handler' + context = {} + if request.method == 'GET': + context['upload_url'] = url_for('upload') + else: + context['visits'] = fetch_visits(10) + return render_template('index.html', **context) diff --git a/mod16a-cloudstorage/requirements.txt b/mod16a-cloudstorage/requirements.txt new file mode 100644 index 0000000..997c814 --- /dev/null +++ b/mod16a-cloudstorage/requirements.txt @@ -0,0 +1,3 @@ +flask +google-cloud-ndb +google-cloud-storage diff --git a/mod16a-cloudstorage/templates/index.html b/mod16a-cloudstorage/templates/index.html new file mode 100644 index 0000000..6f6be5f --- /dev/null +++ b/mod16a-cloudstorage/templates/index.html @@ -0,0 +1,37 @@ + + + +VisitMe Example + + +

VisitMe example

+{% if upload_url %} + +

Welcome... upload a file? (optional)

+
+

+ +
+ +{% else %} + +

Last 10 visits

+
    +{% for visit in visits %} +
  • {{ visit.timestamp.ctime() }} + + {% if visit.file_blob %} + (view) + {% else %} + (none) + {% endif %} + + from {{ visit.visitor }} +
  • +{% endfor %} +
+ +{% endif %} + + + From 5920af938abd4cc190be044b6fd0c089e40a89cd Mon Sep 17 00:00:00 2001 From: wesley chun Date: Wed, 9 Mar 2022 19:14:54 -0800 Subject: [PATCH 29/56] add Python 3 support for Mod 16 app --- README.md | 2 +- {mod16a-cloudstorage => mod16-cloudstorage}/.gcloudignore | 0 {mod16a-cloudstorage => mod16-cloudstorage}/README.md | 5 +++++ {mod16a-cloudstorage => mod16-cloudstorage}/app.yaml | 1 + .../appengine_config.py | 0 {mod16a-cloudstorage => mod16-cloudstorage}/main-migrate.py | 0 {mod16a-cloudstorage => mod16-cloudstorage}/main.py | 0 {mod16a-cloudstorage => mod16-cloudstorage}/requirements.txt | 0 .../templates/index.html | 0 9 files changed, 7 insertions(+), 1 deletion(-) rename {mod16a-cloudstorage => mod16-cloudstorage}/.gcloudignore (100%) rename {mod16a-cloudstorage => mod16-cloudstorage}/README.md (77%) rename {mod16a-cloudstorage => mod16-cloudstorage}/app.yaml (97%) rename {mod16a-cloudstorage => mod16-cloudstorage}/appengine_config.py (100%) rename {mod16a-cloudstorage => mod16-cloudstorage}/main-migrate.py (100%) rename {mod16a-cloudstorage => mod16-cloudstorage}/main.py (100%) rename {mod16a-cloudstorage => mod16-cloudstorage}/requirements.txt (100%) rename {mod16a-cloudstorage => mod16-cloudstorage}/templates/index.html (100%) diff --git a/README.md b/README.md index 1743651..a5dfef0 100644 --- a/README.md +++ b/README.md @@ -114,7 +114,7 @@ Module | Topic | Video | Codelab | START here | FINISH here 13|Migrate to Cloud Memorystore| _TBD_ | _TBD_ | Module 12 [code](/mod12-memcache) (2.x) & [code](/mod12b-memcache) (3.x) | Module 13 [code](/mod13a-memorystore) (2.x) & [code](/mod13b-memorystore) (3.x) 14|Migrate service between projects| _TBD_ | _TBD_ | _TBD_ | _TBD_ 15|Add App Engine `blobstore`| _TBD_ | _TBD_ | Module 0 [code](/mod0-baseline) (2.x) | Module 15 [code](/mod15-blobstore) (2.x) -16|Migrate to Cloud Storage| _TBD_ | _TBD_ | Module 15 [code](/mod15-blobstore) (2.x) | Module 16 [code](/mod16a-cloudstorage) (2.x) +16|Migrate to Cloud Storage| _TBD_ | _TBD_ | Module 15 [code](/mod15-blobstore) (2.x) | Module 16 [code](/mod16-cloudstorage) (2.x & 3.x) ### Table of contents diff --git a/mod16a-cloudstorage/.gcloudignore b/mod16-cloudstorage/.gcloudignore similarity index 100% rename from mod16a-cloudstorage/.gcloudignore rename to mod16-cloudstorage/.gcloudignore diff --git a/mod16a-cloudstorage/README.md b/mod16-cloudstorage/README.md similarity index 77% rename from mod16a-cloudstorage/README.md rename to mod16-cloudstorage/README.md index 35f4eca..2e466cb 100644 --- a/mod16a-cloudstorage/README.md +++ b/mod16-cloudstorage/README.md @@ -7,3 +7,8 @@ This repo folder is the corresponding code to the [Module 16 codelab](http://g.c 1. Migrate from App Engine `blobstore` to Cloud Storage The reason why the web framework requires migration is because `blobstore` has dependencies on `webapp` and `webapp2`, so we could not start directly from a Flask app. + +This app is fully Python 2-3 compatible. To do a Python 3 deployment of this app: + +1. Edit `app.yaml` by enabling/uncommenting the `runtime: python39` line +1. Delete all other lines in `app.yaml`, save, and deploy with `gcloud app deploy` diff --git a/mod16a-cloudstorage/app.yaml b/mod16-cloudstorage/app.yaml similarity index 97% rename from mod16a-cloudstorage/app.yaml rename to mod16-cloudstorage/app.yaml index 676a40a..756abc0 100644 --- a/mod16a-cloudstorage/app.yaml +++ b/mod16-cloudstorage/app.yaml @@ -12,6 +12,7 @@ # See the License for the specific language governing permissions and # limitations under the License. +#runtime: python39 runtime: python27 threadsafe: yes api_version: 1 diff --git a/mod16a-cloudstorage/appengine_config.py b/mod16-cloudstorage/appengine_config.py similarity index 100% rename from mod16a-cloudstorage/appengine_config.py rename to mod16-cloudstorage/appengine_config.py diff --git a/mod16a-cloudstorage/main-migrate.py b/mod16-cloudstorage/main-migrate.py similarity index 100% rename from mod16a-cloudstorage/main-migrate.py rename to mod16-cloudstorage/main-migrate.py diff --git a/mod16a-cloudstorage/main.py b/mod16-cloudstorage/main.py similarity index 100% rename from mod16a-cloudstorage/main.py rename to mod16-cloudstorage/main.py diff --git a/mod16a-cloudstorage/requirements.txt b/mod16-cloudstorage/requirements.txt similarity index 100% rename from mod16a-cloudstorage/requirements.txt rename to mod16-cloudstorage/requirements.txt diff --git a/mod16a-cloudstorage/templates/index.html b/mod16-cloudstorage/templates/index.html similarity index 100% rename from mod16a-cloudstorage/templates/index.html rename to mod16-cloudstorage/templates/index.html From 6ce828c7d83056451c05a61f5980a9770c396601 Mon Sep 17 00:00:00 2001 From: wesley chun Date: Thu, 10 Mar 2022 14:37:48 -0800 Subject: [PATCH 30/56] README cleanup --- mod15-blobstore/README.md | 10 +++++----- mod16-cloudstorage/README.md | 14 +++++++++++++- 2 files changed, 18 insertions(+), 6 deletions(-) diff --git a/mod15-blobstore/README.md b/mod15-blobstore/README.md index b167eaf..55857b1 100644 --- a/mod15-blobstore/README.md +++ b/mod15-blobstore/README.md @@ -1,9 +1,9 @@ -# Module 15 - Add usage of App Engine `blobstore` to baseline sample app (Module 0) +# Module 15 - Add usage of App Engine `blobstore` to `webapp2 ndb` sample app -This repo folder is the corresponding code to the [Module 14 codelab](http://g.co/codelabs/pae-migrate-blobstore). The tutorial STARTs with the Python 2 code in the [Module 0 repo folder](/mod0-baseline) and leads developers through adding use of App Engine `blobstore`. Unlike other sample apps, this does not use the default Django templating system, but instead, uses Jinja2, which is supported in `webapp2_extras`. +This repo folder is the corresponding code to the (forthcoming) Module 15 codelab. The tutorial STARTs with the Python 2 code in the [Module 0 repo folder](/mod0-baseline) and leads developers through adding use of App Engine `blobstore`. Unlike other sample apps, this does not use the default Django templating system, but instead, uses Jinja2, which is supported in `webapp2_extras`. -Blobstore evolved into [Google Cloud Storage](https://cloud.google.com/storage), and all blobs/files created using the Blobstore API go into the default Cloud Storage bucket for your Cloud project, which is the project's ID, meaning it's your `appspot` domain name, e.g., for project `my-project`, your default bucket would be `my-project.appspot.com`. It is programmatically accessible via `google.appengine.api.app_identity.get_default_gcs_bucket_name()`. +Blobstore evolved into [Google Cloud Storage](https://cloud.google.com/storage), and all blobs/files created using the Blobstore API go into the default Cloud Storage bucket for your project. It's named the same as the `appspot` domain name given to your app. For example, if your project is named `my-project`, your default bucket would be `my-project.appspot.com`. The default GCS bucket name is programmatically accessible via `google.appengine.api.app_identity.get_default_gcs_bucket_name()`. -The primary application file [`main.py`](main.py) writes files directly to the default bucket. If you want to customize the GCS location where App Engine writes files, see the alternative [`main-gcs.py`](main-gcs.py) file. It uses `google.appengine.api.app_identity.get_default_gcs_bucket_name()` along with the `gs_bucket_name` parameter when calling `google.appengine.ext.blobstore.create_upload_url()`. +The primary application file [`main.py`](main.py) writes files directly to the default bucket. If you want to customize the GCS location where App Engine writes files, see the alternative [`main-gcs.py`](main-gcs.py) file. In that file, the `gs_bucket_name` parameter is used when calling `google.appengine.ext.blobstore.create_upload_url()` to specify the bucket/location to write the file. -Unlike some of the other migrations, Blobstore usage depends on the `webapp` and `webapp2`, so this migration must start at Module 0 rather than Module 1. One update however, is that this sample does use the [Jinja2 templating system](https://jinja.palletsprojects.com) rather than the default Django templates used in Module 0. It is supported in `webapp2` via the `webapp2_extras` package. +Unlike some of the other migrations, Blobstore usage depends on `webapp` (where as the app uses the `webapp2` micro framework), so this migration must start at Module 0 rather than Module 1. One update however, is that this sample switches to the [Jinja2 templating system](https://jinja.palletsprojects.com) from the default Django template system used in Module 0. Jinja2 is supported as an App Engine [built-in library](https://cloud.google.com/appengine/docs/standard/python/tools/built-in-libraries-27) and accessed via the `webapp2_extras` package. diff --git a/mod16-cloudstorage/README.md b/mod16-cloudstorage/README.md index 2e466cb..074e67c 100644 --- a/mod16-cloudstorage/README.md +++ b/mod16-cloudstorage/README.md @@ -1,5 +1,7 @@ # Module 16 - Migrate from App Engine `blobstore` to Cloud Storage +## Migrations + This repo folder is the corresponding code to the [Module 16 codelab](http://g.co/codelabs/pae-migrate-blobstore). The tutorial STARTs with the Python 2 code in the [Module 15 repo folder](/mod15-blobstore) and leads developers through a set of migrations, culminating in the code in this folder. In addition to migrating to Cloud Storage, a few others are done to get from Modules 15 to 16... here is the complete list: 1. Migrate from App Engine `webapp2` to Flask @@ -8,7 +10,17 @@ This repo folder is the corresponding code to the [Module 16 codelab](http://g.c The reason why the web framework requires migration is because `blobstore` has dependencies on `webapp` and `webapp2`, so we could not start directly from a Flask app. +## Python compatibility + This app is fully Python 2-3 compatible. To do a Python 3 deployment of this app: 1. Edit `app.yaml` by enabling/uncommenting the `runtime: python39` line -1. Delete all other lines in `app.yaml`, save, and deploy with `gcloud app deploy` +1. Delete all other lines in `app.yaml` and save +1. Delete `lib` (if present) and `appengine_config.py` (neither used in Python 3) +1. Deploy with `gcloud app deploy` + +## Backwards compatibility + +One catch with this migration is that `blobstore` has a dependency on `webapp`. By migrating to Cloud Storage, that dependency is not resolved because the app was also migrated from `webapp2` (and `webapp`) to Flask. In real life, there may not be an option to just discard all your data. The [`main.py`](main.py) in this folder is for the easy situation where you _can_, replacing `ndb.BlobKeyProperty` (for Blobstore files) with `ndb.StringProperty` (for Cloud Storage files) in the data model. + +For the rest of us, we may need [`main-migrate.py`](main-migrate.py), an alternative version of the application. The data model here maintains a `ndb.BlobKeyProperty` for backwards-compatibility and creates a 4th field for the Cloud Storage filename (`ndb.StringProperty`). Furthermore, an additional `etl_visits()` function is required to consolidate files created with Blobstore _and_ Cloud Storage without changing the HTML template. From 1c673224d01b277005aa01b106ab70558988299d Mon Sep 17 00:00:00 2001 From: wesley chun Date: Wed, 30 Mar 2022 23:35:34 -0700 Subject: [PATCH 31/56] flake8 & other minor fixes --- mod13b-memorystore/main.py | 2 +- mod16-cloudstorage/main-migrate.py | 18 +++++++----------- mod16-cloudstorage/main.py | 18 +++++++----------- 3 files changed, 15 insertions(+), 23 deletions(-) diff --git a/mod13b-memorystore/main.py b/mod13b-memorystore/main.py index 086c286..fb567e8 100644 --- a/mod13b-memorystore/main.py +++ b/mod13b-memorystore/main.py @@ -22,7 +22,7 @@ ds_client = ndb.Client() HOUR = 3600 REDIS_HOST = os.environ.get('REDIS_HOST', 'localhost') -REDIS_PORT = os.environ.get('REDIS_PORT', '6379') +REDIS_PORT = os.environ.get('REDIS_PORT', 6379) REDIS = redis.Redis(host=REDIS_HOST, port=REDIS_PORT) class Visit(ndb.Model): diff --git a/mod16-cloudstorage/main-migrate.py b/mod16-cloudstorage/main-migrate.py index e280cca..60a3448 100644 --- a/mod16-cloudstorage/main-migrate.py +++ b/mod16-cloudstorage/main-migrate.py @@ -13,14 +13,13 @@ # limitations under the License. import io -import logging from flask import (Flask, abort, redirect, render_template, request, send_file, url_for) from werkzeug.utils import secure_filename import google.auth -from google.cloud import ndb, storage, exceptions +from google.cloud import exceptions, ndb, storage app = Flask(__name__) ds_client = ndb.Client() @@ -29,10 +28,6 @@ BUCKET = '%s.appspot.com' % PROJECT_ID -def store_blob(fname, media): - blob = gcs_client.bucket(BUCKET).blob(fname) - blob.upload_from_file(media, content_type=media.content_type) - class Visit(ndb.Model): 'Visit entity registers visitor IP address & timestamp' @@ -59,10 +54,11 @@ def fetch_visits(limit): def upload(): 'Upload blob (POST) handler' fname = None - uploaded_file = request.files.get('file', None) - if uploaded_file: - fname = secure_filename(uploaded_file.filename) - store_blob(fname, uploaded_file) + upload = request.files.get('file', None) + if upload: + fname = secure_filename(upload.filename) + blob = gcs_client.bucket(BUCKET).blob(fname) + blob.upload_from_file(upload, content_type=upload.content_type) store_visit(request.remote_addr, request.user_agent, fname) return redirect(url_for('root'), code=307) @@ -73,7 +69,7 @@ def view(fname): blob = gcs_client.bucket(BUCKET).blob(fname) try: media = blob.download_as_bytes() - except exceptions.NotFound as e: + except exceptions.NotFound: abort(404) return send_file(io.BytesIO(media), mimetype=blob.content_type) diff --git a/mod16-cloudstorage/main.py b/mod16-cloudstorage/main.py index fd4dc54..a3416bb 100644 --- a/mod16-cloudstorage/main.py +++ b/mod16-cloudstorage/main.py @@ -13,14 +13,13 @@ # limitations under the License. import io -import logging from flask import (Flask, abort, redirect, render_template, request, send_file, url_for) from werkzeug.utils import secure_filename import google.auth -from google.cloud import ndb, storage, exceptions +from google.cloud import exceptions, ndb, storage app = Flask(__name__) ds_client = ndb.Client() @@ -29,10 +28,6 @@ BUCKET = '%s.appspot.com' % PROJECT_ID -def store_blob(fname, media): - blob = gcs_client.bucket(BUCKET).blob(fname) - blob.upload_from_file(media, content_type=media.content_type) - class Visit(ndb.Model): 'Visit entity registers visitor IP address & timestamp' @@ -58,10 +53,11 @@ def fetch_visits(limit): def upload(): 'Upload blob (POST) handler' fname = None - uploaded_file = request.files.get('file', None) - if uploaded_file: - fname = secure_filename(uploaded_file.filename) - store_blob(fname, uploaded_file) + upload = request.files.get('file', None) + if upload: + fname = secure_filename(upload.filename) + blob = gcs_client.bucket(BUCKET).blob(fname) + blob.upload_from_file(upload, content_type=upload.content_type) store_visit(request.remote_addr, request.user_agent, fname) return redirect(url_for('root'), code=307) @@ -72,7 +68,7 @@ def view(fname): blob = gcs_client.bucket(BUCKET).blob(fname) try: media = blob.download_as_bytes() - except exceptions.NotFound as e: + except exceptions.NotFound: abort(404) return send_file(io.BytesIO(media), mimetype=blob.content_type) From ca2ced3237404d7918a466582183b7747841397a Mon Sep 17 00:00:00 2001 From: wesley chun Date: Tue, 24 May 2022 01:23:33 -0700 Subject: [PATCH 32/56] add new codelabs; README cleanup --- README.md | 153 ++++++++++++++++++++++++++++++++---------------------- 1 file changed, 90 insertions(+), 63 deletions(-) diff --git a/README.md b/README.md index a5dfef0..43dd9c9 100644 --- a/README.md +++ b/README.md @@ -99,7 +99,7 @@ The table below summarizes migration module resources currently available along Module | Topic | Video | Codelab | START here | FINISH here --- | --- | --- | --- | --- | --- 0|Baseline app| [link](http://twitter.com/googledevs/status/1407755281786867714?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_smsintro_201023&utm_content=-) | _N/A_ (no tutorial; just review the code) | _N/A_ | Module 0 [code](/mod0-baseline) (2.x) -1|Migrate to Flask| [link](https://twitter.com/googledevs/status/1413273119071031299?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrwa2flsk_201008&u%20tm_content=-) | [link](http://g.co/codelabs/pae-migrate-flask) | Module 0 [code](/mod0-baseline) (2.x) | Module 1 [code](/mod1-flask) (2.x) & [code](/mod1b-flask) (3.x) +1|Migrate to Flask| [link](https://twitter.com/googledevs/status/1413273119071031299?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrwa2flsk_201008&u%20tm_content=-) | [link](http://g.co/codelabs/pae-migrate-flask) | Module 0 [code](/mod0-baseline) (2.x) | Module 1 [code](/mod1-flask) (2.x) (and [code](/mod1b-flask) (3.x)) 2|Migrate to Cloud NDB| [link](https://twitter.com/googledevs/status/1417892493383802883?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrcloudndb_201021&utm_content=-) | [link](http://g.co/codelabs/pae-migrate-cloudndb) | Module 1 [code](/mod1-flask) (2.x) | Module 2 [code](/mod2a-cloudndb) (2.x) & [code](/mod2b-cloudndb) (3.x) 3|Migrate to Cloud Datastore| [link](http://twitter.com/googledevs/status/1422966928910393347?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrcloudds_201003&utm_content=-) | [link](http://g.co/codelabs/pae-migrate-datastore) | Module 2 [code](/mod2a-cloudndb) (2.x) & [code](/mod2b-cloudndb) (3.x) | Module 3 [code](/mod3a-datastore) (2.x) & [code](/mod3b-datastore) (3.x) 4|Migrate to Cloud Run with Docker| [link](https://twitter.com/googledevs/status/1428041270702735362?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrcrdckr_sms_201017&utm_content=-)| [link](http://g.co/codelabs/pae-migrate-rundocker) | Module 2 [code](/mod2a-cloudndb) (2.x) & Module 3 [code](/mod3b-datastore) (3.x) | Module 4 [code](/mod4a-rundocker) (2.x) & [code](/mod4b-rundocker) (3.x) @@ -109,12 +109,13 @@ Module | Topic | Video | Codelab | START here | FINISH here 8|Migrate to Cloud Tasks| [link](https://twitter.com/googledevs/status/1450960021018267656?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrcloudtasks_sms_201112&utm_content=-) | [link](http://g.co/codelabs/pae-migrate-cloudtasks) | Module 7 [code](/mod7-gaetasks) (2.x) | Module 8 [code](/mod8-cloudtasks) (2.x) 9|Migrate to Python 3, Cloud Datastore & Cloud Tasks v2| _TBD_ | _TBD_ | Module 8 [code](/mod8-cloudtasks) (2.x) | Module 9 [code](/mod9-py3dstasks) 10|Migrate Datastore/Firestore data to another project| _TBD_ | _N/A_ | _N/A_ | _TBD_ -11|Migrate to Cloud Functions| _TBD_ | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-11-functions?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrcloudfuncs_sms_202006&utm_content=-) | Module 2 [code](/mod2b-cloudndb) (3.x) | Module 11 [code](/mod11-functions) (3.x) -12|Add App Engine `memcache`| _TBD_ | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-12-memcache?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrmemcache_sms_202006&utm_content=-) | Module 1 [code](/mod1-flask) (2.x) | Module 12 [code](/mod12-memcache) (2.x) & [code](/mod12b-memcache) (3.x) -13|Migrate to Cloud Memorystore| _TBD_ | _TBD_ | Module 12 [code](/mod12-memcache) (2.x) & [code](/mod12b-memcache) (3.x) | Module 13 [code](/mod13a-memorystore) (2.x) & [code](/mod13b-memorystore) (3.x) +11|Migrate to Cloud Functions| [link](https://twitter.com/googledevs/status/1520206298834481153?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrcloudfuncs_sms_202006&utm_content=-) | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-11-functions?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrcloudfuncs_sms_202006&utm_content=-) | Module 2 [code](/mod2b-cloudndb) (3.x) | Module 11 [code](/mod11-functions) (3.x) +12|Add App Engine `memcache`| [link](https://twitter.com/googledevs/status/1527303061953126402?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrmemcache_sms_202006&utm_content=-) | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-12-memcache?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrmemcache_sms_202006&utm_content=-) | Module 1 [code](/mod1-flask) (2.x) | Module 12 [code](/mod12-memcache) (2.x) & [code](/mod12b-memcache) (3.x) +13|Migrate to Cloud Memorystore| _TBD_ | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-13-memorystore?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrmemorystore_sms_202029&utm_content=-) | Module 12 [code](/mod12-memcache) (2.x) & [code](/mod12b-memcache) (3.x) | Module 13 [code](/mod13a-memorystore) (2.x) & [code](/mod13b-memorystore) (3.x) 14|Migrate service between projects| _TBD_ | _TBD_ | _TBD_ | _TBD_ -15|Add App Engine `blobstore`| _TBD_ | _TBD_ | Module 0 [code](/mod0-baseline) (2.x) | Module 15 [code](/mod15-blobstore) (2.x) -16|Migrate to Cloud Storage| _TBD_ | _TBD_ | Module 15 [code](/mod15-blobstore) (2.x) | Module 16 [code](/mod16-cloudstorage) (2.x & 3.x) +15|Add App Engine `blobstore`| _TBD_ | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-15-blobstore?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrblobstore_sms_202029&utm_content=-) | Module 0 [code](/mod0-baseline) (2.x) | Module 15 [code](/mod15-blobstore) (2.x) +16|Migrate to Cloud Storage| _TBD_ | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-16-cloudstorage?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrcloudstorage_sms_202029&utm_content=-) | Module 15 [code](/mod15-blobstore) (2.x) | Module 16 [code](/mod16-cloudstorage) (2.x & 3.x) +17|Migrate to Python 3 bundled services| _TBD_ | _TBD_ | Module 1 [code](/mod1-flask) (2.x) | Module 1 [code](/mod1b-flask) (3.x) ### Table of contents @@ -122,45 +123,40 @@ Module | Topic | Video | Codelab | START here | FINISH here If there is a logical codelab to do immediately after completing one, they will be designated as NEXT. Other recommended codelabs will be listed as RECOMMENDED, and the more optional ones will be labeled as OTHERS (and usually in some kind of priority order). +#### Migrations from legacy App Engine APIs/bundled services + - [Module 1 codelab](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-1-flask?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrwa2flsk_201008&utm_content=-): **Migrate from `webapp2` to [Flask](https://flask.palletsprojects.com)** - **Required** migration (can also pick your own framework) - `webapp2` does not do routing thus unsupported by App Engine (even though a [3.x port exists](https://github.com/fili/webapp2-gae-python37)) - - Python 2 only - - START: [Module 0 code - Baseline](/mod0-baseline) (2.x) - - FINISH: [Module 1 code - Framework](/mod1-flask) (2.x) + - Python 2 + - START: [Module 0 code - Baseline](/mod0-baseline) + - FINISH: [Module 1 code - Framework](/mod1-flask) - NEXT: - Module 2 - migrate to Cloud NDB - [Module 2 codelab](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-2-cloudndb?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrcloudndb_201021&utm_content=-): **Migrate from App Engine `ndb` to [Cloud NDB](https://googleapis.dev/python/python-ndb/latest)** - - **Required** migration - - Migration to Cloud NDB which is supported by Python 3 and the next-gen platform. - Python 2 - - START: [Module 1 code - Framework](/mod1-flask) (2.x) - - FINISH: [Module 2 code - Cloud NDB](/mod2a-cloudndb) (2.x) + - START: [Module 1 code - Framework](/mod1-flask) + - FINISH: [Module 2 code - Cloud NDB](/mod2a-cloudndb) - Codelab bonus port to Python 3.x - FINISH: [Module 2 code - Cloud NDB](/mod2b-cloudndb) (3.x) - RECOMMENDED: - - Module 7 - add App Engine (push) tasks - - OTHERS (somewhat priority order): - - Module 11 - migrate to Cloud Functions - - Module 5 - migrate to Cloud Run container with Cloud Buildpacks - - Module 4 - migrate to Cloud Run container with Docker - - Module 3 - migrate to Cloud Datastore + - Module 7 - add App Engine (push) tasks (and migrate to Cloud Tasks in Module 8) + - Module 12 - add App Engine `memcache` (and migrate to Cloud Memorystore in Module 13) + - Module 15 - add App Engine `blobstore` (and migrate to Cloud Storage in Module 16) -- [Module 7 codelab](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-7-gaetasks?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrgaetasks_sms_201028&utm_content=-): **Add App Engine (push) Task Queues to App Engine `ndb` Flask app** +- [Module 7 codelab](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-7-gaetasks?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrgaetasks_sms_201028&utm_content=-): **Add App Engine (push) Task Queues to existing sample app** - **Not a migration**: add GAE Task Queues to prepare for migration to Cloud Tasks - Python 2 - - START: [Module 1 code - Framework](/mod1-flask) (2.x) - - FINISH: [Module 7 code - GAE Task Queues](/mod7-gaetasks) (2.x) + - START: [Module 1 code - Framework](/mod1-flask) + - FINISH: [Module 7 code - GAE Task Queues](/mod7-gaetasks) - NEXT: Module 8 - migrate App Engine push tasks to Cloud Tasks - [Module 8 codelab](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-8-cloudtasks?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrcloudtasks_sms_201112&utm_content=-): **Migrate from App Engine (push) Task Queues to [Cloud Tasks](http://cloud.google.com/tasks) v1** - - **Required** migration - - Migration to Cloud Tasks which is supported by Python 3 and the next-gen platform. - - Note this is only push tasks... pull tasks will be handled in a different codelab. + - Note only covers push tasks... pull tasks will be handled in a different codelab. - Python 2 - - START: [Module 7 code - GAE Task Queues](/mod7-gaetasks) (2.x) - - FINISH: [Module 8 code - Cloud Tasks](/mod8-cloudtasks) (2.x) + - START: [Module 7 code - GAE Task Queues](/mod7-gaetasks) + - FINISH: [Module 8 code - Cloud Tasks](/mod8-cloudtasks) - NEXT: Module 9 - migrate to Python 3 and Cloud Datastore - **Module 9 codelab** (TBD): **Migrate a Python 2 Cloud NDB & Cloud Tasks (v1) app to a Python 3 Cloud Datastore & Cloud Tasks (v2) app** @@ -168,27 +164,77 @@ If there is a logical codelab to do immediately after completing one, they will - Migrating to Python 3 is not required but recommended as Python 2 has been sunset - Migrating to Cloud Datastore is optional as Cloud NDB works on 3.x - Python 2 - - START: [Module 8 code - Cloud Tasks](/mod8-cloudtasks) (2.x) + - START: [Module 8 code - Cloud Tasks](/mod8-cloudtasks) - Python 3 - FINISH: _TBD_ - RECOMMENDED: - - Module 11 - migrate to Cloud Functions - - Module 5 - migrate to Cloud Run container with Cloud Buildpacks - - Module 4 - migrate to Cloud Run container with Docker + - Module 12 - add App Engine Memcache (and migrate to Cloud Memorystore in Module 13) + - Module 15 - add App Engine Blobstore (and migrate to Cloud Storage in Module 16) + +- [Module 12 codelab](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-12-memcache?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrmemcache_sms_202006&utm_content=-): **Add App Engine Memcache to existing sample app** + - **Not a migration**: add GAE Memcache to prepare for migration to Cloud Memorystore + - Python 2 + - START: [Module 1 code - Framework](/mod1-flask) + - FINISH: [Module 12 code - GAE Memcache](/mod12-memcache) + - NEXT: Module 13 - migrate App Engine Memcache to Cloud Memorystore + +- [Module 13 codelab](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-13-memorystore?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrmemorystore_sms_202029&utm_content=-): **Migrate from App Engine Memcache to [Cloud Memorystore (for Redis)](http://cloud.google.com/memorystore) v1** + - Python 2 + - START: [Module 12 code - GAE Memcache](/mod12-memcache) + - FINISH: [Module 13 code - Cloud Tasks](/mod13a-memorystore) + - Python 3 + - FINISH: [Module 13 code - Cloud Tasks](/mod13b-memorystore) + - RECOMMENDED: + - Module 7 - add App Engine (push) tasks (and migrate to Cloud Tasks in Module 8) + - Module 15 - add App Engine Blobstore (and migrate to Cloud Storage in Module 16) + +- [Module 15 codelab](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-12-memcache?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrmemcache_sms_202006&utm_content=-): **Add App Engine Blobstore to existing sample app** + - **Not a migration**: add GAE Blobstore to prepare for migration to Cloud Storage + - Python 2 + - START: [Module 0 code - Baseline](/mod0-baseline) + - FINISH: [Module 15 code - GAE Blobstore](/mod15-blobstore) + - NEXT: Module 13 - migrate App Engine Blobstore to Cloud Storage + +- [Module 16 codelab](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-13-memorystore?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrmemorystore_sms_202029&utm_content=-): **Migrate from App Engine Blobstore to [Cloud Storage (for Redis)](http://cloud.google.com/storage) v1** + - Python 2 + - START: [Module 15 code - GAE Blobstore](/mod15-blobstore) + - FINISH: [Module 16 code - Cloud Storage](/mod16-cloudstorage) + - Python 3 + - FINISH: [Module 16 code - Cloud Storage](/mod16-cloudstorage) (_same as Python 2 version_) + - RECOMMENDED: + - Module 7 - add App Engine (push) tasks (and migrate to Cloud Tasks in Module 8) + - Module 12 - add App Engine Memcache (and migrate to Cloud Memorystore in Module 13) + + +- [Module 3 codelab](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-3-datastore?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrcloudds_201003&utm_content=-): **Migrate from Cloud NDB to [Cloud Datastore](http://cloud.google.com/datastore)** + - **Optional** migration + - Recommended only if using Cloud Datastore elsewhere (GAE *or* non-App Engine) apps + - Helps w/code consistency & reusability, reduces maintenance costs + - Python 2 + - START: [Module 2 code - Cloud NDB](/mod2a-cloudndb) + - FINISH: [Module 3 code - Cloud Datastore](/mod3a-datastore) + - Python 3 + - START: [Module 2 code - Cloud NDB](/mod2b-cloudndb) + - FINISH: [Module 3 code - Cloud Datastore](/mod3b-datastore) + - RECOMMENDED: + - Module 7 - add App Engine (push) tasks (and migrate to Cloud Tasks in Module 8) + - Module 12 - add App Engine Memcache (and migrate to Cloud Memorystore in Module 13) + - Module 15 - add App Engine Blobstore (and migrate to Cloud Storage in Module 16) + + +#### Migrations to other Cloud serverless platforms - [Module 4 codelab](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-4-rundocker?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrcrdckr_sms_201017&utm_content=-): **Migrate from App Engine to Cloud Run with Docker** - **Optional** migration - "Containerize" your app (migrate your app to a container) with Docker - Python 2 - - START: [Module 2 code - Cloud NDB](/mod2a-cloudndb) (2.x) - - FINISH: [Module 4 code - Cloud Run - Docker 3.x](/mod4a-rundocker) (2.x) + - START: [Module 2 code - Cloud NDB](/mod2a-cloudndb) + - FINISH: [Module 4 code - Cloud Run - Docker 3.x](/mod4a-rundocker) - Python 3 - - START: [Module 3 code - Cloud Datastore](/mod3b-datastore) (3.x) - - FINISH: [Module 4 code - Cloud Run - Docker](/mod4b-rundocker) (3.x) + - START: [Module 3 code - Cloud Datastore](/mod3b-datastore) + - FINISH: [Module 4 code - Cloud Run - Docker](/mod4b-rundocker) - RECOMMENDED: - Module 5 - migrate to Cloud Run container with Cloud Buildpacks - - OTHER OPTIONS (in somewhat priority order): - - Module 7 - add App Engine (push) tasks - Module 11 - migrate to Cloud Functions - [Module 5 codelab](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-5-runbldpks?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrcrbdpk_sms_201031&utm_content=-): **Migrate from App Engine to Cloud Run with Cloud Buildpacks** @@ -196,43 +242,24 @@ If there is a logical codelab to do immediately after completing one, they will - "Containerize" your app (migrate your app to a container) with... - [Cloud Buildpacks]() which lets you containerize your app without `Dockerfile`s - Python 3 only - - START: [Module 2 code - Cloud NDB](/mod2b-cloudndb) (3.x) - - FINISH: [Module 5 code - Cloud Run - Buildpacks 3.x](/mod5-runbldpks) (3.x) + - START: [Module 2 code - Cloud NDB](/mod2b-cloudndb) + - FINISH: [Module 5 code - Cloud Run - Buildpacks 3.x](/mod5-runbldpks) - RECOMMENDED: - Module 4 - migrate to Cloud Run container with Docker - - OTHER OPTIONS (in somewhat priority order): - - Module 7 - add App Engine (push) tasks - Module 11 - migrate to Cloud Functions -- [Module 3 codelab](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-3-datastore?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrcloudds_201003&utm_content=-): **Migrate from Cloud NDB to [Cloud Datastore](http://cloud.google.com/datastore)** - - **Optional** migration - - Recommended only if using Cloud Datastore elsewhere (GAE *or* non-App Engine) apps - - Helps w/code consistency & reusability, reduces maintenance costs - - Python 2 - - START: [Module 2 code - Cloud NDB](/mod2a-cloudndb) (2.x) - - FINISH: [Module 3 code - Cloud Datastore](/mod3a-datastore) (2.x) - - Python 3 - - START: [Module 2 code - Cloud NDB](/mod2b-cloudndb) (3.x) - - FINISH: [Module 3 code - Cloud Datastore](/mod3b-datastore) (3.x) - - RECOMMENDED: - - Module 7 - add App Engine (push) tasks - - OTHER OPTIONS (in somewhat priority order): - - Module 11 - migrate to Cloud Functions - - Module 5 - migrate to Cloud Run container with Cloud Buildpacks - - Module 4 - migrate to Cloud Run container with Docker - - [Module 11 codelab](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-11-functions?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrcloudfuncs_sms_202006&utm_content=-): **Migrate from App Engine to Cloud Functions** - **Optional** migration - Recommended for small apps or for breaking up large apps into multiple microservices - Python 3 only - - START: [Module 2 code - Cloud NDB](/mod2b-cloudndb) (3.x) - - FINISH: [Module 11 code - Cloud Functions](/mod11-functions) (3.x) + - START: [Module 2 code - Cloud NDB](/mod2b-cloudndb) + - FINISH: [Module 11 code - Cloud Functions](/mod11-functions) - RECOMMENDED: - - Module 7 - add App Engine (push) tasks + - Module 7 - add App Engine (push) tasks (and migrate to Cloud Tasks in Module 8) + - Module 12 - add App Engine `memcache` (and migrate to Cloud Memorystore in Module 13) + - Module 15 - add App Engine `blobstore` (and migrate to Cloud Storage in Module 16) - OTHER OPTIONS (in somewhat priority order): - Module 5 - migrate to Cloud Run container with Cloud Buildpacks - - Module 4 - migrate to Cloud Run container with Docker - - Module 3 - migrate to Cloud Datastore ## Considerations for mobile developers From d66bb05459cb254e9698f4c3f7d56196dd4be6af Mon Sep 17 00:00:00 2001 From: wesley chun Date: Fri, 15 Jul 2022 19:39:49 -0700 Subject: [PATCH 33/56] add mods 18 & 19: pull queues to Pub/Sub --- README.md | 87 +++++++++++++++++---------- mod18-gaepull/.gcloudignore | 80 +++++++++++++++++++++++++ mod18-gaepull/README.md | 3 + mod18-gaepull/app.yaml | 21 +++++++ mod18-gaepull/appengine_config.py | 20 +++++++ mod18-gaepull/main.py | 83 ++++++++++++++++++++++++++ mod18-gaepull/queue.yaml | 3 + mod18-gaepull/requirements.txt | 1 + mod18-gaepull/templates/index.html | 25 ++++++++ mod19-pubsub/.gcloudignore | 80 +++++++++++++++++++++++++ mod19-pubsub/README.md | 5 ++ mod19-pubsub/app.yaml | 15 +++++ mod19-pubsub/main.py | 96 ++++++++++++++++++++++++++++++ mod19-pubsub/requirements.txt | 3 + mod19-pubsub/templates/index.html | 25 ++++++++ mod7-gaetasks/README.md | 4 +- 16 files changed, 516 insertions(+), 35 deletions(-) create mode 100644 mod18-gaepull/.gcloudignore create mode 100644 mod18-gaepull/README.md create mode 100644 mod18-gaepull/app.yaml create mode 100644 mod18-gaepull/appengine_config.py create mode 100644 mod18-gaepull/main.py create mode 100644 mod18-gaepull/queue.yaml create mode 100644 mod18-gaepull/requirements.txt create mode 100644 mod18-gaepull/templates/index.html create mode 100644 mod19-pubsub/.gcloudignore create mode 100644 mod19-pubsub/README.md create mode 100644 mod19-pubsub/app.yaml create mode 100644 mod19-pubsub/main.py create mode 100644 mod19-pubsub/requirements.txt create mode 100644 mod19-pubsub/templates/index.html diff --git a/README.md b/README.md index 43dd9c9..d98f789 100644 --- a/README.md +++ b/README.md @@ -6,25 +6,25 @@ # Modernizing Google Cloud serverless compute applications ### To the latest Cloud services and serverless platforms -This is the corresponding repository to the [Serverless Migration Station](https://bit.ly/3xk2Swi) video series whose goal is to help users on a Google Cloud serverless compute platform modernize to newer Cloud products or other serverless compute platforms. Each modernization migration aims to feature a video, codelab (self-paced, hands-on tutorial), and code samples. The content initially focuses on App Engine and Google's earliest Cloud users. Read more about the [codelabs in this announcement](https://developers.googleblog.com/2021/03/modernizing-your-google-app-engine-applications.html?utm_source=ext&utm_medium=partner&utm_campaign=CDR_wes_aap-serverless_modernizegae_codelabsannounce_201031&utm_content=-) as well as [this one introducing the video series](https://developers.googleblog.com/2021/06/introducing-serverless-migration.html?utm_source=blog&utm_medium=partner&utm_campaign=CDR_wes_aap-serverless_smsintro_201023). +This is the corresponding repository to the [Serverless Migration Station](https://bit.ly/3xk2Swi) video series whose goal is to help users on a Google Cloud serverless compute platform modernize to other Cloud or serverless products. Modernization steps generally feature a video, codelab (self-paced, hands-on tutorial), and code samples. The content initially focuses on App Engine and Google's earliest Cloud users. Read more about the [codelabs in this announcement](https://developers.googleblog.com/2021/03/modernizing-your-google-app-engine-applications.html?utm_source=ext&utm_medium=partner&utm_campaign=CDR_wes_aap-serverless_modernizegae_codelabsannounce_201031&utm_content=-) as well as [this one introducing the video series](https://developers.googleblog.com/2021/06/introducing-serverless-migration.html?utm_source=blog&utm_medium=partner&utm_campaign=CDR_wes_aap-serverless_smsintro_201023). This repo is for Python developers; there is another repo for Java developers. -[Google App Engine](https://cloud.google.com/appengine) (Standard) has undergone significant changes between the legacy and next generation platforms. To address this, we've created a set of codelabs (free, online, self-paced, hands-on tutorials) and corresponding videos (when available) to show developers how to perform individual migrations they can apply to modernize their apps for the latest runtimes, with this repo managing the samples from those codelabs. Codelab content typically falls into one of these three topics: +[Google App Engine](https://cloud.google.com/appengine) (Standard) has undergone significant changes between the legacy and next generation platforms. To address this, we've created a set of resources showing developers how to perform individual migrations that can be applied to modernize their apps for the latest runtimes. The content falls into one of these topics: -1. Migrate from a legacy App Engine service to a Cloud equivalent product -1. Migrate to a newer Cloud product, service, or API client library -1. Migrate to another or newer Google Cloud serverless compute platform +1. Migrate from a legacy App Engine service to a similar Cloud product +1. Shift to another Cloud serverless compute platform (e.g., from App Engine to Cloud Run) +1. General app, data, or service migration steps and best practices -Each codelab begins with a "START" code base then walks developers through that migration step, resulting in a "FINISH" repo. If you make any mistakes along the way, you can always go back to START or compare your code with what's in the FINISH folder to see the differences. We also want to help you port to the Python 3 runtime, so some codelabs contain a bonus section for that purpose. +Each codelab begins with a sample app in a "START" repo folder then walks developers through that migration, resulting in code in a "FINISH" repo. If you make mistakes along the way, you can always go back to START or compare your code with what's in the corresponding FINISH folder. The baseline apps are in Python 2, and since we also want to help you port to Python 3, some codelabs contain additional steps to do so. -> **NOTE:** These migrations are *typically* aimed at Python 2 users -> 1. *Python 3.x App Engine users*: You're *already* on the next-gen platform, so only look for **non**-legacy service migrations +> **NOTEs:** +> 1. These migrations are *typically* aimed at our earliest users, e.g., Python 2 +> 1. *Python 3.x App Engine users*: You're *already* on the next-gen platform, so you would focus on migrating away from the legacy bundled services > 1. *Python 2.5 App Engine users*: to revive apps from the original 2.5 runtime, [deprecated in 2013](http://googleappengine.blogspot.com/2013/03/python-25-thanks-for-good-times.html) and [shutdown in 2017](https://cloud.google.com/appengine/docs/standard/python/python25), you must [migrate from `db` to `ndb`](http://cloud.google.com/appengine/docs/standard/python/ndb/db_to_ndb) and get those apps running on Python 2.7 before attempting these migrations. ## Prerequisites -- A Google account (Google Workspace/G Suite accounts may require administrator approval) -- A Google Cloud (GCP) project with an active billing account +- A Google account and Cloud (GCP) project with an active billing account - Familiarity with operating system terminal/shell commands - Familiarity with developing & deploying Python 2 apps to App Engine - General skills in Python 2 and 3 @@ -32,34 +32,30 @@ Each codelab begins with a "START" code base then walks developers through that ## Cost -App Engine, Cloud Functions, and Cloud Run are not free services. While you may not have needed to enable billing in App Engine's early days, [all applications now require an active billing account](https://cloud.google.com/appengine/docs/standard/payment-instrument) backed by a financial instrument (usually a credit card). Don't worry, App Engine (and other GCP products) still have an ["Always Free" tier](https://cloud.google.com/free/docs/gcp-free-tier#free-tier-usage-limits) and as long as you stay within those limits, you won't incur any charges. Also check the App Engine [pricing](https://cloud.google.com/appengine/pricing) and [quotas](https://cloud.google.com/appengine/quotas) pages for more information. +App Engine, Cloud Functions, and Cloud Run are not free services. While you may not have enabled billing in App Engine's early days, [all applications now require an active billing account](https://cloud.google.com/appengine/docs/standard/payment-instrument) backed by a financial instrument (usually a credit card). Don't worry, App Engine (and other GCP products) still have an ["Always Free" tier](https://cloud.google.com/free/docs/gcp-free-tier#free-tier-usage-limits) and as long as you stay within those limits, you won't incur any charges. Also check the App Engine [pricing](https://cloud.google.com/appengine/pricing) and [quotas](https://cloud.google.com/appengine/quotas) pages for more information. Furthermore, deploying to GCP serverless platforms incur [minor build and storage costs](https://cloud.google.com/appengine/pricing#pricing-for-related-google-cloud-products). [Cloud Build](https://cloud.google.com/build/pricing) has its own free quota as does [Cloud Storage](https://cloud.google.com/storage/pricing#cloud-storage-always-free). For greater transparency, Cloud Build builds your application image which is than sent to the [Cloud Container Registry](https://cloud.google.com/container-registry/pricing), or [Artifact Registry](https://cloud.google.com/artifact-registry/pricing), its successor; storage of that image uses up some of that (Cloud Storage) quota as does network egress when transferring that image to the service you're deploying to. However you may live in region that does not have such a free tier, so be aware of your storage usage to minimize potential costs. (You may look at what storage you're using and how much, including deleting build artifacts via [your Cloud Storage browser](https://console.cloud.google.com/storage/browser).) ## Why -In App Engine's early days, users wanted Google to make the platform more flexible for developers and make their apps more portable. As a result, the team made significant changes to its 2nd-generation service which [launched in 2018](https://cloud.google.com/blog/products/gcp/introducing-app-engine-second-generation-runtimes-and-python-3-7). As a result, there are no longer any built-in services, allowing users to select from standalone GCP products or best-of-breed 3rd-party tools used by the broader community. Summary: +App Engine initially [elaunched in 2008](http://googleappengine.blogspot.com/2008/04/introducing-google-app-engine-our-new.html) ([video](http://youtu.be/3Ztr-HhWX1c)), providing a suite of bundled services making it convenient for developers to access a database (Datastore), caching (Memcache), independent task execution (TaskQueue), large "blob" storage (Blobstore) to allow for end-user file uploads or to serve large media files, and other companion services. However, apps leveraging those services can only run their apps on App Engine. -- **Legacy platform**: *Python 2* only, legacy built-in services -- **Next generation**: *Python 3* only, use standalone services, flexible platform, some* built-in services available -* see [Legacy services](#accessing-legacy-services-in-second-generation) section below +To increase app portability, its 2nd-generation service [launched in 2018](https://cloud.google.com/blog/products/gcp/introducing-app-engine-second-generation-runtimes-and-python-3-7), initially removing those legacy bundled services. The main reason to move to the 2nd generation service is that it allows developers to upgrade to the latest language runtimes, such as moving from Python 2 to 3 or Java 8 to 17. Unfortunately, it was mutually exclusive to do so, meaning while you could upgrade language releases, you lost access to those bundled services, making it a showstopper for many users. -While the 2nd-gen platform is more flexible, users of the legacy platform have two challenges: +However, due to their popularity _and_ to help users upgrade, the App Engine team restored access to many (but not all) of those services in Fall 2021. For more on this, see the [Legacy services](#accessing-legacy-services-in-second-generation) section below. As Google is continually striving to have the most [open cloud](https://cloud.google.com/open-cloud) on the market, and while many of those services are now available again, apps can _still_ be more portable if they migrated away from the legacy services to similar Cloud or 3rd-party oferings. Another issue with the bundled services is that they're only available in 2nd generation runtimes that have a 1st generation service (Python, Java, Go, PHP), excluding 2nd generation-only runtimes like Ruby and Node.js. -1. Migrate to unbundled/standalone services -1. Porting to a modern language release +Once apps have moved away from App Engine bundled services to similar Cloud or 3rd-party services. apps are portable enough to: -Neither upgrade may be particularly straightforward and can only be done serially. On top of this, direct replacements are not available for all formerly built-in services; alternatives come in 3 flavors: +1. Run on the [2nd generation App Engine service](https://cloud.google.com/appengine/docs/standard/runtimes) +1. Shift across to other serverless platforms, like [Cloud Functions](https://developers.googleblog.com/2022/04/how-can-app-engine-users-take-advantage-of-cloud-functions.html?utm_source=blog&utm_medium=partner&utm_campaign=CDR_wes_aap-serverless_mgrcloudfuncs_sms_202006) or Cloud Run ([with](https://developers.googleblog.com/2021/08/containerizing-google-app-engine-apps-for-cloud-run.html?utm_source=blog&utm_medium=partner&utm_campaign=CDR_wes_aap-serverless_mgrcrdckr_sms_201017) or [without](https://developers.googleblog.com/2021/09/an-easier-way-to-move-your-app-engine-to-cloud-run.html?utm_source=blog&utm_medium=partner&utm_campaign=CDR_wes_aap-serverless_mgrcrbdpk_sms_201031) Docker), or +1. Move to VM-based services like [GKE](https://cloud.google.com/gke) or [Compute Engine](https://cloud.google.com/compute), or to other compute platforms -1. **Direct replacement**: Legacy services which matured into their own Cloud products *(e.g., App Engine Datastore is now [Cloud Datastore](http://cloud.google.com/datastore))* -1. **Partial replacement**: Some aspects of legacy services *(e.g., [Cloud Tasks](http://cloud.google.com/tasks) supports App Engine **push** tasks; for pull tasks, [Cloud Pub/Sub](http://cloud.google.com/pubsub) is recommended; use of [Cloud MemoryStore with REDIS](http://cloud.google.com/memorystore/docs/redis) as an alternative for Memcache)* -1. **No replacement**: No direct replacement available, so third-party or other tools recommended *(e.g., Search, Images, Users, Email)* - -These are the challenges developers are facing, so the purpose of this content is to reduce the friction in this process and make things more prescriptive. Review the [runtimes chart](https://cloud.google.com/appengine/docs/standard/runtimes) to see the legacy services and current migration recommendation. The [migration guide overview](https://cloud.google.com/appengine/docs/standard/python/migrate-to-python3/migrating-services) has more information. - -> **NOTE:** App Engine ([Flexible](https://cloud.google.com/appengine/docs/flexible/python/runtime?hl=en#interpreter)) is a next-gen service but is not within the scope of these tutorials. Curious developers can compare App Engine [Standard vs. Flexible](https://cloud.google.com/appengine/docs/the-appengine-environments) to find out more. Also, many of the Flexible use cases can now be handled by [Cloud Run](http://cloud.run). +> **NOTEs:** +> 1. App Engine ([Flexible](https://cloud.google.com/appengine/docs/flexible/python/runtime?hl=en#interpreter)) is a next-gen service but is not within the scope of these tutorials. Curious developers can compare App Engine [Standard vs. Flexible](https://cloud.google.com/appengine/docs/the-appengine-environments) to find out more. +> 1. Many use cases for Flexible or a desire for containerization can be handled by [Cloud Run](http://cloud.run). +> 1. Small apps or large monolithic apps broken up into multiple, independent microservices can consider migrating to [Cloud Functions](https://cloud.google.com/functions). ## Progression (what order to do things) @@ -80,13 +76,15 @@ Topic | Module ordering | Description --- | --- | --- Baseline | 0 ⇒ 1 | Not a migration but a description of the baseline application (review this material before doing any migrations) Web framework | 1 ⇒ _everything else_ | Current App Engine runtimes do not come with a web framework, so this must be the first migration performed. All migrations below can be performed after this one. +Bundled services | 17 | This module is for Python bundled services users interested in how to access those services from Python 3. Datastore | 2 [⇒ 3 [⇒ 6]] | Moving off App Engine `ndb` makes your apps more portable, so the **Module 2** Cloud NDB migration is _recommended_. **Module 3:** Migrating to Cloud Datastore (Firestore in Datastore mode) is _optional_ and only recommended if you have other code using Cloud Datastore. **Module 6**: Migrating to Cloud Firestore (Native mode) is generally _not recommended_ unless you must have the Firebase features it has, and those features will eventually be integrated into Cloud Datastore. (Push) Task Queues | [7 ⇒] 8 [⇒ 9] | Moving off App Engine `taskqueue` makes your apps more portable, so the **Module 8** Cloud Tasks migration is _recommended_ for those using push tasks. Those unfamiliar with push tasks should do **Module 7** first to add push tasks to the sample app. **Module 9:** Migrating to Cloud Datastore (Firestore in Datastore mode), Cloud Tasks (v2), and Python 3 is _optional_ and only recommended if you have other code using Cloud Datastore and considering upgrading to Python 3. +(Pull) Task Queues | [18 ⇒] 19 | Moving off App Engine `taskqueue` makes your apps more portable, so the **Module 19** Cloud Pub/Sub migration is _recommended_ for those using pull tasks. The app is also ported to Python 3. Those unfamiliar with pull tasks should do **Module 18** first to add pull tasks to the sample app. Memcache | [12 ⇒] 13 | Moving off App Engine `memcache` makes your apps more portable, so the **Module 13** Cloud Memorystore (for Redis) migration is _recommended_ for those using `memcache`. Those unfamiliar with `memcache` should do **Module 12** first to add its usage to the sample app. Cloud Functions | 11 | Cloud Functions does not support Python 2, so after the Module 1 migration, you need to upgrade your app to Python 3 before attempting this migration, recommended if you have a very small App Engine app, or it has only one function/feature. Cloud Run | 4 or 5 | **Module 4** covers migrating to Cloud Run with Docker. Those unfamiliar with containers or do not wish to create/maintain a `Dockerfile` should do **Module 5**. Those doing **Module 4** will get additional information about Cloud Run in **Module 5** not covered in **Module 4**. Blobstore | [15 ⇒] 16 | Moving off App Engine `blobstore` makes your apps more portable, so the **Module 16** Cloud Storage migration is _recommended_ for those using `blobstore`. Those unfamiliar with `blobstore` should do **Module 15** first to add its usage to the sample app. -general migration | [6 ⇒] 10 [⇒ 14] | This series is more generic and not targeting a specific feature migration, but rather if you need to migrate your App Engine apps from one running project to another. It starts with **Module 6** if you need to migrate your code, say from Datastore to Firestore. **Module 10** is if you need to migrate your data from one project to another, and finally, **Module 14** is after you're done migrating your app, your data, or both, and need to migrate a running service on one GCP project to another. +General migration | 6 ⇒ 10 ⇒ 14 | This series is more generic and not targeting a specific feature migration, but rather if you need to migrate your App Engine apps from one running project to another. It starts with **Module 6** if you need to migrate your code, say from Datastore to Firestore. **Module 10** is if you need to migrate your data from one project to another, and finally, **Module 14** is after you're done migrating your app, your data, or both, and need to migrate a running service on one GCP project to another. ## Migration modules @@ -111,11 +109,13 @@ Module | Topic | Video | Codelab | START here | FINISH here 10|Migrate Datastore/Firestore data to another project| _TBD_ | _N/A_ | _N/A_ | _TBD_ 11|Migrate to Cloud Functions| [link](https://twitter.com/googledevs/status/1520206298834481153?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrcloudfuncs_sms_202006&utm_content=-) | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-11-functions?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrcloudfuncs_sms_202006&utm_content=-) | Module 2 [code](/mod2b-cloudndb) (3.x) | Module 11 [code](/mod11-functions) (3.x) 12|Add App Engine `memcache`| [link](https://twitter.com/googledevs/status/1527303061953126402?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrmemcache_sms_202006&utm_content=-) | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-12-memcache?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrmemcache_sms_202006&utm_content=-) | Module 1 [code](/mod1-flask) (2.x) | Module 12 [code](/mod12-memcache) (2.x) & [code](/mod12b-memcache) (3.x) -13|Migrate to Cloud Memorystore| _TBD_ | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-13-memorystore?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrmemorystore_sms_202029&utm_content=-) | Module 12 [code](/mod12-memcache) (2.x) & [code](/mod12b-memcache) (3.x) | Module 13 [code](/mod13a-memorystore) (2.x) & [code](/mod13b-memorystore) (3.x) +13|Migrate to Cloud Memorystore| [link](https://twitter.com/googledevs/status/1537132939426799616?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrmemorystore_sms_202029&utm_content=-) | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-13-memorystore?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrmemorystore_sms_202029&utm_content=-) | Module 12 [code](/mod12-memcache) (2.x) & [code](/mod12b-memcache) (3.x) | Module 13 [code](/mod13a-memorystore) (2.x) & [code](/mod13b-memorystore) (3.x) 14|Migrate service between projects| _TBD_ | _TBD_ | _TBD_ | _TBD_ 15|Add App Engine `blobstore`| _TBD_ | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-15-blobstore?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrblobstore_sms_202029&utm_content=-) | Module 0 [code](/mod0-baseline) (2.x) | Module 15 [code](/mod15-blobstore) (2.x) 16|Migrate to Cloud Storage| _TBD_ | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-16-cloudstorage?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrcloudstorage_sms_202029&utm_content=-) | Module 15 [code](/mod15-blobstore) (2.x) | Module 16 [code](/mod16-cloudstorage) (2.x & 3.x) 17|Migrate to Python 3 bundled services| _TBD_ | _TBD_ | Module 1 [code](/mod1-flask) (2.x) | Module 1 [code](/mod1b-flask) (3.x) +18|Add App Engine `taskqueue` pull tasks| _TBD_ | _TBD_ | Module 1 [code](/mod1-flask) (2.x) | Module 18 [code](/mod18-gaepull) (2.x) +19|Migrate to Cloud Pub/Sub| _TBD_ | _TBD_ | Module 18 [code](/mod18-gaepull) (2.x) | Module 19 [code](/mod9-pubsub) (3.x) ### Table of contents @@ -149,15 +149,18 @@ If there is a logical codelab to do immediately after completing one, they will - **Not a migration**: add GAE Task Queues to prepare for migration to Cloud Tasks - Python 2 - START: [Module 1 code - Framework](/mod1-flask) - - FINISH: [Module 7 code - GAE Task Queues](/mod7-gaetasks) + - FINISH: [Module 7 code - GAE TaskQueue (push tasks)](/mod7-gaetasks) - NEXT: Module 8 - migrate App Engine push tasks to Cloud Tasks - [Module 8 codelab](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-8-cloudtasks?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrcloudtasks_sms_201112&utm_content=-): **Migrate from App Engine (push) Task Queues to [Cloud Tasks](http://cloud.google.com/tasks) v1** - - Note only covers push tasks... pull tasks will be handled in a different codelab. - Python 2 - - START: [Module 7 code - GAE Task Queues](/mod7-gaetasks) + - START: [Module 7 code - GAE TaskQueue (push tasks)](/mod7-gaetasks) - FINISH: [Module 8 code - Cloud Tasks](/mod8-cloudtasks) - - NEXT: Module 9 - migrate to Python 3 and Cloud Datastore + - RECOMMENDED: + - Module 9 - migrate to Python 3 and Cloud Datastore + - Module 18 - add App Engine TaskQueue pull tasks (and migrate to Cloud Pub/Sub in Module 19) + - Module 12 - add App Engine Memcache (and migrate to Cloud Memorystore in Module 13) + - Module 15 - add App Engine Blobstore (and migrate to Cloud Storage in Module 16) - **Module 9 codelab** (TBD): **Migrate a Python 2 Cloud NDB & Cloud Tasks (v1) app to a Python 3 Cloud Datastore & Cloud Tasks (v2) app** - **Optional** migrations @@ -168,6 +171,24 @@ If there is a logical codelab to do immediately after completing one, they will - Python 3 - FINISH: _TBD_ - RECOMMENDED: + - Module 18 - add App Engine TaskQueue pull tasks (and migrate to Cloud Pub/Sub in Module 19) + - Module 12 - add App Engine Memcache (and migrate to Cloud Memorystore in Module 13) + - Module 15 - add App Engine Blobstore (and migrate to Cloud Storage in Module 16) + +- **Module 18 codelab** (TBD): **Add App Engine (pull) Task Queues to existing sample app** + - **Not a migration**: add GAE Task Queues to prepare for migration to Cloud Tasks + - Python 2 + - START: [Module 1 code - Framework](/mod1-flask) + - FINISH: [Module 18 code - GAE TaskQueue (pull tasks)](/mod18-gaepull) + - NEXT: Module 19 - migrate App Engine pull tasks to Cloud Pub/Sub + +- **Module 19 codelab** (TBD): **Migrate from App Engine (pull) Task Queues to [Cloud Pub/Sub](http://cloud.google.com/pubsub)** + - Python 2 + - START: [Module 18 code - GAE TaskQueue (pull tasks)](/mod18-gaepull) + - Python 3 + - FINISH: [Module 19 code - Cloud Pub/Sub](/mod19-pubsub) + - RECOMMENDED: + - Module 7 - add App Engine TaskQueue push tasks (and migrate to Cloud Tasks in Module 8) - Module 12 - add App Engine Memcache (and migrate to Cloud Memorystore in Module 13) - Module 15 - add App Engine Blobstore (and migrate to Cloud Storage in Module 16) @@ -283,7 +304,7 @@ If your original app users does *not* have a user interface, i.e., mobile backen ## Accessing legacy services in second generation -Many legacy App Engine first generation platform (Python 2, Java 8, PHP 5, and Go 1.11 & older) services are available ([as of Sep 2021](https://twitter.com/googledevs/status/1445916786755571712) for second generation runtimes (Python 3, Java 11, PHP 7, and Go 1.12 & newer) in a public preview. There are no videos or codelabs yet, however the Module 1 Flask migration using App Engine `ndb` [Python 2 sample ](/mod1-flask) is available in [Python 3](/mod1b-flask) if you have access. Similarly, Python 3 editions are also available for Modules 7 and 12 which add usage of App Engine `taskqueue` and `memcache`, respectively. +Many legacy App Engine first generation platform (Python 2, Java 8, PHP 5, and Go 1.11 & older) services are available ([as of Sep 2021](https://twitter.com/googledevs/status/1445916786755571712) for second generation runtimes (Python 3, Java 11, PHP 7, and Go 1.12 & newer) in a public preview. There are no videos or codelabs yet, however the Module 1 Flask migration using App Engine `ndb` [Python 2 sample ](/mod1-flask) is available in [Python 3](/mod1b-flask) if you have access. Similarly, Python 3 editions are also available for Modules 7 and 12 which add usage of App Engine `taskqueue` and `memcache`, respectively. Also see the [documentation on accessing bundled services from Python 3](https://cloud.google.com/appengine/docs/standard/python3/services/access). ## References diff --git a/mod18-gaepull/.gcloudignore b/mod18-gaepull/.gcloudignore new file mode 100644 index 0000000..b4dbc7e --- /dev/null +++ b/mod18-gaepull/.gcloudignore @@ -0,0 +1,80 @@ +# This file specifies files that are *not* uploaded to Google Cloud Platform +# using gcloud. It follows the same syntax as .gitignore, with the addition of +# "#!include" directives (which insert the entries of the given .gitignore-style +# file at that point). +# +# For more information, run: +# $ gcloud topic gcloudignore +# +.gcloudignore + +# Source code control files +.git/ +.gitignore +.hgignore +.hg/ + +# README/text files +LICENSE +*.md + +# Tests/results (not in .gitignore) +noxfile.py +test_translate.py +pylintrc +pylintrc.test + +# most of .gitignore (except `lib`) +# +# Python +*.py[cod] +__pycache__/ +/setup.cfg + +# C extensions +*.so + +# Packages +*.egg +*.egg-info +dist +build +eggs +.eggs +parts +bin +var +sdist +develop-eggs +.installed.cfg +lib64 +*.tgz + +# Installer logs +pip-log.txt + +# Tests/results +.nox/ +.pytest_cache/ +.cache +.pytype +.coverage +coverage.xml +*sponge_log.xml +system_tests/local_test_setup + +# Mac +.DS_Store + +# IDEs/editors +*.sw[op] +*~ +.vscode +.idea + +# Built documentation +docs/_build +docs.metadata + +# Virtual environment +env/ diff --git a/mod18-gaepull/README.md b/mod18-gaepull/README.md new file mode 100644 index 0000000..19d98a3 --- /dev/null +++ b/mod18-gaepull/README.md @@ -0,0 +1,3 @@ +# Module 18 - Add usage of App Engine TaskQueue (pull tasks) to NDB Flask sample app + +This repo folder is the corresponding Python 2 code to the [Module 18 codelab](http://g.co/codelabs/pae-migrate-gaepull). The tutorial STARTs with the Python 2 code in the [Module 1 repo folder](/mod1-flask) and leads developers through adding usage of pull tasks via App Engine TaskQueue, culminating in the code in this folder. diff --git a/mod18-gaepull/app.yaml b/mod18-gaepull/app.yaml new file mode 100644 index 0000000..4b7d197 --- /dev/null +++ b/mod18-gaepull/app.yaml @@ -0,0 +1,21 @@ +# Copyright 2022 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +runtime: python27 +threadsafe: yes +api_version: 1 + +handlers: +- url: /.* + script: main.app diff --git a/mod18-gaepull/appengine_config.py b/mod18-gaepull/appengine_config.py new file mode 100644 index 0000000..b0decff --- /dev/null +++ b/mod18-gaepull/appengine_config.py @@ -0,0 +1,20 @@ +# Copyright 2022 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from google.appengine.ext import vendor + +# Set PATH to your libraries folder. +PATH = 'lib' +# Add libraries installed in the PATH folder. +vendor.add(PATH) diff --git a/mod18-gaepull/main.py b/mod18-gaepull/main.py new file mode 100644 index 0000000..df30f37 --- /dev/null +++ b/mod18-gaepull/main.py @@ -0,0 +1,83 @@ +# Copyright 2022 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from flask import Flask, render_template, request +from google.appengine.api import taskqueue +from google.appengine.ext import ndb + +HOUR = 3600 +LIMIT = 10 +TASKS = 1000 +QUEUE = 'pullq' +app = Flask(__name__) + + +class Visit(ndb.Model): + 'Visit entity registers visitor IP address & timestamp' + visitor = ndb.StringProperty() + timestamp = ndb.DateTimeProperty(auto_now_add=True) + +def store_visit(remote_addr, user_agent): + 'create new Visit in Datastore and queue request to bump visitor count' + Visit(visitor='{}: {}'.format(remote_addr, user_agent)).put() + q = taskqueue.Queue(QUEUE) + q.add(taskqueue.Task(payload=remote_addr, method='PULL')) + +def fetch_visits(limit): + 'get most recent visits' + return Visit.query().order(-Visit.timestamp).fetch(limit) + + +class VisitorCount(ndb.Model): + visitor = ndb.StringProperty(repeated=False, required=True) + counter = ndb.IntegerProperty() + +def fetch_counts(limit): + 'get top visitors' + return VisitorCount.query().order(-VisitorCount.counter).fetch(limit) + + +@app.route('/log') +def log_visitors(): + 'worker processes recent visitor counts and updates them in Datastore' + # tally recent visitor counts from queue then delete those tasks + tallies = {} + q = taskqueue.Queue(QUEUE) + tasks = q.lease_tasks(HOUR, TASKS) + for task in tasks: + visitor = task.payload + tallies[visitor] = tallies.get(visitor, 0) + 1 + q.delete_tasks(tasks) + + # increment those counts in Datastore and return + for visitor in tallies: + counter = VisitorCount.query(VisitorCount.visitor == visitor).get() + if not counter: + counter = VisitorCount(visitor=visitor, counter=0) + counter.put() + counter.counter += tallies[visitor] + counter.put() + return 'DONE (with %d task[s] logging %d visitor[s])\r\n' % (len(tasks), len(tallies)) + + +@app.route('/') +def root(): + 'main application (GET) handler' + store_visit(request.remote_addr, request.user_agent) + context = { + 'limit': LIMIT, + 'visits': fetch_visits(LIMIT), + 'counts': fetch_counts(LIMIT), + } + return render_template('index.html', **context) diff --git a/mod18-gaepull/queue.yaml b/mod18-gaepull/queue.yaml new file mode 100644 index 0000000..f78fe75 --- /dev/null +++ b/mod18-gaepull/queue.yaml @@ -0,0 +1,3 @@ +queue: +- name: pullq + mode: pull diff --git a/mod18-gaepull/requirements.txt b/mod18-gaepull/requirements.txt new file mode 100644 index 0000000..7e10602 --- /dev/null +++ b/mod18-gaepull/requirements.txt @@ -0,0 +1 @@ +flask diff --git a/mod18-gaepull/templates/index.html b/mod18-gaepull/templates/index.html new file mode 100644 index 0000000..b88f744 --- /dev/null +++ b/mod18-gaepull/templates/index.html @@ -0,0 +1,25 @@ + + + +VisitMe Example + + +

VisitMe example

+ +

Top {{ limit }} visitors

+ + +{% for count in counts %} + +{% endfor %} +
VisitorVisits
{{ count.visitor|e }}{{ count.counter }}
+ +

Last {{ limit }} visits

+
    +{% for visit in visits %} +
  • {{ visit.timestamp.ctime() }} from {{ visit.visitor }}
  • +{% endfor %} +
+ + + diff --git a/mod19-pubsub/.gcloudignore b/mod19-pubsub/.gcloudignore new file mode 100644 index 0000000..b4dbc7e --- /dev/null +++ b/mod19-pubsub/.gcloudignore @@ -0,0 +1,80 @@ +# This file specifies files that are *not* uploaded to Google Cloud Platform +# using gcloud. It follows the same syntax as .gitignore, with the addition of +# "#!include" directives (which insert the entries of the given .gitignore-style +# file at that point). +# +# For more information, run: +# $ gcloud topic gcloudignore +# +.gcloudignore + +# Source code control files +.git/ +.gitignore +.hgignore +.hg/ + +# README/text files +LICENSE +*.md + +# Tests/results (not in .gitignore) +noxfile.py +test_translate.py +pylintrc +pylintrc.test + +# most of .gitignore (except `lib`) +# +# Python +*.py[cod] +__pycache__/ +/setup.cfg + +# C extensions +*.so + +# Packages +*.egg +*.egg-info +dist +build +eggs +.eggs +parts +bin +var +sdist +develop-eggs +.installed.cfg +lib64 +*.tgz + +# Installer logs +pip-log.txt + +# Tests/results +.nox/ +.pytest_cache/ +.cache +.pytype +.coverage +coverage.xml +*sponge_log.xml +system_tests/local_test_setup + +# Mac +.DS_Store + +# IDEs/editors +*.sw[op] +*~ +.vscode +.idea + +# Built documentation +docs/_build +docs.metadata + +# Virtual environment +env/ diff --git a/mod19-pubsub/README.md b/mod19-pubsub/README.md new file mode 100644 index 0000000..6640585 --- /dev/null +++ b/mod19-pubsub/README.md @@ -0,0 +1,5 @@ +# Module 19 - Migrate from App Engine `taskqueue` (pull tasks) to Cloud Pub/Sub (and Python 3) + +This repo folder is the corresponding Python 2 code to the [Module 19 codelab](http://g.co/codelabs/pae-migrate-pubsub). The tutorial STARTs with the Python 2 code in the [Module 18 repo folder](/mod18-gaepull) and leads developers through its migration from pull tasks via App Engine `taskqueue` to Cloud Pub/Sub and Python 3, culminating in the code in this folder. The migration from App Engine `ndb` to Cloud NDB, covered in Module 2, also takes place. + +**NOTE:** While this solution Python 2 compatible, a library conflict on App Engine servers does not allow it to run properly under Python 2 on App Engine.) diff --git a/mod19-pubsub/app.yaml b/mod19-pubsub/app.yaml new file mode 100644 index 0000000..9aac415 --- /dev/null +++ b/mod19-pubsub/app.yaml @@ -0,0 +1,15 @@ +# Copyright 2022 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +runtime: python310 diff --git a/mod19-pubsub/main.py b/mod19-pubsub/main.py new file mode 100644 index 0000000..7791ec6 --- /dev/null +++ b/mod19-pubsub/main.py @@ -0,0 +1,96 @@ +# Copyright 2022 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from flask import Flask, render_template, request +import google.auth +from google.cloud import ndb, pubsub + +LIMIT = 10 +TASKS = 1000 +TOPIC = 'pullq' +SBSCR = 'worker' + +app = Flask(__name__) +ds_client = ndb.Client() +ppc_client = pubsub.PublisherClient() +psc_client = pubsub.SubscriberClient() +_, PROJECT_ID = google.auth.default() +TOP_PATH = ppc_client.topic_path(PROJECT_ID, TOPIC) +SUB_PATH = psc_client.subscription_path(PROJECT_ID, SBSCR) + + +class Visit(ndb.Model): + 'Visit entity registers visitor IP address & timestamp' + visitor = ndb.StringProperty() + timestamp = ndb.DateTimeProperty(auto_now_add=True) + +def store_visit(remote_addr, user_agent): + 'create new Visit in Datastore and queue request to bump visitor count' + with ds_client.context(): + Visit(visitor='{}: {}'.format(remote_addr, user_agent)).put() + ppc_client.publish(TOP_PATH, remote_addr.encode('utf-8')) + +def fetch_visits(limit): + 'get most recent visits' + with ds_client.context(): + return Visit.query().order(-Visit.timestamp).fetch(limit) + + +class VisitorCount(ndb.Model): + visitor = ndb.StringProperty(repeated=False, required=True) + counter = ndb.IntegerProperty() + +def fetch_counts(limit): + 'get top visitors' + with ds_client.context(): + return VisitorCount.query().order(-VisitorCount.counter).fetch(limit) + + +@app.route('/log') +def log_visitors(): + 'worker processes recent visitor counts and updates them in Datastore' + # tally recent visitor counts from queue then delete those tasks + tallies = {} + acks = set() + with psc_client: + rsp = psc_client.pull(subscription=SUB_PATH, max_messages=TASKS) + msgs = rsp.received_messages + for rcvd_msg in msgs: + acks.add(rcvd_msg.ack_id) + visitor = rcvd_msg.message.data.decode('utf-8') + tallies[visitor] = tallies.get(visitor, 0) + 1 + psc_client.acknowledge(subscription=SUB_PATH, ack_ids=acks) + + # increment those counts in Datastore and return + with ds_client.context(): + for visitor in tallies: + counter = VisitorCount.query(VisitorCount.visitor == visitor).get() + if not counter: + counter = VisitorCount(visitor=visitor, counter=0) + counter.put() + counter.counter += tallies[visitor] + counter.put() + return 'DONE (with %d task[s] logging %d visitor[s])\r\n' % (len(msgs), len(tallies)) + + +@app.route('/') +def root(): + 'main application (GET) handler' + store_visit(request.remote_addr, request.user_agent) + context = { + 'limit': LIMIT, + 'visits': fetch_visits(LIMIT), + 'counts': fetch_counts(LIMIT), + } + return render_template('index.html', **context) diff --git a/mod19-pubsub/requirements.txt b/mod19-pubsub/requirements.txt new file mode 100644 index 0000000..3dec648 --- /dev/null +++ b/mod19-pubsub/requirements.txt @@ -0,0 +1,3 @@ +flask +google-cloud-ndb +google-cloud-pubsub diff --git a/mod19-pubsub/templates/index.html b/mod19-pubsub/templates/index.html new file mode 100644 index 0000000..b88f744 --- /dev/null +++ b/mod19-pubsub/templates/index.html @@ -0,0 +1,25 @@ + + + +VisitMe Example + + +

VisitMe example

+ +

Top {{ limit }} visitors

+ + +{% for count in counts %} + +{% endfor %} +
VisitorVisits
{{ count.visitor|e }}{{ count.counter }}
+ +

Last {{ limit }} visits

+
    +{% for visit in visits %} +
  • {{ visit.timestamp.ctime() }} from {{ visit.visitor }}
  • +{% endfor %} +
+ + + diff --git a/mod7-gaetasks/README.md b/mod7-gaetasks/README.md index f9c4658..d76f17e 100644 --- a/mod7-gaetasks/README.md +++ b/mod7-gaetasks/README.md @@ -1,3 +1,3 @@ -# Module 7 - Add usage of App Engine `taskqueue` to Flask `ndb` sample app +# Module 7 - Add usage of App Engine TaskQueue (push tasks) to NDB Flask sample app -This repo folder is the corresponding Python 2 code to the [Module 7 codelab](http://g.co/codelabs/pae-migrate-gaetasks). The tutorial STARTs with the Python 2 code in the [Module 1 repo folder](/mod1-flask) and leads developers through adding usage of App Engine's `taskqueue`, culminating in the code in this folder. +This repo folder is the corresponding Python 2 code to the [Module 7 codelab](http://g.co/codelabs/pae-migrate-gaetasks). The tutorial STARTs with the Python 2 code in the [Module 1 repo folder](/mod1-flask) and leads developers through adding usage of push tasks via App Engine TaskQueue, culminating in the code in this folder. From 244aa6474bdc2fda8a3e6221e334c198d54461e0 Mon Sep 17 00:00:00 2001 From: wesley chun Date: Fri, 15 Jul 2022 23:51:30 -0700 Subject: [PATCH 34/56] update codelab links --- README.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index d98f789..1e82699 100644 --- a/README.md +++ b/README.md @@ -97,14 +97,14 @@ The table below summarizes migration module resources currently available along Module | Topic | Video | Codelab | START here | FINISH here --- | --- | --- | --- | --- | --- 0|Baseline app| [link](http://twitter.com/googledevs/status/1407755281786867714?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_smsintro_201023&utm_content=-) | _N/A_ (no tutorial; just review the code) | _N/A_ | Module 0 [code](/mod0-baseline) (2.x) -1|Migrate to Flask| [link](https://twitter.com/googledevs/status/1413273119071031299?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrwa2flsk_201008&u%20tm_content=-) | [link](http://g.co/codelabs/pae-migrate-flask) | Module 0 [code](/mod0-baseline) (2.x) | Module 1 [code](/mod1-flask) (2.x) (and [code](/mod1b-flask) (3.x)) -2|Migrate to Cloud NDB| [link](https://twitter.com/googledevs/status/1417892493383802883?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrcloudndb_201021&utm_content=-) | [link](http://g.co/codelabs/pae-migrate-cloudndb) | Module 1 [code](/mod1-flask) (2.x) | Module 2 [code](/mod2a-cloudndb) (2.x) & [code](/mod2b-cloudndb) (3.x) -3|Migrate to Cloud Datastore| [link](http://twitter.com/googledevs/status/1422966928910393347?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrcloudds_201003&utm_content=-) | [link](http://g.co/codelabs/pae-migrate-datastore) | Module 2 [code](/mod2a-cloudndb) (2.x) & [code](/mod2b-cloudndb) (3.x) | Module 3 [code](/mod3a-datastore) (2.x) & [code](/mod3b-datastore) (3.x) -4|Migrate to Cloud Run with Docker| [link](https://twitter.com/googledevs/status/1428041270702735362?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrcrdckr_sms_201017&utm_content=-)| [link](http://g.co/codelabs/pae-migrate-rundocker) | Module 2 [code](/mod2a-cloudndb) (2.x) & Module 3 [code](/mod3b-datastore) (3.x) | Module 4 [code](/mod4a-rundocker) (2.x) & [code](/mod4b-rundocker) (3.x) -5|Migrate to Cloud Run with Buildpacks| [link](https://twitter.com/googledevs/status/1433113274984271875?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrcrbdpk_sms_201031&utm_content=-) | [link](http://g.co/codelabs/pae-migrate-runbldpks) | Module 2 [code](/mod2b-cloudndb) (3.x) | Module 5 [code](/mod5-runbldpks) (3.x) +1|Migrate to Flask| [link](https://twitter.com/googledevs/status/1413273119071031299?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrwa2flsk_201008&u%20tm_content=-) | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-1-flask?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrwa2flsk_201008&utm_content=-) | Module 0 [code](/mod0-baseline) (2.x) | Module 1 [code](/mod1-flask) (2.x) (and [code](/mod1b-flask) (3.x)) +2|Migrate to Cloud NDB| [link](https://twitter.com/googledevs/status/1417892493383802883?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrcloudndb_201021&utm_content=-) | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-2-cloudndb?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrcloudndb_201021&utm_content=-) | Module 1 [code](/mod1-flask) (2.x) | Module 2 [code](/mod2a-cloudndb) (2.x) & [code](/mod2b-cloudndb) (3.x) +3|Migrate to Cloud Datastore| [link](http://twitter.com/googledevs/status/1422966928910393347?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrcloudds_201003&utm_content=-) | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-3-datastore?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrcloudds_201003&utm_content=-) | Module 2 [code](/mod2a-cloudndb) (2.x) & [code](/mod2b-cloudndb) (3.x) | Module 3 [code](/mod3a-datastore) (2.x) & [code](/mod3b-datastore) (3.x) +4|Migrate to Cloud Run with Docker| [link](https://twitter.com/googledevs/status/1428041270702735362?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrcrdckr_sms_201017&utm_content=-)| [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-4-rundocker?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrcrdckr_sms_201017&utm_content=-) | Module 2 [code](/mod2a-cloudndb) (2.x) & Module 3 [code](/mod3b-datastore) (3.x) | Module 4 [code](/mod4a-rundocker) (2.x) & [code](/mod4b-rundocker) (3.x) +5|Migrate to Cloud Run with Buildpacks| [link](https://twitter.com/googledevs/status/1433113274984271875?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrcrbdpk_sms_201031&utm_content=-) | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-5-runbldpks?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrcrbdpk_sms_201031&utm_content=-) | Module 2 [code](/mod2b-cloudndb) (3.x) | Module 5 [code](/mod5-runbldpks) (3.x) 6|Migrate to Cloud Firestore| _N/A_ | _N/A_ | Module 3 [code](/mod3b-datastore) (3.x) | _no work required; [Datastore upgrade automatic](https://cloud.google.com/datastore/docs/upgrade-to-firestore)_ -7|Add App Engine `taskqueue` push tasks| [link](https://twitter.com/googledevs/status/1443410302113099778?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrgaetasks_sms_201028&utm_content=-) | [link](http://g.co/codelabs/pae-migrate-gaetasks) | Module 1 [code](/mod1-flask) (2.x) | Module 7 [code](/mod7-gaetasks) (2.x) & [code](/mod7b-gaetasks) (3.x) -8|Migrate to Cloud Tasks| [link](https://twitter.com/googledevs/status/1450960021018267656?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrcloudtasks_sms_201112&utm_content=-) | [link](http://g.co/codelabs/pae-migrate-cloudtasks) | Module 7 [code](/mod7-gaetasks) (2.x) | Module 8 [code](/mod8-cloudtasks) (2.x) +7|Add App Engine `taskqueue` push tasks| [link](https://twitter.com/googledevs/status/1443410302113099778?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrgaetasks_sms_201028&utm_content=-) | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-7-gaetasks?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrgaetasks_sms_201028&utm_content=-) | Module 1 [code](/mod1-flask) (2.x) | Module 7 [code](/mod7-gaetasks) (2.x) & [code](/mod7b-gaetasks) (3.x) +8|Migrate to Cloud Tasks| [link](https://twitter.com/googledevs/status/1450960021018267656?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrcloudtasks_sms_201112&utm_content=-) | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-8-cloudtasks?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrcloudtasks_sms_201112&utm_content=-) | Module 7 [code](/mod7-gaetasks) (2.x) | Module 8 [code](/mod8-cloudtasks) (2.x) 9|Migrate to Python 3, Cloud Datastore & Cloud Tasks v2| _TBD_ | _TBD_ | Module 8 [code](/mod8-cloudtasks) (2.x) | Module 9 [code](/mod9-py3dstasks) 10|Migrate Datastore/Firestore data to another project| _TBD_ | _N/A_ | _N/A_ | _TBD_ 11|Migrate to Cloud Functions| [link](https://twitter.com/googledevs/status/1520206298834481153?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrcloudfuncs_sms_202006&utm_content=-) | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-11-functions?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrcloudfuncs_sms_202006&utm_content=-) | Module 2 [code](/mod2b-cloudndb) (3.x) | Module 11 [code](/mod11-functions) (3.x) From c0b6e3094951455bdc64683b1609db29bac6362c Mon Sep 17 00:00:00 2001 From: wesley chun Date: Tue, 19 Jul 2022 01:22:21 -0700 Subject: [PATCH 35/56] mod19: update tmpl, add maker script --- README.md | 2 +- mod19-pubsub/maker.py | 44 +++++++++++++++++++++++++++++++ mod19-pubsub/templates/index.html | 10 ++++--- 3 files changed, 52 insertions(+), 4 deletions(-) create mode 100644 mod19-pubsub/maker.py diff --git a/README.md b/README.md index 1e82699..56e5cfc 100644 --- a/README.md +++ b/README.md @@ -304,7 +304,7 @@ If your original app users does *not* have a user interface, i.e., mobile backen ## Accessing legacy services in second generation -Many legacy App Engine first generation platform (Python 2, Java 8, PHP 5, and Go 1.11 & older) services are available ([as of Sep 2021](https://twitter.com/googledevs/status/1445916786755571712) for second generation runtimes (Python 3, Java 11, PHP 7, and Go 1.12 & newer) in a public preview. There are no videos or codelabs yet, however the Module 1 Flask migration using App Engine `ndb` [Python 2 sample ](/mod1-flask) is available in [Python 3](/mod1b-flask) if you have access. Similarly, Python 3 editions are also available for Modules 7 and 12 which add usage of App Engine `taskqueue` and `memcache`, respectively. Also see the [documentation on accessing bundled services from Python 3](https://cloud.google.com/appengine/docs/standard/python3/services/access). +Many legacy App Engine first generation platform (Python 2, Java 8, PHP 5, and Go 1.11 & older) services are available ([as of Sep 2021](https://twitter.com/googledevs/status/1445916786755571712) for second generation runtimes (Python 3, Java 11/17, PHP 7/8, and Go 1.12 & newer) in a public preview. There are no videos or codelabs yet, however the Module 1 Flask migration using App Engine `ndb` [Python 2 sample](/mod1-flask) is available in [Python 3](/mod1b-flask) if you have access. Similarly, Python 3 editions are also available for Modules 7 and 12 which add usage of App Engine `taskqueue` and `memcache`, respectively. Also see the [documentation on accessing bundled services from Python 3](https://cloud.google.com/appengine/docs/standard/python3/services/access). ## References diff --git a/mod19-pubsub/maker.py b/mod19-pubsub/maker.py new file mode 100644 index 0000000..856f66a --- /dev/null +++ b/mod19-pubsub/maker.py @@ -0,0 +1,44 @@ +# Copyright 2022 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from __future__ import print_function +import google.auth +from google.api_core import exceptions +from google.cloud import pubsub + +_, PROJECT_ID = google.auth.default() +TOPIC = 'pullq' +SBSCR = 'worker' +ppc_client = pubsub.PublisherClient() +psc_client = pubsub.SubscriberClient() +TOP_PATH = ppc_client.topic_path(PROJECT_ID, TOPIC) +SUB_PATH = psc_client.subscription_path(PROJECT_ID, SBSCR) + +def make_top(): + try: + top = ppc_client.create_topic(name=TOP_PATH) + print('Created topic %r (%s)' % (TOPIC, top.name)) + except exceptions.AlreadyExists: + print('Topic %r already exists at %r' % (TOPIC, TOP_PATH)) + +def make_sub(): + with psc_client: + try: + sub = psc_client.create_subscription(name=SUB_PATH, topic=TOP_PATH) + print('Subscription created %r (%s)' % (SBSCR, sub.name)) + except exceptions.AlreadyExists: + print('Subscription %r already exists at %r' % (SBSCR, SUB_PATH)) + +make_top() +make_sub() diff --git a/mod19-pubsub/templates/index.html b/mod19-pubsub/templates/index.html index b88f744..19bf331 100644 --- a/mod19-pubsub/templates/index.html +++ b/mod19-pubsub/templates/index.html @@ -9,9 +9,13 @@

VisitMe example

Top {{ limit }} visitors

-{% for count in counts %} - -{% endfor %} +{% if counts %} + {% for count in counts %} + + {% endfor %} +{% else %} + +{% endif %}
VisitorVisits
{{ count.visitor|e }}{{ count.counter }}
{{ count.visitor|e }}{{ count.counter }}
(none yet; visit the /log endpoint)

Last {{ limit }} visits

From f551864122976af54f619348599604273e9f653c Mon Sep 17 00:00:00 2001 From: wesley chun Date: Fri, 22 Jul 2022 00:57:56 -0700 Subject: [PATCH 36/56] add checks for mods18-19, use latest 3P built-in libs --- README.md | 2 +- mod13a-memorystore/app.yaml | 4 ++-- mod18-gaepull/main.py | 3 ++- mod19-pubsub/main.py | 18 ++++++++++-------- mod2a-cloudndb/app.yaml | 4 ++-- mod3a-datastore/app.yaml | 4 ++-- mod8-cloudtasks/app.yaml | 4 ++-- 7 files changed, 21 insertions(+), 18 deletions(-) diff --git a/README.md b/README.md index 56e5cfc..4cc569c 100644 --- a/README.md +++ b/README.md @@ -113,7 +113,7 @@ Module | Topic | Video | Codelab | START here | FINISH here 14|Migrate service between projects| _TBD_ | _TBD_ | _TBD_ | _TBD_ 15|Add App Engine `blobstore`| _TBD_ | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-15-blobstore?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrblobstore_sms_202029&utm_content=-) | Module 0 [code](/mod0-baseline) (2.x) | Module 15 [code](/mod15-blobstore) (2.x) 16|Migrate to Cloud Storage| _TBD_ | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-16-cloudstorage?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrcloudstorage_sms_202029&utm_content=-) | Module 15 [code](/mod15-blobstore) (2.x) | Module 16 [code](/mod16-cloudstorage) (2.x & 3.x) -17|Migrate to Python 3 bundled services| _TBD_ | _TBD_ | Module 1 [code](/mod1-flask) (2.x) | Module 1 [code](/mod1b-flask) (3.x) +17|Migrate to Python 3 bundled services| _TBD_ | [link](http://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-17-bundled?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrwormhole_sms_202002&utm_content=-) | Module 1 [code](/mod1-flask) (2.x) | Module 1 [code](/mod1b-flask) (3.x) 18|Add App Engine `taskqueue` pull tasks| _TBD_ | _TBD_ | Module 1 [code](/mod1-flask) (2.x) | Module 18 [code](/mod18-gaepull) (2.x) 19|Migrate to Cloud Pub/Sub| _TBD_ | _TBD_ | Module 18 [code](/mod18-gaepull) (2.x) | Module 19 [code](/mod9-pubsub) (3.x) diff --git a/mod13a-memorystore/app.yaml b/mod13a-memorystore/app.yaml index db67a62..9021a81 100644 --- a/mod13a-memorystore/app.yaml +++ b/mod13a-memorystore/app.yaml @@ -22,9 +22,9 @@ handlers: libraries: - name: grpcio - version: 1.0.0 + version: latest - name: setuptools - version: 36.6.0 + version: latest env_variables: REDIS_HOST: 'YOUR_REDIS_HOST' diff --git a/mod18-gaepull/main.py b/mod18-gaepull/main.py index df30f37..53de1e2 100644 --- a/mod18-gaepull/main.py +++ b/mod18-gaepull/main.py @@ -58,7 +58,8 @@ def log_visitors(): for task in tasks: visitor = task.payload tallies[visitor] = tallies.get(visitor, 0) + 1 - q.delete_tasks(tasks) + if tasks: + q.delete_tasks(tasks) # increment those counts in Datastore and return for visitor in tallies: diff --git a/mod19-pubsub/main.py b/mod19-pubsub/main.py index 7791ec6..3f7698e 100644 --- a/mod19-pubsub/main.py +++ b/mod19-pubsub/main.py @@ -70,17 +70,19 @@ def log_visitors(): acks.add(rcvd_msg.ack_id) visitor = rcvd_msg.message.data.decode('utf-8') tallies[visitor] = tallies.get(visitor, 0) + 1 - psc_client.acknowledge(subscription=SUB_PATH, ack_ids=acks) + if acks: + psc_client.acknowledge(subscription=SUB_PATH, ack_ids=acks) # increment those counts in Datastore and return - with ds_client.context(): - for visitor in tallies: - counter = VisitorCount.query(VisitorCount.visitor == visitor).get() - if not counter: - counter = VisitorCount(visitor=visitor, counter=0) + if tallies: + with ds_client.context(): + for visitor in tallies: + counter = VisitorCount.query(VisitorCount.visitor == visitor).get() + if not counter: + counter = VisitorCount(visitor=visitor, counter=0) + counter.put() + counter.counter += tallies[visitor] counter.put() - counter.counter += tallies[visitor] - counter.put() return 'DONE (with %d task[s] logging %d visitor[s])\r\n' % (len(msgs), len(tallies)) diff --git a/mod2a-cloudndb/app.yaml b/mod2a-cloudndb/app.yaml index 0877f79..ba543cd 100644 --- a/mod2a-cloudndb/app.yaml +++ b/mod2a-cloudndb/app.yaml @@ -22,6 +22,6 @@ handlers: libraries: - name: grpcio - version: 1.0.0 + version: latest - name: setuptools - version: 36.6.0 + version: latest diff --git a/mod3a-datastore/app.yaml b/mod3a-datastore/app.yaml index 0877f79..ba543cd 100644 --- a/mod3a-datastore/app.yaml +++ b/mod3a-datastore/app.yaml @@ -22,6 +22,6 @@ handlers: libraries: - name: grpcio - version: 1.0.0 + version: latest - name: setuptools - version: 36.6.0 + version: latest diff --git a/mod8-cloudtasks/app.yaml b/mod8-cloudtasks/app.yaml index 0877f79..ba543cd 100644 --- a/mod8-cloudtasks/app.yaml +++ b/mod8-cloudtasks/app.yaml @@ -22,6 +22,6 @@ handlers: libraries: - name: grpcio - version: 1.0.0 + version: latest - name: setuptools - version: 36.6.0 + version: latest From 691998213291236879c4da668edc20584e401d45 Mon Sep 17 00:00:00 2001 From: wesley chun Date: Fri, 22 Jul 2022 20:00:17 -0700 Subject: [PATCH 37/56] mod19 fixes to support Py2 --- mod19-pubsub/app.yaml | 15 ++++++++++++++- mod19-pubsub/app3.yaml | 15 +++++++++++++++ mod19-pubsub/appengine_config.py | 23 +++++++++++++++++++++++ mod19-pubsub/main.py | 23 ++++++++++++++--------- 4 files changed, 66 insertions(+), 10 deletions(-) create mode 100644 mod19-pubsub/app3.yaml create mode 100644 mod19-pubsub/appengine_config.py diff --git a/mod19-pubsub/app.yaml b/mod19-pubsub/app.yaml index 9aac415..81a4730 100644 --- a/mod19-pubsub/app.yaml +++ b/mod19-pubsub/app.yaml @@ -12,4 +12,17 @@ # See the License for the specific language governing permissions and # limitations under the License. -runtime: python310 +#runtime: python310 +runtime: python27 +threadsafe: yes +api_version: 1 + +handlers: +- url: /.* + script: main.app + +libraries: +- name: setuptools + version: latest +- name: grpcio + version: latest diff --git a/mod19-pubsub/app3.yaml b/mod19-pubsub/app3.yaml new file mode 100644 index 0000000..9aac415 --- /dev/null +++ b/mod19-pubsub/app3.yaml @@ -0,0 +1,15 @@ +# Copyright 2022 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +runtime: python310 diff --git a/mod19-pubsub/appengine_config.py b/mod19-pubsub/appengine_config.py new file mode 100644 index 0000000..2a41fb4 --- /dev/null +++ b/mod19-pubsub/appengine_config.py @@ -0,0 +1,23 @@ +# Copyright 2021 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import pkg_resources +from google.appengine.ext import vendor + +# Set PATH to your libraries folder. +PATH = 'lib' +# Add libraries installed in the PATH folder. +vendor.add(PATH) +# Add libraries to pkg_resources working set to find the distribution. +pkg_resources.working_set.add_entry(PATH) diff --git a/mod19-pubsub/main.py b/mod19-pubsub/main.py index 3f7698e..ab15231 100644 --- a/mod19-pubsub/main.py +++ b/mod19-pubsub/main.py @@ -63,15 +63,20 @@ def log_visitors(): # tally recent visitor counts from queue then delete those tasks tallies = {} acks = set() - with psc_client: - rsp = psc_client.pull(subscription=SUB_PATH, max_messages=TASKS) - msgs = rsp.received_messages - for rcvd_msg in msgs: - acks.add(rcvd_msg.ack_id) - visitor = rcvd_msg.message.data.decode('utf-8') - tallies[visitor] = tallies.get(visitor, 0) + 1 - if acks: - psc_client.acknowledge(subscription=SUB_PATH, ack_ids=acks) + #with psc_client: + rsp = psc_client.pull(subscription=SUB_PATH, max_messages=TASKS) + msgs = rsp.received_messages + for rcvd_msg in msgs: + acks.add(rcvd_msg.ack_id) + visitor = rcvd_msg.message.data.decode('utf-8') + tallies[visitor] = tallies.get(visitor, 0) + 1 + if acks: + psc_client.acknowledge(subscription=SUB_PATH, ack_ids=acks) + if hasattr(psc_client, 'close'): + try: + psc_client.close() + except AttributeError: + pass # increment those counts in Datastore and return if tallies: From 28e90b1612f426866748b7e22c959f3f9a7de8f0 Mon Sep 17 00:00:00 2001 From: wesley chun Date: Fri, 22 Jul 2022 20:03:59 -0700 Subject: [PATCH 38/56] README fix --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 4cc569c..c04d97e 100644 --- a/README.md +++ b/README.md @@ -115,7 +115,7 @@ Module | Topic | Video | Codelab | START here | FINISH here 16|Migrate to Cloud Storage| _TBD_ | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-16-cloudstorage?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrcloudstorage_sms_202029&utm_content=-) | Module 15 [code](/mod15-blobstore) (2.x) | Module 16 [code](/mod16-cloudstorage) (2.x & 3.x) 17|Migrate to Python 3 bundled services| _TBD_ | [link](http://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-17-bundled?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrwormhole_sms_202002&utm_content=-) | Module 1 [code](/mod1-flask) (2.x) | Module 1 [code](/mod1b-flask) (3.x) 18|Add App Engine `taskqueue` pull tasks| _TBD_ | _TBD_ | Module 1 [code](/mod1-flask) (2.x) | Module 18 [code](/mod18-gaepull) (2.x) -19|Migrate to Cloud Pub/Sub| _TBD_ | _TBD_ | Module 18 [code](/mod18-gaepull) (2.x) | Module 19 [code](/mod9-pubsub) (3.x) +19|Migrate to Cloud Pub/Sub| _TBD_ | _TBD_ | Module 18 [code](/mod18-gaepull) (2.x) | Module 19 [code](/mod19-pubsub) (2.x & 3.x) ### Table of contents From 5ea1cda71a8fc11779984d38e692ce3797a3f175 Mon Sep 17 00:00:00 2001 From: wesley chun Date: Mon, 15 Aug 2022 15:44:19 -0700 Subject: [PATCH 39/56] README updates & other minor config fixes --- README.md | 14 +++++++++++--- mod19-pubsub/.gcloudignore | 1 + mod19-pubsub/README.md | 4 +--- mod9-py3dstasks/app.yaml | 2 +- 4 files changed, 14 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index c04d97e..c21d15a 100644 --- a/README.md +++ b/README.md @@ -111,10 +111,10 @@ Module | Topic | Video | Codelab | START here | FINISH here 12|Add App Engine `memcache`| [link](https://twitter.com/googledevs/status/1527303061953126402?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrmemcache_sms_202006&utm_content=-) | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-12-memcache?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrmemcache_sms_202006&utm_content=-) | Module 1 [code](/mod1-flask) (2.x) | Module 12 [code](/mod12-memcache) (2.x) & [code](/mod12b-memcache) (3.x) 13|Migrate to Cloud Memorystore| [link](https://twitter.com/googledevs/status/1537132939426799616?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrmemorystore_sms_202029&utm_content=-) | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-13-memorystore?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrmemorystore_sms_202029&utm_content=-) | Module 12 [code](/mod12-memcache) (2.x) & [code](/mod12b-memcache) (3.x) | Module 13 [code](/mod13a-memorystore) (2.x) & [code](/mod13b-memorystore) (3.x) 14|Migrate service between projects| _TBD_ | _TBD_ | _TBD_ | _TBD_ -15|Add App Engine `blobstore`| _TBD_ | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-15-blobstore?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrblobstore_sms_202029&utm_content=-) | Module 0 [code](/mod0-baseline) (2.x) | Module 15 [code](/mod15-blobstore) (2.x) -16|Migrate to Cloud Storage| _TBD_ | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-16-cloudstorage?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrcloudstorage_sms_202029&utm_content=-) | Module 15 [code](/mod15-blobstore) (2.x) | Module 16 [code](/mod16-cloudstorage) (2.x & 3.x) +15|Add App Engine `blobstore`| [link](https://twitter.com/googledevs/status/1552384740052934657?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrblobstore_sms_202029&utm_content=-) | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-15-blobstore?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrblobstore_sms_202029&utm_content=-) | Module 0 [code](/mod0-baseline) (2.x) | Module 15 [code](/mod15-blobstore) (2.x) +16|Migrate to Cloud Storage| [link](https://twitter.com/googledevs/status/1559285905961123845?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrcloudstorage_sms_202029&utm_content=-) | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-16-cloudstorage?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrcloudstorage_sms_202029&utm_content=-) | Module 15 [code](/mod15-blobstore) (2.x) | Module 16 [code](/mod16-cloudstorage) (2.x & 3.x) 17|Migrate to Python 3 bundled services| _TBD_ | [link](http://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-17-bundled?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrwormhole_sms_202002&utm_content=-) | Module 1 [code](/mod1-flask) (2.x) | Module 1 [code](/mod1b-flask) (3.x) -18|Add App Engine `taskqueue` pull tasks| _TBD_ | _TBD_ | Module 1 [code](/mod1-flask) (2.x) | Module 18 [code](/mod18-gaepull) (2.x) +18|Add App Engine `taskqueue` pull tasks| _TBD_ | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-18-gaepull?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrgaepull_sms_202013&utm_content=-) | Module 1 [code](/mod1-flask) (2.x) | Module 18 [code](/mod18-gaepull) (2.x) 19|Migrate to Cloud Pub/Sub| _TBD_ | _TBD_ | Module 18 [code](/mod18-gaepull) (2.x) | Module 19 [code](/mod19-pubsub) (2.x & 3.x) @@ -307,6 +307,14 @@ If your original app users does *not* have a user interface, i.e., mobile backen Many legacy App Engine first generation platform (Python 2, Java 8, PHP 5, and Go 1.11 & older) services are available ([as of Sep 2021](https://twitter.com/googledevs/status/1445916786755571712) for second generation runtimes (Python 3, Java 11/17, PHP 7/8, and Go 1.12 & newer) in a public preview. There are no videos or codelabs yet, however the Module 1 Flask migration using App Engine `ndb` [Python 2 sample](/mod1-flask) is available in [Python 3](/mod1b-flask) if you have access. Similarly, Python 3 editions are also available for Modules 7 and 12 which add usage of App Engine `taskqueue` and `memcache`, respectively. Also see the [documentation on accessing bundled services from Python 3](https://cloud.google.com/appengine/docs/standard/python3/services/access). +## Community + +Python App Engine developers hang out in various online communities, including these: +- [Slack](https://googlecloud-community.slack.com) (`#app-engine`, `#python`, and other channels); visit [this link](https://join.slack.com/t/googlecloud-community/shared_invite/zt-ywj8ieuc-BrAaHC~qe5IgelXS9vzNRA) to join +- [Reddit](http://reddit.com) in the [Google Cloud](https://reddit.com/googlecloud) or [App Engine](https://reddit.com/appengine) subs (subReddits). +- [App Engine mailing list](http://groups.google.com/group/google-appengine) + + ## References - App Engine Migration diff --git a/mod19-pubsub/.gcloudignore b/mod19-pubsub/.gcloudignore index b4dbc7e..d9ce03e 100644 --- a/mod19-pubsub/.gcloudignore +++ b/mod19-pubsub/.gcloudignore @@ -21,6 +21,7 @@ LICENSE # Tests/results (not in .gitignore) noxfile.py test_translate.py +maker.py pylintrc pylintrc.test diff --git a/mod19-pubsub/README.md b/mod19-pubsub/README.md index 6640585..737b286 100644 --- a/mod19-pubsub/README.md +++ b/mod19-pubsub/README.md @@ -1,5 +1,3 @@ # Module 19 - Migrate from App Engine `taskqueue` (pull tasks) to Cloud Pub/Sub (and Python 3) -This repo folder is the corresponding Python 2 code to the [Module 19 codelab](http://g.co/codelabs/pae-migrate-pubsub). The tutorial STARTs with the Python 2 code in the [Module 18 repo folder](/mod18-gaepull) and leads developers through its migration from pull tasks via App Engine `taskqueue` to Cloud Pub/Sub and Python 3, culminating in the code in this folder. The migration from App Engine `ndb` to Cloud NDB, covered in Module 2, also takes place. - -**NOTE:** While this solution Python 2 compatible, a library conflict on App Engine servers does not allow it to run properly under Python 2 on App Engine.) +This repo folder is the corresponding Python 2 and 3 code to the [Module 19 codelab](http://g.co/codelabs/pae-migrate-pubsub). The tutorial STARTs with the Python 2 code in the [Module 18 repo folder](/mod18-gaepull) and leads developers through its migration from pull tasks via App Engine `taskqueue` to Cloud Pub/Sub, culminating in the code in this folder. The code is both Python 2 and 3 compatible, and either uncomment the Python 3 runtime in `app.yaml` and delete all other lines, or just use `app3.yaml`, delete `appengine_config.py` and any `lib` folder. The migration from App Engine `ndb` to Cloud NDB, covered in Module 2, also takes place. diff --git a/mod9-py3dstasks/app.yaml b/mod9-py3dstasks/app.yaml index a926609..16a11a3 100644 --- a/mod9-py3dstasks/app.yaml +++ b/mod9-py3dstasks/app.yaml @@ -12,4 +12,4 @@ # See the License for the specific language governing permissions and # limitations under the License. -runtime: python39 +runtime: python310 From 92536230aa610f7e16e2c7d13c78e7f441596ed0 Mon Sep 17 00:00:00 2001 From: wesley chun Date: Wed, 17 Aug 2022 16:53:31 -0700 Subject: [PATCH 40/56] add Mod19 codelab & other minor fixes --- README.md | 6 +++--- mod1-flask/main.py | 2 ++ mod18-gaepull/main.py | 3 ++- mod19-pubsub/main.py | 13 ++++++------- mod1b-flask/main.py | 2 ++ 5 files changed, 15 insertions(+), 11 deletions(-) diff --git a/README.md b/README.md index c21d15a..ef0ef0b 100644 --- a/README.md +++ b/README.md @@ -115,7 +115,7 @@ Module | Topic | Video | Codelab | START here | FINISH here 16|Migrate to Cloud Storage| [link](https://twitter.com/googledevs/status/1559285905961123845?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrcloudstorage_sms_202029&utm_content=-) | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-16-cloudstorage?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrcloudstorage_sms_202029&utm_content=-) | Module 15 [code](/mod15-blobstore) (2.x) | Module 16 [code](/mod16-cloudstorage) (2.x & 3.x) 17|Migrate to Python 3 bundled services| _TBD_ | [link](http://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-17-bundled?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrwormhole_sms_202002&utm_content=-) | Module 1 [code](/mod1-flask) (2.x) | Module 1 [code](/mod1b-flask) (3.x) 18|Add App Engine `taskqueue` pull tasks| _TBD_ | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-18-gaepull?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrgaepull_sms_202013&utm_content=-) | Module 1 [code](/mod1-flask) (2.x) | Module 18 [code](/mod18-gaepull) (2.x) -19|Migrate to Cloud Pub/Sub| _TBD_ | _TBD_ | Module 18 [code](/mod18-gaepull) (2.x) | Module 19 [code](/mod19-pubsub) (2.x & 3.x) +19|Migrate to Cloud Pub/Sub| _TBD_ | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-19-pubsub?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrpubsub_sms_202016&utm_content=-) | Module 18 [code](/mod18-gaepull) (2.x) | Module 19 [code](/mod19-pubsub) (2.x & 3.x) ### Table of contents @@ -162,7 +162,7 @@ If there is a logical codelab to do immediately after completing one, they will - Module 12 - add App Engine Memcache (and migrate to Cloud Memorystore in Module 13) - Module 15 - add App Engine Blobstore (and migrate to Cloud Storage in Module 16) -- **Module 9 codelab** (TBD): **Migrate a Python 2 Cloud NDB & Cloud Tasks (v1) app to a Python 3 Cloud Datastore & Cloud Tasks (v2) app** +- [Module 9 codelab](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-9-py3dstasks?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrpy3fstasks_sms_201126&utm_content=-): **Migrate a Python 2 Cloud NDB & Cloud Tasks (v1) app to a Python 3 Cloud Datastore & Cloud Tasks (v2) app** - **Optional** migrations - Migrating to Python 3 is not required but recommended as Python 2 has been sunset - Migrating to Cloud Datastore is optional as Cloud NDB works on 3.x @@ -175,7 +175,7 @@ If there is a logical codelab to do immediately after completing one, they will - Module 12 - add App Engine Memcache (and migrate to Cloud Memorystore in Module 13) - Module 15 - add App Engine Blobstore (and migrate to Cloud Storage in Module 16) -- **Module 18 codelab** (TBD): **Add App Engine (pull) Task Queues to existing sample app** +- [Module 18 codelab](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-18-gaepull?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrgaepull_sms_202013&utm_content=-): **Add App Engine (pull) Task Queues to existing sample app** - **Not a migration**: add GAE Task Queues to prepare for migration to Cloud Tasks - Python 2 - START: [Module 1 code - Framework](/mod1-flask) diff --git a/mod1-flask/main.py b/mod1-flask/main.py index 14f5f89..f36f3e5 100644 --- a/mod1-flask/main.py +++ b/mod1-flask/main.py @@ -17,6 +17,7 @@ app = Flask(__name__) + class Visit(ndb.Model): 'Visit entity registers visitor IP address & timestamp' visitor = ndb.StringProperty() @@ -30,6 +31,7 @@ def fetch_visits(limit): 'get most recent visits' return Visit.query().order(-Visit.timestamp).fetch(limit) + @app.route('/') def root(): 'main application (GET) handler' diff --git a/mod18-gaepull/main.py b/mod18-gaepull/main.py index 53de1e2..706605a 100644 --- a/mod18-gaepull/main.py +++ b/mod18-gaepull/main.py @@ -69,7 +69,8 @@ def log_visitors(): counter.put() counter.counter += tallies[visitor] counter.put() - return 'DONE (with %d task[s] logging %d visitor[s])\r\n' % (len(tasks), len(tallies)) + return 'DONE (with %d task[s] logging %d visitor[s])\r\n' % ( + len(tasks), len(tallies)) @app.route('/') diff --git a/mod19-pubsub/main.py b/mod19-pubsub/main.py index ab15231..1724fc9 100644 --- a/mod19-pubsub/main.py +++ b/mod19-pubsub/main.py @@ -63,7 +63,6 @@ def log_visitors(): # tally recent visitor counts from queue then delete those tasks tallies = {} acks = set() - #with psc_client: rsp = psc_client.pull(subscription=SUB_PATH, max_messages=TASKS) msgs = rsp.received_messages for rcvd_msg in msgs: @@ -72,11 +71,10 @@ def log_visitors(): tallies[visitor] = tallies.get(visitor, 0) + 1 if acks: psc_client.acknowledge(subscription=SUB_PATH, ack_ids=acks) - if hasattr(psc_client, 'close'): - try: - psc_client.close() - except AttributeError: - pass + try: + psc_client.close() + except AttributeError: # special handler for grpcio<1.12.0 + pass # increment those counts in Datastore and return if tallies: @@ -88,7 +86,8 @@ def log_visitors(): counter.put() counter.counter += tallies[visitor] counter.put() - return 'DONE (with %d task[s] logging %d visitor[s])\r\n' % (len(msgs), len(tallies)) + return 'DONE (with %d task[s] logging %d visitor[s])\r\n' % ( + len(msgs), len(tallies)) @app.route('/') diff --git a/mod1b-flask/main.py b/mod1b-flask/main.py index 8d34768..0508b53 100644 --- a/mod1b-flask/main.py +++ b/mod1b-flask/main.py @@ -20,6 +20,7 @@ app = Flask(__name__) app.wsgi_app = wrap_wsgi_app(app.wsgi_app) + class Visit(ndb.Model): 'Visit entity registers visitor IP address & timestamp' visitor = ndb.StringProperty() @@ -33,6 +34,7 @@ def fetch_visits(limit): 'get most recent visits' return Visit.query().order(-Visit.timestamp).fetch(limit) + @app.route('/') def root(): 'main application (GET) handler' From 22efc23dc2a025518de2afdd72a2f9da144fe014 Mon Sep 17 00:00:00 2001 From: wesley chun Date: Mon, 22 Aug 2022 19:13:08 -0700 Subject: [PATCH 41/56] add Mod9 codelab; Mod18 & 19 app fixes --- README.md | 2 +- mod18-gaepull/main.py | 11 +++++------ mod19-pubsub/appengine_config.py | 2 +- mod19-pubsub/main.py | 2 +- mod19-pubsub/maker.py | 15 +++++++++------ 5 files changed, 17 insertions(+), 15 deletions(-) diff --git a/README.md b/README.md index ef0ef0b..b3066b3 100644 --- a/README.md +++ b/README.md @@ -105,7 +105,7 @@ Module | Topic | Video | Codelab | START here | FINISH here 6|Migrate to Cloud Firestore| _N/A_ | _N/A_ | Module 3 [code](/mod3b-datastore) (3.x) | _no work required; [Datastore upgrade automatic](https://cloud.google.com/datastore/docs/upgrade-to-firestore)_ 7|Add App Engine `taskqueue` push tasks| [link](https://twitter.com/googledevs/status/1443410302113099778?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrgaetasks_sms_201028&utm_content=-) | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-7-gaetasks?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrgaetasks_sms_201028&utm_content=-) | Module 1 [code](/mod1-flask) (2.x) | Module 7 [code](/mod7-gaetasks) (2.x) & [code](/mod7b-gaetasks) (3.x) 8|Migrate to Cloud Tasks| [link](https://twitter.com/googledevs/status/1450960021018267656?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrcloudtasks_sms_201112&utm_content=-) | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-8-cloudtasks?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrcloudtasks_sms_201112&utm_content=-) | Module 7 [code](/mod7-gaetasks) (2.x) | Module 8 [code](/mod8-cloudtasks) (2.x) -9|Migrate to Python 3, Cloud Datastore & Cloud Tasks v2| _TBD_ | _TBD_ | Module 8 [code](/mod8-cloudtasks) (2.x) | Module 9 [code](/mod9-py3dstasks) +9|Migrate to Python 3, Cloud Datastore & Cloud Tasks v2| _TBD_ | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-9-py3dstasks?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrpy3fstasks_sms_201126&utm_content=-) | Module 8 [code](/mod8-cloudtasks) (2.x) | Module 9 [code](/mod9-py3dstasks) 10|Migrate Datastore/Firestore data to another project| _TBD_ | _N/A_ | _N/A_ | _TBD_ 11|Migrate to Cloud Functions| [link](https://twitter.com/googledevs/status/1520206298834481153?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrcloudfuncs_sms_202006&utm_content=-) | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-11-functions?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrcloudfuncs_sms_202006&utm_content=-) | Module 2 [code](/mod2b-cloudndb) (3.x) | Module 11 [code](/mod11-functions) (3.x) 12|Add App Engine `memcache`| [link](https://twitter.com/googledevs/status/1527303061953126402?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrmemcache_sms_202006&utm_content=-) | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-12-memcache?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrmemcache_sms_202006&utm_content=-) | Module 1 [code](/mod1-flask) (2.x) | Module 12 [code](/mod12-memcache) (2.x) & [code](/mod12b-memcache) (3.x) diff --git a/mod18-gaepull/main.py b/mod18-gaepull/main.py index 706605a..a10eba2 100644 --- a/mod18-gaepull/main.py +++ b/mod18-gaepull/main.py @@ -19,7 +19,8 @@ HOUR = 3600 LIMIT = 10 TASKS = 1000 -QUEUE = 'pullq' +QNAME = 'pullq' +QUEUE = taskqueue.Queue(QNAME) app = Flask(__name__) @@ -31,8 +32,7 @@ class Visit(ndb.Model): def store_visit(remote_addr, user_agent): 'create new Visit in Datastore and queue request to bump visitor count' Visit(visitor='{}: {}'.format(remote_addr, user_agent)).put() - q = taskqueue.Queue(QUEUE) - q.add(taskqueue.Task(payload=remote_addr, method='PULL')) + QUEUE.add(taskqueue.Task(payload=remote_addr, method='PULL')) def fetch_visits(limit): 'get most recent visits' @@ -53,13 +53,12 @@ def log_visitors(): 'worker processes recent visitor counts and updates them in Datastore' # tally recent visitor counts from queue then delete those tasks tallies = {} - q = taskqueue.Queue(QUEUE) - tasks = q.lease_tasks(HOUR, TASKS) + tasks = QUEUE.lease_tasks(HOUR, TASKS) for task in tasks: visitor = task.payload tallies[visitor] = tallies.get(visitor, 0) + 1 if tasks: - q.delete_tasks(tasks) + QUEUE.delete_tasks(tasks) # increment those counts in Datastore and return for visitor in tallies: diff --git a/mod19-pubsub/appengine_config.py b/mod19-pubsub/appengine_config.py index 2a41fb4..3747d18 100644 --- a/mod19-pubsub/appengine_config.py +++ b/mod19-pubsub/appengine_config.py @@ -1,4 +1,4 @@ -# Copyright 2021 Google LLC +# Copyright 2022 Google LLC # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/mod19-pubsub/main.py b/mod19-pubsub/main.py index 1724fc9..276b433 100644 --- a/mod19-pubsub/main.py +++ b/mod19-pubsub/main.py @@ -73,7 +73,7 @@ def log_visitors(): psc_client.acknowledge(subscription=SUB_PATH, ack_ids=acks) try: psc_client.close() - except AttributeError: # special handler for grpcio<1.12.0 + except AttributeError: # special Py2 handler for grpcio<1.12.0 pass # increment those counts in Datastore and return diff --git a/mod19-pubsub/maker.py b/mod19-pubsub/maker.py index 856f66a..3635720 100644 --- a/mod19-pubsub/maker.py +++ b/mod19-pubsub/maker.py @@ -33,12 +33,15 @@ def make_top(): print('Topic %r already exists at %r' % (TOPIC, TOP_PATH)) def make_sub(): - with psc_client: - try: - sub = psc_client.create_subscription(name=SUB_PATH, topic=TOP_PATH) - print('Subscription created %r (%s)' % (SBSCR, sub.name)) - except exceptions.AlreadyExists: - print('Subscription %r already exists at %r' % (SBSCR, SUB_PATH)) + try: + sub = psc_client.create_subscription(name=SUB_PATH, topic=TOP_PATH) + print('Subscription created %r (%s)' % (SBSCR, sub.name)) + except exceptions.AlreadyExists: + print('Subscription %r already exists at %r' % (SBSCR, SUB_PATH)) + try: + psc_client.close() + except AttributeError: # special Py2 handler for grpcio<1.12.0 + pass make_top() make_sub() From 7bd35d95c442b139c2c2d1c736f8887bcfe9ab42 Mon Sep 17 00:00:00 2001 From: wesley chun Date: Thu, 15 Sep 2022 16:38:07 -0700 Subject: [PATCH 42/56] update `.gcloudignore` files & Blobstore/GCS `README`s --- mod0-baseline/.gcloudignore | 1 - mod1-flask/.gcloudignore | 1 - mod11-functions/.gcloudignore | 1 - mod12-memcache/.gcloudignore | 1 - mod12b-memcache/.gcloudignore | 1 - mod13a-memorystore/.gcloudignore | 1 - mod13b-memorystore/.gcloudignore | 1 - mod15-blobstore/.gcloudignore | 4 +++- mod15-blobstore/README.md | 2 +- mod16-cloudstorage/.gcloudignore | 4 +++- mod16-cloudstorage/README.md | 2 +- mod18-gaepull/.gcloudignore | 1 - mod19-pubsub/.gcloudignore | 5 +++-- mod1b-flask/.gcloudignore | 1 - mod2a-cloudndb/.gcloudignore | 1 - mod2b-cloudndb/.gcloudignore | 1 - mod3a-datastore/.gcloudignore | 1 - mod3b-datastore/.gcloudignore | 1 - mod4a-rundocker/.gcloudignore | 1 - mod4b-rundocker/.gcloudignore | 1 - mod5-runbldpks/.gcloudignore | 1 - mod6-firestore/.gcloudignore | 1 - mod7-gaetasks/.gcloudignore | 1 - mod7b-gaetasks/.gcloudignore | 1 - mod8-cloudtasks/.gcloudignore | 1 - mod9-py3dstasks/.gcloudignore | 1 - 26 files changed, 11 insertions(+), 27 deletions(-) diff --git a/mod0-baseline/.gcloudignore b/mod0-baseline/.gcloudignore index b4dbc7e..af73b5d 100644 --- a/mod0-baseline/.gcloudignore +++ b/mod0-baseline/.gcloudignore @@ -20,7 +20,6 @@ LICENSE # Tests/results (not in .gitignore) noxfile.py -test_translate.py pylintrc pylintrc.test diff --git a/mod1-flask/.gcloudignore b/mod1-flask/.gcloudignore index b4dbc7e..af73b5d 100644 --- a/mod1-flask/.gcloudignore +++ b/mod1-flask/.gcloudignore @@ -20,7 +20,6 @@ LICENSE # Tests/results (not in .gitignore) noxfile.py -test_translate.py pylintrc pylintrc.test diff --git a/mod11-functions/.gcloudignore b/mod11-functions/.gcloudignore index b4dbc7e..af73b5d 100644 --- a/mod11-functions/.gcloudignore +++ b/mod11-functions/.gcloudignore @@ -20,7 +20,6 @@ LICENSE # Tests/results (not in .gitignore) noxfile.py -test_translate.py pylintrc pylintrc.test diff --git a/mod12-memcache/.gcloudignore b/mod12-memcache/.gcloudignore index b4dbc7e..af73b5d 100644 --- a/mod12-memcache/.gcloudignore +++ b/mod12-memcache/.gcloudignore @@ -20,7 +20,6 @@ LICENSE # Tests/results (not in .gitignore) noxfile.py -test_translate.py pylintrc pylintrc.test diff --git a/mod12b-memcache/.gcloudignore b/mod12b-memcache/.gcloudignore index b4dbc7e..af73b5d 100644 --- a/mod12b-memcache/.gcloudignore +++ b/mod12b-memcache/.gcloudignore @@ -20,7 +20,6 @@ LICENSE # Tests/results (not in .gitignore) noxfile.py -test_translate.py pylintrc pylintrc.test diff --git a/mod13a-memorystore/.gcloudignore b/mod13a-memorystore/.gcloudignore index b4dbc7e..af73b5d 100644 --- a/mod13a-memorystore/.gcloudignore +++ b/mod13a-memorystore/.gcloudignore @@ -20,7 +20,6 @@ LICENSE # Tests/results (not in .gitignore) noxfile.py -test_translate.py pylintrc pylintrc.test diff --git a/mod13b-memorystore/.gcloudignore b/mod13b-memorystore/.gcloudignore index b4dbc7e..af73b5d 100644 --- a/mod13b-memorystore/.gcloudignore +++ b/mod13b-memorystore/.gcloudignore @@ -20,7 +20,6 @@ LICENSE # Tests/results (not in .gitignore) noxfile.py -test_translate.py pylintrc pylintrc.test diff --git a/mod15-blobstore/.gcloudignore b/mod15-blobstore/.gcloudignore index b4dbc7e..114f630 100644 --- a/mod15-blobstore/.gcloudignore +++ b/mod15-blobstore/.gcloudignore @@ -8,6 +8,9 @@ # .gcloudignore +# Special files in this dir +main-gcs.py + # Source code control files .git/ .gitignore @@ -20,7 +23,6 @@ LICENSE # Tests/results (not in .gitignore) noxfile.py -test_translate.py pylintrc pylintrc.test diff --git a/mod15-blobstore/README.md b/mod15-blobstore/README.md index 55857b1..1228943 100644 --- a/mod15-blobstore/README.md +++ b/mod15-blobstore/README.md @@ -1,6 +1,6 @@ # Module 15 - Add usage of App Engine `blobstore` to `webapp2 ndb` sample app -This repo folder is the corresponding code to the (forthcoming) Module 15 codelab. The tutorial STARTs with the Python 2 code in the [Module 0 repo folder](/mod0-baseline) and leads developers through adding use of App Engine `blobstore`. Unlike other sample apps, this does not use the default Django templating system, but instead, uses Jinja2, which is supported in `webapp2_extras`. +This repo folder is the corresponding code to the [Module 15 codelab](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-15-blobstore?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrblobstore_sms_202029&utm_content=-). The tutorial STARTs with the Python 2 code in the [Module 0 repo folder](/mod0-baseline) and leads developers through adding use of App Engine `blobstore`, resulting in the code in _this_ folder. Unlike other sample apps, this does not use the default Django templating system, but instead, uses Jinja2, which is supported in `webapp2_extras`. Blobstore evolved into [Google Cloud Storage](https://cloud.google.com/storage), and all blobs/files created using the Blobstore API go into the default Cloud Storage bucket for your project. It's named the same as the `appspot` domain name given to your app. For example, if your project is named `my-project`, your default bucket would be `my-project.appspot.com`. The default GCS bucket name is programmatically accessible via `google.appengine.api.app_identity.get_default_gcs_bucket_name()`. diff --git a/mod16-cloudstorage/.gcloudignore b/mod16-cloudstorage/.gcloudignore index b4dbc7e..1133073 100644 --- a/mod16-cloudstorage/.gcloudignore +++ b/mod16-cloudstorage/.gcloudignore @@ -8,6 +8,9 @@ # .gcloudignore +# Special files in this dir +main-migrate.py + # Source code control files .git/ .gitignore @@ -20,7 +23,6 @@ LICENSE # Tests/results (not in .gitignore) noxfile.py -test_translate.py pylintrc pylintrc.test diff --git a/mod16-cloudstorage/README.md b/mod16-cloudstorage/README.md index 074e67c..a051615 100644 --- a/mod16-cloudstorage/README.md +++ b/mod16-cloudstorage/README.md @@ -2,7 +2,7 @@ ## Migrations -This repo folder is the corresponding code to the [Module 16 codelab](http://g.co/codelabs/pae-migrate-blobstore). The tutorial STARTs with the Python 2 code in the [Module 15 repo folder](/mod15-blobstore) and leads developers through a set of migrations, culminating in the code in this folder. In addition to migrating to Cloud Storage, a few others are done to get from Modules 15 to 16... here is the complete list: +This repo folder is the corresponding code to the [Module 16 codelab](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-16-cloudstorage?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrcloudstorage_sms_202029&utm_content=-). The tutorial STARTs with the Python 2 code in the [Module 15 repo folder](/mod15-blobstore) and leads developers through a set of migrations, culminating in the code in _this_ folder. In addition to migrating to Cloud Storage, a few others are done to get from Modules 15 to 16... here is the complete list: 1. Migrate from App Engine `webapp2` to Flask 1. Migrate from App Engine `ndb` to Cloud NDB diff --git a/mod18-gaepull/.gcloudignore b/mod18-gaepull/.gcloudignore index b4dbc7e..af73b5d 100644 --- a/mod18-gaepull/.gcloudignore +++ b/mod18-gaepull/.gcloudignore @@ -20,7 +20,6 @@ LICENSE # Tests/results (not in .gitignore) noxfile.py -test_translate.py pylintrc pylintrc.test diff --git a/mod19-pubsub/.gcloudignore b/mod19-pubsub/.gcloudignore index d9ce03e..82df3a9 100644 --- a/mod19-pubsub/.gcloudignore +++ b/mod19-pubsub/.gcloudignore @@ -8,6 +8,9 @@ # .gcloudignore +# Special files in this dir +maker.py + # Source code control files .git/ .gitignore @@ -20,8 +23,6 @@ LICENSE # Tests/results (not in .gitignore) noxfile.py -test_translate.py -maker.py pylintrc pylintrc.test diff --git a/mod1b-flask/.gcloudignore b/mod1b-flask/.gcloudignore index b4dbc7e..af73b5d 100644 --- a/mod1b-flask/.gcloudignore +++ b/mod1b-flask/.gcloudignore @@ -20,7 +20,6 @@ LICENSE # Tests/results (not in .gitignore) noxfile.py -test_translate.py pylintrc pylintrc.test diff --git a/mod2a-cloudndb/.gcloudignore b/mod2a-cloudndb/.gcloudignore index b4dbc7e..af73b5d 100644 --- a/mod2a-cloudndb/.gcloudignore +++ b/mod2a-cloudndb/.gcloudignore @@ -20,7 +20,6 @@ LICENSE # Tests/results (not in .gitignore) noxfile.py -test_translate.py pylintrc pylintrc.test diff --git a/mod2b-cloudndb/.gcloudignore b/mod2b-cloudndb/.gcloudignore index b4dbc7e..af73b5d 100644 --- a/mod2b-cloudndb/.gcloudignore +++ b/mod2b-cloudndb/.gcloudignore @@ -20,7 +20,6 @@ LICENSE # Tests/results (not in .gitignore) noxfile.py -test_translate.py pylintrc pylintrc.test diff --git a/mod3a-datastore/.gcloudignore b/mod3a-datastore/.gcloudignore index b4dbc7e..af73b5d 100644 --- a/mod3a-datastore/.gcloudignore +++ b/mod3a-datastore/.gcloudignore @@ -20,7 +20,6 @@ LICENSE # Tests/results (not in .gitignore) noxfile.py -test_translate.py pylintrc pylintrc.test diff --git a/mod3b-datastore/.gcloudignore b/mod3b-datastore/.gcloudignore index b4dbc7e..af73b5d 100644 --- a/mod3b-datastore/.gcloudignore +++ b/mod3b-datastore/.gcloudignore @@ -20,7 +20,6 @@ LICENSE # Tests/results (not in .gitignore) noxfile.py -test_translate.py pylintrc pylintrc.test diff --git a/mod4a-rundocker/.gcloudignore b/mod4a-rundocker/.gcloudignore index b4dbc7e..af73b5d 100644 --- a/mod4a-rundocker/.gcloudignore +++ b/mod4a-rundocker/.gcloudignore @@ -20,7 +20,6 @@ LICENSE # Tests/results (not in .gitignore) noxfile.py -test_translate.py pylintrc pylintrc.test diff --git a/mod4b-rundocker/.gcloudignore b/mod4b-rundocker/.gcloudignore index b4dbc7e..af73b5d 100644 --- a/mod4b-rundocker/.gcloudignore +++ b/mod4b-rundocker/.gcloudignore @@ -20,7 +20,6 @@ LICENSE # Tests/results (not in .gitignore) noxfile.py -test_translate.py pylintrc pylintrc.test diff --git a/mod5-runbldpks/.gcloudignore b/mod5-runbldpks/.gcloudignore index b4dbc7e..af73b5d 100644 --- a/mod5-runbldpks/.gcloudignore +++ b/mod5-runbldpks/.gcloudignore @@ -20,7 +20,6 @@ LICENSE # Tests/results (not in .gitignore) noxfile.py -test_translate.py pylintrc pylintrc.test diff --git a/mod6-firestore/.gcloudignore b/mod6-firestore/.gcloudignore index b4dbc7e..af73b5d 100644 --- a/mod6-firestore/.gcloudignore +++ b/mod6-firestore/.gcloudignore @@ -20,7 +20,6 @@ LICENSE # Tests/results (not in .gitignore) noxfile.py -test_translate.py pylintrc pylintrc.test diff --git a/mod7-gaetasks/.gcloudignore b/mod7-gaetasks/.gcloudignore index b4dbc7e..af73b5d 100644 --- a/mod7-gaetasks/.gcloudignore +++ b/mod7-gaetasks/.gcloudignore @@ -20,7 +20,6 @@ LICENSE # Tests/results (not in .gitignore) noxfile.py -test_translate.py pylintrc pylintrc.test diff --git a/mod7b-gaetasks/.gcloudignore b/mod7b-gaetasks/.gcloudignore index b4dbc7e..af73b5d 100644 --- a/mod7b-gaetasks/.gcloudignore +++ b/mod7b-gaetasks/.gcloudignore @@ -20,7 +20,6 @@ LICENSE # Tests/results (not in .gitignore) noxfile.py -test_translate.py pylintrc pylintrc.test diff --git a/mod8-cloudtasks/.gcloudignore b/mod8-cloudtasks/.gcloudignore index b4dbc7e..af73b5d 100644 --- a/mod8-cloudtasks/.gcloudignore +++ b/mod8-cloudtasks/.gcloudignore @@ -20,7 +20,6 @@ LICENSE # Tests/results (not in .gitignore) noxfile.py -test_translate.py pylintrc pylintrc.test diff --git a/mod9-py3dstasks/.gcloudignore b/mod9-py3dstasks/.gcloudignore index b4dbc7e..af73b5d 100644 --- a/mod9-py3dstasks/.gcloudignore +++ b/mod9-py3dstasks/.gcloudignore @@ -20,7 +20,6 @@ LICENSE # Tests/results (not in .gitignore) noxfile.py -test_translate.py pylintrc pylintrc.test From 32fcf8c6a598dceec08b3efb3851d1d4ca66d0f2 Mon Sep 17 00:00:00 2001 From: wesley chun Date: Mon, 19 Sep 2022 17:02:51 -0700 Subject: [PATCH 43/56] add tags and Module 20 --- README.md | 3 + mod1-flask/templates/index.html | 1 + mod11-functions/templates/index.html | 1 + mod12-memcache/templates/index.html | 1 + mod12b-memcache/templates/index.html | 1 + mod13a-memorystore/templates/index.html | 1 + mod13b-memorystore/templates/index.html | 1 + mod15-blobstore/templates/index.html | 1 + mod16-cloudstorage/templates/index.html | 1 + mod18-gaepull/templates/index.html | 1 + mod19-pubsub/templates/index.html | 1 + mod1b-flask/templates/index.html | 1 + mod20-gaeusers/.gcloudignore | 79 +++++++++++++++++++++++++ mod20-gaeusers/README.md | 3 + mod20-gaeusers/app.yaml | 21 +++++++ mod20-gaeusers/appengine_config.py | 20 +++++++ mod20-gaeusers/main.py | 59 ++++++++++++++++++ mod20-gaeusers/requirements.txt | 1 + mod20-gaeusers/templates/index.html | 26 ++++++++ mod2a-cloudndb/templates/index.html | 1 + mod2b-cloudndb/templates/index.html | 1 + mod3a-datastore/templates/index.html | 1 + mod3b-datastore/templates/index.html | 1 + mod4a-rundocker/templates/index.html | 1 + mod4b-rundocker/templates/index.html | 1 + mod5-runbldpks/templates/index.html | 1 + mod6-firestore/templates/index.html | 1 + mod7-gaetasks/templates/index.html | 1 + mod7b-gaetasks/templates/index.html | 1 + mod8-cloudtasks/templates/index.html | 1 + mod9-py3dstasks/templates/index.html | 1 + 31 files changed, 235 insertions(+) create mode 100644 mod20-gaeusers/.gcloudignore create mode 100644 mod20-gaeusers/README.md create mode 100644 mod20-gaeusers/app.yaml create mode 100644 mod20-gaeusers/appengine_config.py create mode 100644 mod20-gaeusers/main.py create mode 100644 mod20-gaeusers/requirements.txt create mode 100644 mod20-gaeusers/templates/index.html diff --git a/README.md b/README.md index b3066b3..fe76aa7 100644 --- a/README.md +++ b/README.md @@ -84,6 +84,7 @@ Memcache | [12 ⇒] 13 | Moving off App Engine `memcache` makes your apps mor Cloud Functions | 11 | Cloud Functions does not support Python 2, so after the Module 1 migration, you need to upgrade your app to Python 3 before attempting this migration, recommended if you have a very small App Engine app, or it has only one function/feature. Cloud Run | 4 or 5 | **Module 4** covers migrating to Cloud Run with Docker. Those unfamiliar with containers or do not wish to create/maintain a `Dockerfile` should do **Module 5**. Those doing **Module 4** will get additional information about Cloud Run in **Module 5** not covered in **Module 4**. Blobstore | [15 ⇒] 16 | Moving off App Engine `blobstore` makes your apps more portable, so the **Module 16** Cloud Storage migration is _recommended_ for those using `blobstore`. Those unfamiliar with `blobstore` should do **Module 15** first to add its usage to the sample app. +Users | [20 ⇒] 21 | Moving off App Engine `users` makes your apps more portable, so the **Module 21** Cloud Storage migration is _recommended_ for those using `users`. Those unfamiliar with `users` should do **Module 20** first to add its usage to the sample app. General migration | 6 ⇒ 10 ⇒ 14 | This series is more generic and not targeting a specific feature migration, but rather if you need to migrate your App Engine apps from one running project to another. It starts with **Module 6** if you need to migrate your code, say from Datastore to Firestore. **Module 10** is if you need to migrate your data from one project to another, and finally, **Module 14** is after you're done migrating your app, your data, or both, and need to migrate a running service on one GCP project to another. @@ -116,6 +117,8 @@ Module | Topic | Video | Codelab | START here | FINISH here 17|Migrate to Python 3 bundled services| _TBD_ | [link](http://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-17-bundled?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrwormhole_sms_202002&utm_content=-) | Module 1 [code](/mod1-flask) (2.x) | Module 1 [code](/mod1b-flask) (3.x) 18|Add App Engine `taskqueue` pull tasks| _TBD_ | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-18-gaepull?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrgaepull_sms_202013&utm_content=-) | Module 1 [code](/mod1-flask) (2.x) | Module 18 [code](/mod18-gaepull) (2.x) 19|Migrate to Cloud Pub/Sub| _TBD_ | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-19-pubsub?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrpubsub_sms_202016&utm_content=-) | Module 18 [code](/mod18-gaepull) (2.x) | Module 19 [code](/mod19-pubsub) (2.x & 3.x) +20|Add App Engine `users` | _TBD_ | _TBD_ | Module 1 [code](/mod1-flask) (2.x) | Module 20 [code](/mod20-gaeusers) (2.x) +21|Migrate to Cloud Identity Platform | _TBD_ | _TBD_ | Module 20 [code](/mod20-gaeusers) (2.x) | _TBD_ ### Table of contents diff --git a/mod1-flask/templates/index.html b/mod1-flask/templates/index.html index e140206..920068e 100644 --- a/mod1-flask/templates/index.html +++ b/mod1-flask/templates/index.html @@ -2,6 +2,7 @@ VisitMe Example +

VisitMe example

diff --git a/mod11-functions/templates/index.html b/mod11-functions/templates/index.html index e140206..920068e 100644 --- a/mod11-functions/templates/index.html +++ b/mod11-functions/templates/index.html @@ -2,6 +2,7 @@ VisitMe Example +

VisitMe example

diff --git a/mod12-memcache/templates/index.html b/mod12-memcache/templates/index.html index e140206..920068e 100644 --- a/mod12-memcache/templates/index.html +++ b/mod12-memcache/templates/index.html @@ -2,6 +2,7 @@ VisitMe Example +

VisitMe example

diff --git a/mod12b-memcache/templates/index.html b/mod12b-memcache/templates/index.html index e140206..920068e 100644 --- a/mod12b-memcache/templates/index.html +++ b/mod12b-memcache/templates/index.html @@ -2,6 +2,7 @@ VisitMe Example +

VisitMe example

diff --git a/mod13a-memorystore/templates/index.html b/mod13a-memorystore/templates/index.html index e140206..920068e 100644 --- a/mod13a-memorystore/templates/index.html +++ b/mod13a-memorystore/templates/index.html @@ -2,6 +2,7 @@ VisitMe Example +

VisitMe example

diff --git a/mod13b-memorystore/templates/index.html b/mod13b-memorystore/templates/index.html index e140206..920068e 100644 --- a/mod13b-memorystore/templates/index.html +++ b/mod13b-memorystore/templates/index.html @@ -2,6 +2,7 @@ VisitMe Example +

VisitMe example

diff --git a/mod15-blobstore/templates/index.html b/mod15-blobstore/templates/index.html index 6f6be5f..e0e6a75 100644 --- a/mod15-blobstore/templates/index.html +++ b/mod15-blobstore/templates/index.html @@ -2,6 +2,7 @@ VisitMe Example +

VisitMe example

diff --git a/mod16-cloudstorage/templates/index.html b/mod16-cloudstorage/templates/index.html index 6f6be5f..e0e6a75 100644 --- a/mod16-cloudstorage/templates/index.html +++ b/mod16-cloudstorage/templates/index.html @@ -2,6 +2,7 @@ VisitMe Example +

VisitMe example

diff --git a/mod18-gaepull/templates/index.html b/mod18-gaepull/templates/index.html index b88f744..a280ed1 100644 --- a/mod18-gaepull/templates/index.html +++ b/mod18-gaepull/templates/index.html @@ -2,6 +2,7 @@ VisitMe Example +

VisitMe example

diff --git a/mod19-pubsub/templates/index.html b/mod19-pubsub/templates/index.html index 19bf331..06c7384 100644 --- a/mod19-pubsub/templates/index.html +++ b/mod19-pubsub/templates/index.html @@ -2,6 +2,7 @@ VisitMe Example +

VisitMe example

diff --git a/mod1b-flask/templates/index.html b/mod1b-flask/templates/index.html index e140206..920068e 100644 --- a/mod1b-flask/templates/index.html +++ b/mod1b-flask/templates/index.html @@ -2,6 +2,7 @@ VisitMe Example +

VisitMe example

diff --git a/mod20-gaeusers/.gcloudignore b/mod20-gaeusers/.gcloudignore new file mode 100644 index 0000000..af73b5d --- /dev/null +++ b/mod20-gaeusers/.gcloudignore @@ -0,0 +1,79 @@ +# This file specifies files that are *not* uploaded to Google Cloud Platform +# using gcloud. It follows the same syntax as .gitignore, with the addition of +# "#!include" directives (which insert the entries of the given .gitignore-style +# file at that point). +# +# For more information, run: +# $ gcloud topic gcloudignore +# +.gcloudignore + +# Source code control files +.git/ +.gitignore +.hgignore +.hg/ + +# README/text files +LICENSE +*.md + +# Tests/results (not in .gitignore) +noxfile.py +pylintrc +pylintrc.test + +# most of .gitignore (except `lib`) +# +# Python +*.py[cod] +__pycache__/ +/setup.cfg + +# C extensions +*.so + +# Packages +*.egg +*.egg-info +dist +build +eggs +.eggs +parts +bin +var +sdist +develop-eggs +.installed.cfg +lib64 +*.tgz + +# Installer logs +pip-log.txt + +# Tests/results +.nox/ +.pytest_cache/ +.cache +.pytype +.coverage +coverage.xml +*sponge_log.xml +system_tests/local_test_setup + +# Mac +.DS_Store + +# IDEs/editors +*.sw[op] +*~ +.vscode +.idea + +# Built documentation +docs/_build +docs.metadata + +# Virtual environment +env/ diff --git a/mod20-gaeusers/README.md b/mod20-gaeusers/README.md new file mode 100644 index 0000000..bc50f8e --- /dev/null +++ b/mod20-gaeusers/README.md @@ -0,0 +1,3 @@ +# Module 20 - Add usage of App Engine `users` to Flask `ndb` sample app + +This repo folder is the corresponding (Python 2) code to the _forthcoming_ Module 20 codelab. The tutorial STARTs with the Python 2 code in the [Module 1 repo folder](/mod1-flask) and leads developers through adding usage of the Users API via App Engine `users`, culminating in the code in this folder. diff --git a/mod20-gaeusers/app.yaml b/mod20-gaeusers/app.yaml new file mode 100644 index 0000000..0d4d8d6 --- /dev/null +++ b/mod20-gaeusers/app.yaml @@ -0,0 +1,21 @@ +# Copyright 2021 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +runtime: python27 +threadsafe: yes +api_version: 1 + +handlers: +- url: /.* + script: main.app diff --git a/mod20-gaeusers/appengine_config.py b/mod20-gaeusers/appengine_config.py new file mode 100644 index 0000000..0ca8634 --- /dev/null +++ b/mod20-gaeusers/appengine_config.py @@ -0,0 +1,20 @@ +# Copyright 2020 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from google.appengine.ext import vendor + +# Set PATH to your libraries folder. +PATH = 'lib' +# Add libraries installed in the PATH folder. +vendor.add(PATH) diff --git a/mod20-gaeusers/main.py b/mod20-gaeusers/main.py new file mode 100644 index 0000000..8b706f5 --- /dev/null +++ b/mod20-gaeusers/main.py @@ -0,0 +1,59 @@ +# Copyright 2022 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from flask import Flask, render_template, request +from google.appengine.api import app_identity, users +from google.appengine.ext import ndb + +app = Flask(__name__) + + +class Visit(ndb.Model): + 'Visit entity registers visitor IP address & timestamp' + visitor = ndb.StringProperty() + timestamp = ndb.DateTimeProperty(auto_now_add=True) + +def store_visit(remote_addr, user_agent): + 'create new Visit entity in Datastore' + Visit(visitor='{}: {}'.format(remote_addr, user_agent)).put() + +def fetch_visits(limit): + 'get most recent visits' + return Visit.query().order(-Visit.timestamp).fetch(limit) + + +@app.route('/') +def root(): + 'main application (GET) handler' + store_visit(request.remote_addr, request.user_agent) + visits = fetch_visits(10) + + # put together users context for web template + user = users.get_current_user() + context = { # logged in + 'who': user.nickname(), + 'admin': '(admin)' if users.is_current_user_admin() else '', + 'sign': 'Logout', + 'link': '/_ah/logout?continue=https://%s/' % \ + app_identity.get_default_version_hostname() + } if user else { # not logged in + 'who': 'user', + 'admin': '', + 'sign': 'Login', + 'link': users.create_login_url('https://github.com/'), + } + + # add visits to context and render template + context['visits'] = visits + return render_template('index.html', **context) diff --git a/mod20-gaeusers/requirements.txt b/mod20-gaeusers/requirements.txt new file mode 100644 index 0000000..7e10602 --- /dev/null +++ b/mod20-gaeusers/requirements.txt @@ -0,0 +1 @@ +flask diff --git a/mod20-gaeusers/templates/index.html b/mod20-gaeusers/templates/index.html new file mode 100644 index 0000000..239c014 --- /dev/null +++ b/mod20-gaeusers/templates/index.html @@ -0,0 +1,26 @@ + + + +VisitMe Example + + + +

+Welcome, {{ who }} {{ admin }} +


+ +

VisitMe example

+

Last 10 visits

+
    +{% for visit in visits %} +
  • {{ visit.timestamp.ctime() }} from {{ visit.visitor }}
  • +{% endfor %} +
+ + + + diff --git a/mod2a-cloudndb/templates/index.html b/mod2a-cloudndb/templates/index.html index e140206..920068e 100644 --- a/mod2a-cloudndb/templates/index.html +++ b/mod2a-cloudndb/templates/index.html @@ -2,6 +2,7 @@ VisitMe Example +

VisitMe example

diff --git a/mod2b-cloudndb/templates/index.html b/mod2b-cloudndb/templates/index.html index e140206..920068e 100644 --- a/mod2b-cloudndb/templates/index.html +++ b/mod2b-cloudndb/templates/index.html @@ -2,6 +2,7 @@ VisitMe Example +

VisitMe example

diff --git a/mod3a-datastore/templates/index.html b/mod3a-datastore/templates/index.html index e140206..920068e 100644 --- a/mod3a-datastore/templates/index.html +++ b/mod3a-datastore/templates/index.html @@ -2,6 +2,7 @@ VisitMe Example +

VisitMe example

diff --git a/mod3b-datastore/templates/index.html b/mod3b-datastore/templates/index.html index e140206..920068e 100644 --- a/mod3b-datastore/templates/index.html +++ b/mod3b-datastore/templates/index.html @@ -2,6 +2,7 @@ VisitMe Example +

VisitMe example

diff --git a/mod4a-rundocker/templates/index.html b/mod4a-rundocker/templates/index.html index e140206..920068e 100644 --- a/mod4a-rundocker/templates/index.html +++ b/mod4a-rundocker/templates/index.html @@ -2,6 +2,7 @@ VisitMe Example +

VisitMe example

diff --git a/mod4b-rundocker/templates/index.html b/mod4b-rundocker/templates/index.html index e140206..920068e 100644 --- a/mod4b-rundocker/templates/index.html +++ b/mod4b-rundocker/templates/index.html @@ -2,6 +2,7 @@ VisitMe Example +

VisitMe example

diff --git a/mod5-runbldpks/templates/index.html b/mod5-runbldpks/templates/index.html index e140206..920068e 100644 --- a/mod5-runbldpks/templates/index.html +++ b/mod5-runbldpks/templates/index.html @@ -2,6 +2,7 @@ VisitMe Example +

VisitMe example

diff --git a/mod6-firestore/templates/index.html b/mod6-firestore/templates/index.html index e140206..920068e 100644 --- a/mod6-firestore/templates/index.html +++ b/mod6-firestore/templates/index.html @@ -2,6 +2,7 @@ VisitMe Example +

VisitMe example

diff --git a/mod7-gaetasks/templates/index.html b/mod7-gaetasks/templates/index.html index 952961e..8a0b9cb 100644 --- a/mod7-gaetasks/templates/index.html +++ b/mod7-gaetasks/templates/index.html @@ -2,6 +2,7 @@ VisitMe Example +

VisitMe example

diff --git a/mod7b-gaetasks/templates/index.html b/mod7b-gaetasks/templates/index.html index 952961e..8a0b9cb 100644 --- a/mod7b-gaetasks/templates/index.html +++ b/mod7b-gaetasks/templates/index.html @@ -2,6 +2,7 @@ VisitMe Example +

VisitMe example

diff --git a/mod8-cloudtasks/templates/index.html b/mod8-cloudtasks/templates/index.html index 952961e..8a0b9cb 100644 --- a/mod8-cloudtasks/templates/index.html +++ b/mod8-cloudtasks/templates/index.html @@ -2,6 +2,7 @@ VisitMe Example +

VisitMe example

diff --git a/mod9-py3dstasks/templates/index.html b/mod9-py3dstasks/templates/index.html index 952961e..8a0b9cb 100644 --- a/mod9-py3dstasks/templates/index.html +++ b/mod9-py3dstasks/templates/index.html @@ -2,6 +2,7 @@ VisitMe Example +

VisitMe example

From 0108eaf7044546a700b600e5243f43e88425135b Mon Sep 17 00:00:00 2001 From: wesley chun Date: Wed, 21 Sep 2022 11:43:31 -0700 Subject: [PATCH 44/56] Mod20 fixes and added Mod21 --- README.md | 2 +- mod20-gaeusers/main.py | 10 ++-- mod20-gaeusers/templates/index.html | 6 +- mod21a-idenplat/.gcloudignore | 79 +++++++++++++++++++++++++ mod21a-idenplat/README.md | 3 + mod21a-idenplat/app.yaml | 29 ++++++++++ mod21a-idenplat/appengine_config.py | 23 ++++++++ mod21a-idenplat/main.py | 78 +++++++++++++++++++++++++ mod21a-idenplat/requirements.txt | 13 +++++ mod21a-idenplat/templates/index.html | 87 ++++++++++++++++++++++++++++ mod21b-idenplat/.gcloudignore | 79 +++++++++++++++++++++++++ mod21b-idenplat/README.md | 9 +++ mod21b-idenplat/app.yaml | 15 +++++ mod21b-idenplat/main.py | 78 +++++++++++++++++++++++++ mod21b-idenplat/requirements.txt | 5 ++ mod21b-idenplat/templates/index.html | 87 ++++++++++++++++++++++++++++ 16 files changed, 595 insertions(+), 8 deletions(-) create mode 100644 mod21a-idenplat/.gcloudignore create mode 100644 mod21a-idenplat/README.md create mode 100644 mod21a-idenplat/app.yaml create mode 100644 mod21a-idenplat/appengine_config.py create mode 100644 mod21a-idenplat/main.py create mode 100644 mod21a-idenplat/requirements.txt create mode 100644 mod21a-idenplat/templates/index.html create mode 100644 mod21b-idenplat/.gcloudignore create mode 100644 mod21b-idenplat/README.md create mode 100644 mod21b-idenplat/app.yaml create mode 100644 mod21b-idenplat/main.py create mode 100644 mod21b-idenplat/requirements.txt create mode 100644 mod21b-idenplat/templates/index.html diff --git a/README.md b/README.md index fe76aa7..27e69f2 100644 --- a/README.md +++ b/README.md @@ -118,7 +118,7 @@ Module | Topic | Video | Codelab | START here | FINISH here 18|Add App Engine `taskqueue` pull tasks| _TBD_ | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-18-gaepull?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrgaepull_sms_202013&utm_content=-) | Module 1 [code](/mod1-flask) (2.x) | Module 18 [code](/mod18-gaepull) (2.x) 19|Migrate to Cloud Pub/Sub| _TBD_ | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-19-pubsub?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrpubsub_sms_202016&utm_content=-) | Module 18 [code](/mod18-gaepull) (2.x) | Module 19 [code](/mod19-pubsub) (2.x & 3.x) 20|Add App Engine `users` | _TBD_ | _TBD_ | Module 1 [code](/mod1-flask) (2.x) | Module 20 [code](/mod20-gaeusers) (2.x) -21|Migrate to Cloud Identity Platform | _TBD_ | _TBD_ | Module 20 [code](/mod20-gaeusers) (2.x) | _TBD_ +21|Migrate to Cloud Identity Platform | _TBD_ | _TBD_ | Module 20 [code](/mod20-gaeusers) (2.x) | Module 21 [code](/mod21a-idenplat) (2.x) & [code](/mod21b-idenplat) (3.x) ### Table of contents diff --git a/mod20-gaeusers/main.py b/mod20-gaeusers/main.py index 8b706f5..14d9126 100644 --- a/mod20-gaeusers/main.py +++ b/mod20-gaeusers/main.py @@ -13,7 +13,7 @@ # limitations under the License. from flask import Flask, render_template, request -from google.appengine.api import app_identity, users +from google.appengine.api import users from google.appengine.ext import ndb app = Flask(__name__) @@ -45,8 +45,10 @@ def root(): 'who': user.nickname(), 'admin': '(admin)' if users.is_current_user_admin() else '', 'sign': 'Logout', - 'link': '/_ah/logout?continue=https://%s/' % \ - app_identity.get_default_version_hostname() + 'link': '/_ah/logout?continue=%s://%s/' % ( + request.environ['wsgi.url_scheme'], + request.environ['HTTP_HOST'], + ), # alternative to users.create_logout_url() } if user else { # not logged in 'who': 'user', 'admin': '', @@ -55,5 +57,5 @@ def root(): } # add visits to context and render template - context['visits'] = visits + context['visits'] = visits # display whether logged in or not return render_template('index.html', **context) diff --git a/mod20-gaeusers/templates/index.html b/mod20-gaeusers/templates/index.html index 239c014..13a7e37 100644 --- a/mod20-gaeusers/templates/index.html +++ b/mod20-gaeusers/templates/index.html @@ -3,10 +3,10 @@ VisitMe Example -

-Welcome, {{ who }} {{ admin }} +Welcome, {{ who }} {{ admin }} +


VisitMe example

@@ -18,7 +18,7 @@

Last 10 visits

diff --git a/mod21a-idenplat/.gcloudignore b/mod21a-idenplat/.gcloudignore new file mode 100644 index 0000000..af73b5d --- /dev/null +++ b/mod21a-idenplat/.gcloudignore @@ -0,0 +1,79 @@ +# This file specifies files that are *not* uploaded to Google Cloud Platform +# using gcloud. It follows the same syntax as .gitignore, with the addition of +# "#!include" directives (which insert the entries of the given .gitignore-style +# file at that point). +# +# For more information, run: +# $ gcloud topic gcloudignore +# +.gcloudignore + +# Source code control files +.git/ +.gitignore +.hgignore +.hg/ + +# README/text files +LICENSE +*.md + +# Tests/results (not in .gitignore) +noxfile.py +pylintrc +pylintrc.test + +# most of .gitignore (except `lib`) +# +# Python +*.py[cod] +__pycache__/ +/setup.cfg + +# C extensions +*.so + +# Packages +*.egg +*.egg-info +dist +build +eggs +.eggs +parts +bin +var +sdist +develop-eggs +.installed.cfg +lib64 +*.tgz + +# Installer logs +pip-log.txt + +# Tests/results +.nox/ +.pytest_cache/ +.cache +.pytype +.coverage +coverage.xml +*sponge_log.xml +system_tests/local_test_setup + +# Mac +.DS_Store + +# IDEs/editors +*.sw[op] +*~ +.vscode +.idea + +# Built documentation +docs/_build +docs.metadata + +# Virtual environment +env/ diff --git a/mod21a-idenplat/README.md b/mod21a-idenplat/README.md new file mode 100644 index 0000000..349501c --- /dev/null +++ b/mod21a-idenplat/README.md @@ -0,0 +1,3 @@ +# Module 21 - Migrate from App Engine `users` to Cloud Identity Platform + +This repo folder is the corresponding Python 2 code to the _forthcoming_ Module 21 codelab. The tutorial STARTs with the Python 2 code in the [Module 20 repo folder](/mod20-gaeusers) and leads developers through a migration to Cloud Identity Platform, culminating in the code in this (`mod21a-idenplat`) folder. Also included is a migration from App Engine `ndb` to Google Cloud NDB, mirroring the content covered in [Module 2](http://g.co/codelabs/pae-migrate-cloudndb). There is also a Python 3 version of the app in the [Module 21b](/mod21b-idenplat) folder. diff --git a/mod21a-idenplat/app.yaml b/mod21a-idenplat/app.yaml new file mode 100644 index 0000000..bec5174 --- /dev/null +++ b/mod21a-idenplat/app.yaml @@ -0,0 +1,29 @@ +# Copyright 2021 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +runtime: python27 +threadsafe: yes +api_version: 1 + +handlers: +- url: /.* + script: main.app + +libraries: +- name: grpcio + version: latest +- name: setuptools + version: latest +- name: ssl + version: latest diff --git a/mod21a-idenplat/appengine_config.py b/mod21a-idenplat/appengine_config.py new file mode 100644 index 0000000..2a41fb4 --- /dev/null +++ b/mod21a-idenplat/appengine_config.py @@ -0,0 +1,23 @@ +# Copyright 2021 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import pkg_resources +from google.appengine.ext import vendor + +# Set PATH to your libraries folder. +PATH = 'lib' +# Add libraries installed in the PATH folder. +vendor.add(PATH) +# Add libraries to pkg_resources working set to find the distribution. +pkg_resources.working_set.add_entry(PATH) diff --git a/mod21a-idenplat/main.py b/mod21a-idenplat/main.py new file mode 100644 index 0000000..0f9280e --- /dev/null +++ b/mod21a-idenplat/main.py @@ -0,0 +1,78 @@ +# Copyright 2022 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from flask import Flask, render_template, request +from google.auth import default +from google.cloud import ndb +from googleapiclient import discovery +from firebase_admin import auth, initialize_app + +def _get_gae_admins(): + 'return set of App Engine admins' + # setup constants for calling Cloud IAM Resource Manager + CREDS, PROJ_ID = default( # Application Default Credentials and project ID + ['https://www.googleapis.com/auth/cloud-platform']) + IAM = discovery.build('cloudresourcemanager', 'v1', credentials=CREDS) + _TARGETS = frozenset(( # App Engine admin roles + 'roles/viewer', + 'roles/editor', + 'roles/owner', + 'roles/appengine.appAdmin', + )) + + # collate all users who are members of at least one GAE admin role (TARGETS) + admins = set() # set of all App Engine admins + allow_policy = IAM.projects().getIamPolicy(resource=PROJ_ID).execute() + for b in allow_policy['bindings']: # bindings in IAM allow policy + if b['role'] in _TARGETS: # only look at GAE admin roles + admins.update(user.split(':', 1)[1] for user in b['members']) + return admins + +@app.route('/is_admin', methods=['POST']) +def is_admin(): + 'check if user (via their Firebase ID token) is GAE admin (POST) handler' + id_token = request.headers.get('Authorization') + email = auth.verify_id_token(id_token).get('email') + return {'admin': email in _ADMINS}, 200 + + +# initialize Flask, Firebase, Cloud NDB; fetch set of App Engine admins +app = Flask(__name__) +initialize_app() +ds_client = ndb.Client() +_ADMINS = _get_gae_admins() + + +class Visit(ndb.Model): + 'Visit entity registers visitor IP address & timestamp' + visitor = ndb.StringProperty() + timestamp = ndb.DateTimeProperty(auto_now_add=True) + +def store_visit(remote_addr, user_agent): + 'create new Visit entity in Datastore' + with ds_client.context(): + Visit(visitor='{}: {}'.format(remote_addr, user_agent)).put() + +def fetch_visits(limit): + 'get most recent visits' + with ds_client.context(): + return Visit.query().order(-Visit.timestamp).fetch(limit) + + +@app.route('/') +def root(): + 'main application (GET) handler' + store_visit(request.remote_addr, request.user_agent) + visits = fetch_visits(10) + return render_template('index.html', visits=visits) diff --git a/mod21a-idenplat/requirements.txt b/mod21a-idenplat/requirements.txt new file mode 100644 index 0000000..1840686 --- /dev/null +++ b/mod21a-idenplat/requirements.txt @@ -0,0 +1,13 @@ +grpcio==1.0.0 +protobuf<3.18.0 +six>=1.13.0 +flask +google-gax<0.13.0 +google-api-core==1.31.1 +google-api-python-client<=1.11.0 +google-auth<2.0dev +google-cloud-datastore==1.15.3 +google-cloud-firestore==1.9.0 +google-cloud-ndb +google-cloud-pubsub==1.7.0 +firebase-admin diff --git a/mod21a-idenplat/templates/index.html b/mod21a-idenplat/templates/index.html new file mode 100644 index 0000000..fe6167c --- /dev/null +++ b/mod21a-idenplat/templates/index.html @@ -0,0 +1,87 @@ + + + +VisitMe Example + + + + + +

+Welcome, (admin) + +


+ +

VisitMe example

+

Last 10 visits

+
    +{% for visit in visits %} +
  • {{ visit.timestamp.ctime() }} from {{ visit.visitor }}
  • +{% endfor %} +
+ + + + diff --git a/mod21b-idenplat/.gcloudignore b/mod21b-idenplat/.gcloudignore new file mode 100644 index 0000000..af73b5d --- /dev/null +++ b/mod21b-idenplat/.gcloudignore @@ -0,0 +1,79 @@ +# This file specifies files that are *not* uploaded to Google Cloud Platform +# using gcloud. It follows the same syntax as .gitignore, with the addition of +# "#!include" directives (which insert the entries of the given .gitignore-style +# file at that point). +# +# For more information, run: +# $ gcloud topic gcloudignore +# +.gcloudignore + +# Source code control files +.git/ +.gitignore +.hgignore +.hg/ + +# README/text files +LICENSE +*.md + +# Tests/results (not in .gitignore) +noxfile.py +pylintrc +pylintrc.test + +# most of .gitignore (except `lib`) +# +# Python +*.py[cod] +__pycache__/ +/setup.cfg + +# C extensions +*.so + +# Packages +*.egg +*.egg-info +dist +build +eggs +.eggs +parts +bin +var +sdist +develop-eggs +.installed.cfg +lib64 +*.tgz + +# Installer logs +pip-log.txt + +# Tests/results +.nox/ +.pytest_cache/ +.cache +.pytype +.coverage +coverage.xml +*sponge_log.xml +system_tests/local_test_setup + +# Mac +.DS_Store + +# IDEs/editors +*.sw[op] +*~ +.vscode +.idea + +# Built documentation +docs/_build +docs.metadata + +# Virtual environment +env/ diff --git a/mod21b-idenplat/README.md b/mod21b-idenplat/README.md new file mode 100644 index 0000000..e03bad5 --- /dev/null +++ b/mod21b-idenplat/README.md @@ -0,0 +1,9 @@ +# Module 21 - Migrate from App Engine `users` to Cloud Identity Platform + +This repo folder is the corresponding Python 3 version of the Module 21 app. + +- All files in this folder are identical to the _Python 2_ code in the [Module 21a repo folder](/mod21a-idenplat) **except**: + 1. `app.yaml` was modified for the Python 3 runtime. + 1. `appengine_config.py` is unused and thus deleted. +- An optional migration from Cloud NDB to Cloud Datastore can be achieved via the content covered in [Module 3](http://g.co/codelabs/pae-migrate-datastore). +- The _Python 3_ version of the Module 20 app ([Module 20b repo folder](/mod20b-gaeusers)) features additional code to support those App Engine legacy ("bundled") services (like `memcache`). Because the app in this folder does not use such services (moved to Cloud Memorystore), that extra support does not appear, so the code here should not be considered a direct migration of that app to Cloud Memorystore (and Cloud NDB), unlike the Python 2 equivalents (Modules [20](/mod20-gaeusers) and [21a](/mod21a-idenplat)) which can. diff --git a/mod21b-idenplat/app.yaml b/mod21b-idenplat/app.yaml new file mode 100644 index 0000000..9aac415 --- /dev/null +++ b/mod21b-idenplat/app.yaml @@ -0,0 +1,15 @@ +# Copyright 2022 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +runtime: python310 diff --git a/mod21b-idenplat/main.py b/mod21b-idenplat/main.py new file mode 100644 index 0000000..0f9280e --- /dev/null +++ b/mod21b-idenplat/main.py @@ -0,0 +1,78 @@ +# Copyright 2022 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from flask import Flask, render_template, request +from google.auth import default +from google.cloud import ndb +from googleapiclient import discovery +from firebase_admin import auth, initialize_app + +def _get_gae_admins(): + 'return set of App Engine admins' + # setup constants for calling Cloud IAM Resource Manager + CREDS, PROJ_ID = default( # Application Default Credentials and project ID + ['https://www.googleapis.com/auth/cloud-platform']) + IAM = discovery.build('cloudresourcemanager', 'v1', credentials=CREDS) + _TARGETS = frozenset(( # App Engine admin roles + 'roles/viewer', + 'roles/editor', + 'roles/owner', + 'roles/appengine.appAdmin', + )) + + # collate all users who are members of at least one GAE admin role (TARGETS) + admins = set() # set of all App Engine admins + allow_policy = IAM.projects().getIamPolicy(resource=PROJ_ID).execute() + for b in allow_policy['bindings']: # bindings in IAM allow policy + if b['role'] in _TARGETS: # only look at GAE admin roles + admins.update(user.split(':', 1)[1] for user in b['members']) + return admins + +@app.route('/is_admin', methods=['POST']) +def is_admin(): + 'check if user (via their Firebase ID token) is GAE admin (POST) handler' + id_token = request.headers.get('Authorization') + email = auth.verify_id_token(id_token).get('email') + return {'admin': email in _ADMINS}, 200 + + +# initialize Flask, Firebase, Cloud NDB; fetch set of App Engine admins +app = Flask(__name__) +initialize_app() +ds_client = ndb.Client() +_ADMINS = _get_gae_admins() + + +class Visit(ndb.Model): + 'Visit entity registers visitor IP address & timestamp' + visitor = ndb.StringProperty() + timestamp = ndb.DateTimeProperty(auto_now_add=True) + +def store_visit(remote_addr, user_agent): + 'create new Visit entity in Datastore' + with ds_client.context(): + Visit(visitor='{}: {}'.format(remote_addr, user_agent)).put() + +def fetch_visits(limit): + 'get most recent visits' + with ds_client.context(): + return Visit.query().order(-Visit.timestamp).fetch(limit) + + +@app.route('/') +def root(): + 'main application (GET) handler' + store_visit(request.remote_addr, request.user_agent) + visits = fetch_visits(10) + return render_template('index.html', visits=visits) diff --git a/mod21b-idenplat/requirements.txt b/mod21b-idenplat/requirements.txt new file mode 100644 index 0000000..10ec369 --- /dev/null +++ b/mod21b-idenplat/requirements.txt @@ -0,0 +1,5 @@ +flask +google-api-python-client +google-auth +google-cloud-ndb +firebase-admin diff --git a/mod21b-idenplat/templates/index.html b/mod21b-idenplat/templates/index.html new file mode 100644 index 0000000..fe6167c --- /dev/null +++ b/mod21b-idenplat/templates/index.html @@ -0,0 +1,87 @@ + + + +VisitMe Example + + + + + +

+Welcome, (admin) + +


+ +

VisitMe example

+

Last 10 visits

+
    +{% for visit in visits %} +
  • {{ visit.timestamp.ctime() }} from {{ visit.visitor }}
  • +{% endfor %} +
+ + + + From a3aa6b52e52ddf5d510cfa13df010ad1573e6d3b Mon Sep 17 00:00:00 2001 From: wesley chun Date: Thu, 29 Sep 2022 18:49:40 -0700 Subject: [PATCH 45/56] Mod21 client lib tweaks --- README.md | 2 +- mod21a-idenplat/main.py | 21 ++++++++++-------- mod21a-idenplat/templates/index.html | 9 +++++--- mod21b-idenplat/main.py | 32 +++++++++++++++------------- mod21b-idenplat/requirements.txt | 2 +- mod21b-idenplat/templates/index.html | 9 +++++--- 6 files changed, 43 insertions(+), 32 deletions(-) diff --git a/README.md b/README.md index 27e69f2..778a7b7 100644 --- a/README.md +++ b/README.md @@ -84,7 +84,7 @@ Memcache | [12 ⇒] 13 | Moving off App Engine `memcache` makes your apps mor Cloud Functions | 11 | Cloud Functions does not support Python 2, so after the Module 1 migration, you need to upgrade your app to Python 3 before attempting this migration, recommended if you have a very small App Engine app, or it has only one function/feature. Cloud Run | 4 or 5 | **Module 4** covers migrating to Cloud Run with Docker. Those unfamiliar with containers or do not wish to create/maintain a `Dockerfile` should do **Module 5**. Those doing **Module 4** will get additional information about Cloud Run in **Module 5** not covered in **Module 4**. Blobstore | [15 ⇒] 16 | Moving off App Engine `blobstore` makes your apps more portable, so the **Module 16** Cloud Storage migration is _recommended_ for those using `blobstore`. Those unfamiliar with `blobstore` should do **Module 15** first to add its usage to the sample app. -Users | [20 ⇒] 21 | Moving off App Engine `users` makes your apps more portable, so the **Module 21** Cloud Storage migration is _recommended_ for those using `users`. Those unfamiliar with `users` should do **Module 20** first to add its usage to the sample app. +Users | [20 ⇒] 21 | Moving off App Engine `users` makes your apps more portable, so the **Module 21** Cloud Identity Platform migration is _recommended_ for those using `users`. Those unfamiliar with `users` should do **Module 20** first to add its usage to the sample app. General migration | 6 ⇒ 10 ⇒ 14 | This series is more generic and not targeting a specific feature migration, but rather if you need to migrate your App Engine apps from one running project to another. It starts with **Module 6** if you need to migrate your code, say from Datastore to Firestore. **Module 10** is if you need to migrate your data from one project to another, and finally, **Module 14** is after you're done migrating your app, your data, or both, and need to migrate a running service on one GCP project to another. diff --git a/mod21a-idenplat/main.py b/mod21a-idenplat/main.py index 0f9280e..5f13b08 100644 --- a/mod21a-idenplat/main.py +++ b/mod21a-idenplat/main.py @@ -18,12 +18,17 @@ from googleapiclient import discovery from firebase_admin import auth, initialize_app +# initialize Flask and Cloud NDB API client +app = Flask(__name__) +ds_client = ndb.Client() + + def _get_gae_admins(): 'return set of App Engine admins' - # setup constants for calling Cloud IAM Resource Manager + # setup constants for calling Cloud IAM Resource Manager API CREDS, PROJ_ID = default( # Application Default Credentials and project ID ['https://www.googleapis.com/auth/cloud-platform']) - IAM = discovery.build('cloudresourcemanager', 'v1', credentials=CREDS) + rm_client = discovery.build('cloudresourcemanager', 'v1', credentials=CREDS) _TARGETS = frozenset(( # App Engine admin roles 'roles/viewer', 'roles/editor', @@ -31,12 +36,12 @@ def _get_gae_admins(): 'roles/appengine.appAdmin', )) - # collate all users who are members of at least one GAE admin role (TARGETS) - admins = set() # set of all App Engine admins - allow_policy = IAM.projects().getIamPolicy(resource=PROJ_ID).execute() + # collate all users who are members of at least one GAE admin role (_TARGETS) + admins = set() # set of all App Engine admins + allow_policy = rm_client.projects().getIamPolicy(resource=PROJ_ID).execute() for b in allow_policy['bindings']: # bindings in IAM allow policy if b['role'] in _TARGETS: # only look at GAE admin roles - admins.update(user.split(':', 1)[1] for user in b['members']) + admins.update(user.split(':', 1).pop() for user in b['members']) return admins @app.route('/is_admin', methods=['POST']) @@ -47,10 +52,8 @@ def is_admin(): return {'admin': email in _ADMINS}, 200 -# initialize Flask, Firebase, Cloud NDB; fetch set of App Engine admins -app = Flask(__name__) +# initialize Firebase and fetch set of App Engine admins initialize_app() -ds_client = ndb.Client() _ADMINS = _get_gae_admins() diff --git a/mod21a-idenplat/templates/index.html b/mod21a-idenplat/templates/index.html index fe6167c..0d1b97b 100644 --- a/mod21a-idenplat/templates/index.html +++ b/mod21a-idenplat/templates/index.html @@ -16,9 +16,11 @@ signOut } from "https://www.gstatic.com/firebasejs/9.10.0/firebase-auth.js"; -// Firebase config: at least 'apiKey' & 'authDomain' are required; go to -// console.firebase.google.com/project/PROJECT_ID/settings/general/web OR -// console.firebase.google.com/project/_/settings/general/web & pick project +// Firebase config: +// 1a. Go to: console.cloud.google.com/customer-identity/providers +// 1b. May be prompted to enable GCIP and upgrade from Firebase +// 2. Click: "Application Setup Details" button +// 3. Copy: 'apiKey' and 'authDomain' from 'config' variable var firebaseConfig = { apiKey: "YOUR_API_KEY", authDomain: "YOUR_AUTH_DOMAIN", @@ -28,6 +30,7 @@ initializeApp(firebaseConfig); var auth = getAuth(); var provider = new GoogleAuthProvider(); +//provider.setCustomParameters({prompt: 'select_account'}); // define login and logout button functions function login() { diff --git a/mod21b-idenplat/main.py b/mod21b-idenplat/main.py index 0f9280e..60b956f 100644 --- a/mod21b-idenplat/main.py +++ b/mod21b-idenplat/main.py @@ -14,16 +14,20 @@ from flask import Flask, render_template, request from google.auth import default -from google.cloud import ndb -from googleapiclient import discovery +from google.cloud import ndb, resourcemanager from firebase_admin import auth, initialize_app +# initialize Flask and Cloud NDB API client +app = Flask(__name__) +ds_client = ndb.Client() + + def _get_gae_admins(): 'return set of App Engine admins' - # setup constants for calling Cloud IAM Resource Manager - CREDS, PROJ_ID = default( # Application Default Credentials and project ID - ['https://www.googleapis.com/auth/cloud-platform']) - IAM = discovery.build('cloudresourcemanager', 'v1', credentials=CREDS) + # setup constants for calling Cloud IAM Resource Manager API + _, PROJ_ID = default( # Application Default Credentials and project ID + ['https://www.googleapis.com/auth/cloudplatformprojects.readonly']) + rm_client = resourcemanager.ProjectsClient() _TARGETS = frozenset(( # App Engine admin roles 'roles/viewer', 'roles/editor', @@ -31,12 +35,12 @@ def _get_gae_admins(): 'roles/appengine.appAdmin', )) - # collate all users who are members of at least one GAE admin role (TARGETS) - admins = set() # set of all App Engine admins - allow_policy = IAM.projects().getIamPolicy(resource=PROJ_ID).execute() - for b in allow_policy['bindings']: # bindings in IAM allow policy - if b['role'] in _TARGETS: # only look at GAE admin roles - admins.update(user.split(':', 1)[1] for user in b['members']) + # collate all users who are members of at least one GAE admin role (_TARGETS) + admins = set() # set of all App Engine admins + allow_policy = rm_client.get_iam_policy(resource='projects/%s' % PROJ_ID) + for b in allow_policy.bindings: # bindings in IAM allow policy + if b.role in _TARGETS: # only look at GAE admin roles + admins.update(user.split(':', 1).pop() for user in b.members) return admins @app.route('/is_admin', methods=['POST']) @@ -47,10 +51,8 @@ def is_admin(): return {'admin': email in _ADMINS}, 200 -# initialize Flask, Firebase, Cloud NDB; fetch set of App Engine admins -app = Flask(__name__) +# initialize Firebase and fetch set of App Engine admins initialize_app() -ds_client = ndb.Client() _ADMINS = _get_gae_admins() diff --git a/mod21b-idenplat/requirements.txt b/mod21b-idenplat/requirements.txt index 10ec369..e0008d7 100644 --- a/mod21b-idenplat/requirements.txt +++ b/mod21b-idenplat/requirements.txt @@ -1,5 +1,5 @@ flask -google-api-python-client google-auth google-cloud-ndb +google-cloud-resource-manager firebase-admin diff --git a/mod21b-idenplat/templates/index.html b/mod21b-idenplat/templates/index.html index fe6167c..0d1b97b 100644 --- a/mod21b-idenplat/templates/index.html +++ b/mod21b-idenplat/templates/index.html @@ -16,9 +16,11 @@ signOut } from "https://www.gstatic.com/firebasejs/9.10.0/firebase-auth.js"; -// Firebase config: at least 'apiKey' & 'authDomain' are required; go to -// console.firebase.google.com/project/PROJECT_ID/settings/general/web OR -// console.firebase.google.com/project/_/settings/general/web & pick project +// Firebase config: +// 1a. Go to: console.cloud.google.com/customer-identity/providers +// 1b. May be prompted to enable GCIP and upgrade from Firebase +// 2. Click: "Application Setup Details" button +// 3. Copy: 'apiKey' and 'authDomain' from 'config' variable var firebaseConfig = { apiKey: "YOUR_API_KEY", authDomain: "YOUR_AUTH_DOMAIN", @@ -28,6 +30,7 @@ initializeApp(firebaseConfig); var auth = getAuth(); var provider = new GoogleAuthProvider(); +//provider.setCustomParameters({prompt: 'select_account'}); // define login and logout button functions function login() { From c5bb5b6b2624384029579fd3e608d272fea6c760 Mon Sep 17 00:00:00 2001 From: wesley chun Date: Wed, 5 Oct 2022 17:12:50 -0700 Subject: [PATCH 46/56] minor Mod 21 updates --- mod21a-idenplat/README.md | 2 ++ mod21a-idenplat/main.py | 6 +++--- mod21b-idenplat/main.py | 4 ++-- 3 files changed, 7 insertions(+), 5 deletions(-) diff --git a/mod21a-idenplat/README.md b/mod21a-idenplat/README.md index 349501c..aa51a11 100644 --- a/mod21a-idenplat/README.md +++ b/mod21a-idenplat/README.md @@ -1,3 +1,5 @@ # Module 21 - Migrate from App Engine `users` to Cloud Identity Platform This repo folder is the corresponding Python 2 code to the _forthcoming_ Module 21 codelab. The tutorial STARTs with the Python 2 code in the [Module 20 repo folder](/mod20-gaeusers) and leads developers through a migration to Cloud Identity Platform, culminating in the code in this (`mod21a-idenplat`) folder. Also included is a migration from App Engine `ndb` to Google Cloud NDB, mirroring the content covered in [Module 2](http://g.co/codelabs/pae-migrate-cloudndb). There is also a Python 3 version of the app in the [Module 21b](/mod21b-idenplat) folder. + +NOTE: While we generally recommend using [Google Cloud client libraries](https://cloud.google.com/apis/docs/cloud-client-libraries) for GCP API access, we have an exception here because the [final Python 2 version](https://googleapis.dev/python/cloudresourcemanager/0.30.2) of the [Cloud Resource Manager client library](https://github.com/googleapis/python-resource-manager) (before the 2.x support was deprecated) did not have an implemented [get IAM policy](https://cloud.google.com/python/docs/reference/cloudresourcemanager/latest/google.cloud.resourcemanager_v3.services.projects.ProjectsClient#google_cloud_resourcemanager_v3_services_projects_ProjectsClient_get_iam_policy) feature, hence the need to use the [lower-level Google APIs client library](https://developers.google.com/api-client-library) to access this functionality from the API. See the [Python 3 `main.py`](/mod21b-idenplat/main.py) which uses latest Resource Manager client library. diff --git a/mod21a-idenplat/main.py b/mod21a-idenplat/main.py index 5f13b08..967e0ee 100644 --- a/mod21a-idenplat/main.py +++ b/mod21a-idenplat/main.py @@ -25,9 +25,9 @@ def _get_gae_admins(): 'return set of App Engine admins' - # setup constants for calling Cloud IAM Resource Manager API + # setup constants for calling Cloud Resource Manager API CREDS, PROJ_ID = default( # Application Default Credentials and project ID - ['https://www.googleapis.com/auth/cloud-platform']) + ['https://www.googleapis.com/auth/cloudplatformprojects.readonly']) rm_client = discovery.build('cloudresourcemanager', 'v1', credentials=CREDS) _TARGETS = frozenset(( # App Engine admin roles 'roles/viewer', @@ -36,7 +36,7 @@ def _get_gae_admins(): 'roles/appengine.appAdmin', )) - # collate all users who are members of at least one GAE admin role (_TARGETS) + # collate users who are members of at least one GAE admin role (_TARGETS) admins = set() # set of all App Engine admins allow_policy = rm_client.projects().getIamPolicy(resource=PROJ_ID).execute() for b in allow_policy['bindings']: # bindings in IAM allow policy diff --git a/mod21b-idenplat/main.py b/mod21b-idenplat/main.py index 60b956f..10fe5cb 100644 --- a/mod21b-idenplat/main.py +++ b/mod21b-idenplat/main.py @@ -24,7 +24,7 @@ def _get_gae_admins(): 'return set of App Engine admins' - # setup constants for calling Cloud IAM Resource Manager API + # setup constants for calling Cloud Resource Manager API _, PROJ_ID = default( # Application Default Credentials and project ID ['https://www.googleapis.com/auth/cloudplatformprojects.readonly']) rm_client = resourcemanager.ProjectsClient() @@ -35,7 +35,7 @@ def _get_gae_admins(): 'roles/appengine.appAdmin', )) - # collate all users who are members of at least one GAE admin role (_TARGETS) + # collate users who are members of at least one GAE admin role (_TARGETS) admins = set() # set of all App Engine admins allow_policy = rm_client.get_iam_policy(resource='projects/%s' % PROJ_ID) for b in allow_policy.bindings: # bindings in IAM allow policy From f22351f2a09dca41d869eb9593088256d8e2e625 Mon Sep 17 00:00:00 2001 From: wesley chun Date: Wed, 19 Oct 2022 17:36:43 -0700 Subject: [PATCH 47/56] add Mods 20 & 21 codelabs, minor README fixes --- README.md | 87 ++++++++++++++++++++++++++++++++++++++----------------- 1 file changed, 60 insertions(+), 27 deletions(-) diff --git a/README.md b/README.md index 778a7b7..edad757 100644 --- a/README.md +++ b/README.md @@ -8,7 +8,7 @@ This is the corresponding repository to the [Serverless Migration Station](https://bit.ly/3xk2Swi) video series whose goal is to help users on a Google Cloud serverless compute platform modernize to other Cloud or serverless products. Modernization steps generally feature a video, codelab (self-paced, hands-on tutorial), and code samples. The content initially focuses on App Engine and Google's earliest Cloud users. Read more about the [codelabs in this announcement](https://developers.googleblog.com/2021/03/modernizing-your-google-app-engine-applications.html?utm_source=ext&utm_medium=partner&utm_campaign=CDR_wes_aap-serverless_modernizegae_codelabsannounce_201031&utm_content=-) as well as [this one introducing the video series](https://developers.googleblog.com/2021/06/introducing-serverless-migration.html?utm_source=blog&utm_medium=partner&utm_campaign=CDR_wes_aap-serverless_smsintro_201023). This repo is for Python developers; there is another repo for Java developers. -[Google App Engine](https://cloud.google.com/appengine) (Standard) has undergone significant changes between the legacy and next generation platforms. To address this, we've created a set of resources showing developers how to perform individual migrations that can be applied to modernize their apps for the latest runtimes. The content falls into one of these topics: +[Google App Engine](https://cloud.google.com/appengine) (Standard) has undergone significant changes between the legacy and next generation platforms. To address this, we've created a set of resources showing developers how to perform individual migrations that can be applied to modernize their apps for the latest runtimes, meaning to Python 3 even though [Google expressed long-term support for legacy runtimes](https://cloud.google.com/appengine/docs/standard/long-term-support) like Python 2. The content falls into one of these topics: 1. Migrate from a legacy App Engine service to a similar Cloud product 1. Shift to another Cloud serverless compute platform (e.g., from App Engine to Cloud Run) @@ -39,12 +39,12 @@ Furthermore, deploying to GCP serverless platforms incur [minor build and storag ## Why -App Engine initially [elaunched in 2008](http://googleappengine.blogspot.com/2008/04/introducing-google-app-engine-our-new.html) ([video](http://youtu.be/3Ztr-HhWX1c)), providing a suite of bundled services making it convenient for developers to access a database (Datastore), caching (Memcache), independent task execution (TaskQueue), large "blob" storage (Blobstore) to allow for end-user file uploads or to serve large media files, and other companion services. However, apps leveraging those services can only run their apps on App Engine. +App Engine initially [launched in 2008](http://googleappengine.blogspot.com/2008/04/introducing-google-app-engine-our-new.html) ([video](http://youtu.be/3Ztr-HhWX1c)), providing a suite of bundled services making it convenient for developers to access a database (Datastore), caching (Memcache), independent task execution (Task Queue), large "blob" storage (Blobstore) to allow for end-user file uploads or to serve large media files, and other companion services. However, apps leveraging those services can only run their apps on App Engine. To increase app portability, its 2nd-generation service [launched in 2018](https://cloud.google.com/blog/products/gcp/introducing-app-engine-second-generation-runtimes-and-python-3-7), initially removing those legacy bundled services. The main reason to move to the 2nd generation service is that it allows developers to upgrade to the latest language runtimes, such as moving from Python 2 to 3 or Java 8 to 17. Unfortunately, it was mutually exclusive to do so, meaning while you could upgrade language releases, you lost access to those bundled services, making it a showstopper for many users. -However, due to their popularity _and_ to help users upgrade, the App Engine team restored access to many (but not all) of those services in Fall 2021. For more on this, see the [Legacy services](#accessing-legacy-services-in-second-generation) section below. As Google is continually striving to have the most [open cloud](https://cloud.google.com/open-cloud) on the market, and while many of those services are now available again, apps can _still_ be more portable if they migrated away from the legacy services to similar Cloud or 3rd-party oferings. Another issue with the bundled services is that they're only available in 2nd generation runtimes that have a 1st generation service (Python, Java, Go, PHP), excluding 2nd generation-only runtimes like Ruby and Node.js. +However, due to their popularity _and_ to help users upgrade, the App Engine team [restored access to many (but not all) of those services in Fall 2021](https://cloud.google.com/blog/products/serverless/support-for-app-engine-services-in-second-generation-runtimes). For more on this, see the [Legacy services](#accessing-legacy-services-in-second-generation) section below. As Google is continually striving to have the most [open cloud](https://cloud.google.com/open-cloud) on the market, and while many of those services are now available again, apps can _still_ be more portable if they migrated away from the legacy services to similar Cloud or 3rd-party oferings. Another issue with the bundled services is that they're only available in 2nd generation runtimes that have a 1st generation service (Python, Java, Go, PHP), excluding 2nd generation-only runtimes like Ruby and Node.js. Once apps have moved away from App Engine bundled services to similar Cloud or 3rd-party services. apps are portable enough to: @@ -117,8 +117,8 @@ Module | Topic | Video | Codelab | START here | FINISH here 17|Migrate to Python 3 bundled services| _TBD_ | [link](http://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-17-bundled?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrwormhole_sms_202002&utm_content=-) | Module 1 [code](/mod1-flask) (2.x) | Module 1 [code](/mod1b-flask) (3.x) 18|Add App Engine `taskqueue` pull tasks| _TBD_ | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-18-gaepull?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrgaepull_sms_202013&utm_content=-) | Module 1 [code](/mod1-flask) (2.x) | Module 18 [code](/mod18-gaepull) (2.x) 19|Migrate to Cloud Pub/Sub| _TBD_ | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-19-pubsub?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrpubsub_sms_202016&utm_content=-) | Module 18 [code](/mod18-gaepull) (2.x) | Module 19 [code](/mod19-pubsub) (2.x & 3.x) -20|Add App Engine `users` | _TBD_ | _TBD_ | Module 1 [code](/mod1-flask) (2.x) | Module 20 [code](/mod20-gaeusers) (2.x) -21|Migrate to Cloud Identity Platform | _TBD_ | _TBD_ | Module 20 [code](/mod20-gaeusers) (2.x) | Module 21 [code](/mod21a-idenplat) (2.x) & [code](/mod21b-idenplat) (3.x) +20|Add App Engine `users` | _TBD_ | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-20-gaeusers?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrgaeusers_sms_202119&utm_content=-)| Module 1 [code](/mod1-flask) (2.x) | Module 20 [code](/mod20-gaeusers) (2.x) +21|Migrate to Cloud Identity Platform | _TBD_ | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-21-idenplat?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgridenplat_sms_202119&utm_content=-)| Module 20 [code](/mod20-gaeusers) (2.x) | Module 21 [code](/mod21a-idenplat) (2.x) & [code](/mod21b-idenplat) (3.x) ### Table of contents @@ -135,7 +135,7 @@ If there is a logical codelab to do immediately after completing one, they will - START: [Module 0 code - Baseline](/mod0-baseline) - FINISH: [Module 1 code - Framework](/mod1-flask) - NEXT: - - Module 2 - migrate to Cloud NDB + - Module 2 - migrate from App Engine NDB to Cloud NDB (for Datastore access) - [Module 2 codelab](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-2-cloudndb?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrcloudndb_201021&utm_content=-): **Migrate from App Engine `ndb` to [Cloud NDB](https://googleapis.dev/python/python-ndb/latest)** - Python 2 @@ -144,56 +144,62 @@ If there is a logical codelab to do immediately after completing one, they will - Codelab bonus port to Python 3.x - FINISH: [Module 2 code - Cloud NDB](/mod2b-cloudndb) (3.x) - RECOMMENDED: - - Module 7 - add App Engine (push) tasks (and migrate to Cloud Tasks in Module 8) - - Module 12 - add App Engine `memcache` (and migrate to Cloud Memorystore in Module 13) - - Module 15 - add App Engine `blobstore` (and migrate to Cloud Storage in Module 16) + - Module 7 - add App Engine Task Queue push tasks (and migrate to Cloud Tasks in Module 8) + - Module 18 - add App Engine Task Queue pull tasks (and migrate to Cloud Pub/Sub in Module 19) + - Module 12 - add App Engine Memcache (and migrate to Cloud Memorystore in Module 13) + - Module 15 - add App Engine Blobstore (and migrate to Cloud Storage in Module 16) + - Module 20 - add App Engine Users (and migrate to Cloud Identity Platform in Module 21) -- [Module 7 codelab](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-7-gaetasks?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrgaetasks_sms_201028&utm_content=-): **Add App Engine (push) Task Queues to existing sample app** +- [Module 7 codelab](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-7-gaetasks?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrgaetasks_sms_201028&utm_content=-): **Add App Engine Task Queues push tasks to existing sample app** - **Not a migration**: add GAE Task Queues to prepare for migration to Cloud Tasks - Python 2 - START: [Module 1 code - Framework](/mod1-flask) - - FINISH: [Module 7 code - GAE TaskQueue (push tasks)](/mod7-gaetasks) - - NEXT: Module 8 - migrate App Engine push tasks to Cloud Tasks + - FINISH: [Module 7 code - GAE Task Queue push tasks](/mod7-gaetasks) + - NEXT: + - Module 8 - migrate App Engine Task Queue push tasks to Cloud Tasks -- [Module 8 codelab](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-8-cloudtasks?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrcloudtasks_sms_201112&utm_content=-): **Migrate from App Engine (push) Task Queues to [Cloud Tasks](http://cloud.google.com/tasks) v1** +- [Module 8 codelab](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-8-cloudtasks?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrcloudtasks_sms_201112&utm_content=-): **Migrate from App Engine Task Queues push tasks to [Cloud Tasks](http://cloud.google.com/tasks) v1** - Python 2 - - START: [Module 7 code - GAE TaskQueue (push tasks)](/mod7-gaetasks) + - START: [Module 7 code - GAE Task Queue push tasks](/mod7-gaetasks) - FINISH: [Module 8 code - Cloud Tasks](/mod8-cloudtasks) - RECOMMENDED: - Module 9 - migrate to Python 3 and Cloud Datastore - - Module 18 - add App Engine TaskQueue pull tasks (and migrate to Cloud Pub/Sub in Module 19) + - Module 18 - add App Engine Task Queue pull tasks (and migrate to Cloud Pub/Sub in Module 19) - Module 12 - add App Engine Memcache (and migrate to Cloud Memorystore in Module 13) - Module 15 - add App Engine Blobstore (and migrate to Cloud Storage in Module 16) + - Module 20 - add App Engine Users (and migrate to Cloud Identity Platform in Module 21) - [Module 9 codelab](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-9-py3dstasks?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrpy3fstasks_sms_201126&utm_content=-): **Migrate a Python 2 Cloud NDB & Cloud Tasks (v1) app to a Python 3 Cloud Datastore & Cloud Tasks (v2) app** - **Optional** migrations - - Migrating to Python 3 is not required but recommended as Python 2 has been sunset + - Migrating to Python 3 is not required but recommended as [Python 2 has been sunset](http://python.org/doc/sunset-python-2) - Migrating to Cloud Datastore is optional as Cloud NDB works on 3.x - Python 2 - START: [Module 8 code - Cloud Tasks](/mod8-cloudtasks) - Python 3 - FINISH: _TBD_ - RECOMMENDED: - - Module 18 - add App Engine TaskQueue pull tasks (and migrate to Cloud Pub/Sub in Module 19) + - Module 18 - add App Engine Task Queue pull tasks (and migrate to Cloud Pub/Sub in Module 19) - Module 12 - add App Engine Memcache (and migrate to Cloud Memorystore in Module 13) - Module 15 - add App Engine Blobstore (and migrate to Cloud Storage in Module 16) + - Module 20 - add App Engine Users (and migrate to Cloud Identity Platform in Module 21) -- [Module 18 codelab](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-18-gaepull?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrgaepull_sms_202013&utm_content=-): **Add App Engine (pull) Task Queues to existing sample app** +- [Module 18 codelab](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-18-gaepull?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrgaepull_sms_202013&utm_content=-): **Add App Engine Task Queues pull tasks to existing sample app** - **Not a migration**: add GAE Task Queues to prepare for migration to Cloud Tasks - Python 2 - START: [Module 1 code - Framework](/mod1-flask) - - FINISH: [Module 18 code - GAE TaskQueue (pull tasks)](/mod18-gaepull) + - FINISH: [Module 18 code - GAE Task Queue pull tasks](/mod18-gaepull) - NEXT: Module 19 - migrate App Engine pull tasks to Cloud Pub/Sub -- **Module 19 codelab** (TBD): **Migrate from App Engine (pull) Task Queues to [Cloud Pub/Sub](http://cloud.google.com/pubsub)** +- [Module 19 codelab](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-19-pubsub?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrpubsub_sms_202016&utm_content=-): **Migrate from App Engine Task Queues pull tasks to [Cloud Pub/Sub](http://cloud.google.com/pubsub)** - Python 2 - - START: [Module 18 code - GAE TaskQueue (pull tasks)](/mod18-gaepull) + - START: [Module 18 code - GAE Task Queue pull tasks](/mod18-gaepull) - Python 3 - FINISH: [Module 19 code - Cloud Pub/Sub](/mod19-pubsub) - RECOMMENDED: - - Module 7 - add App Engine TaskQueue push tasks (and migrate to Cloud Tasks in Module 8) + - Module 7 - add App Engine Task Queue push tasks (and migrate to Cloud Tasks in Module 8) - Module 12 - add App Engine Memcache (and migrate to Cloud Memorystore in Module 13) - Module 15 - add App Engine Blobstore (and migrate to Cloud Storage in Module 16) + - Module 20 - add App Engine `users` (and migrate to Cloud Identity Platform in Module 21) - [Module 12 codelab](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-12-memcache?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrmemcache_sms_202006&utm_content=-): **Add App Engine Memcache to existing sample app** - **Not a migration**: add GAE Memcache to prepare for migration to Cloud Memorystore @@ -209,15 +215,17 @@ If there is a logical codelab to do immediately after completing one, they will - Python 3 - FINISH: [Module 13 code - Cloud Tasks](/mod13b-memorystore) - RECOMMENDED: - - Module 7 - add App Engine (push) tasks (and migrate to Cloud Tasks in Module 8) + - Module 7 - add App Engine Task Queue push tasks (and migrate to Cloud Tasks in Module 8) + - Module 18 - add App Engine Task Queue pull tasks (and migrate to Cloud Pub/Sub in Module 19) - Module 15 - add App Engine Blobstore (and migrate to Cloud Storage in Module 16) + - Module 20 - add App Engine `users` (and migrate to Cloud Identity Platform in Module 21) - [Module 15 codelab](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-12-memcache?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrmemcache_sms_202006&utm_content=-): **Add App Engine Blobstore to existing sample app** - **Not a migration**: add GAE Blobstore to prepare for migration to Cloud Storage - Python 2 - START: [Module 0 code - Baseline](/mod0-baseline) - FINISH: [Module 15 code - GAE Blobstore](/mod15-blobstore) - - NEXT: Module 13 - migrate App Engine Blobstore to Cloud Storage + - NEXT: Module 16 - migrate App Engine Blobstore to Cloud Storage - [Module 16 codelab](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-13-memorystore?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrmemorystore_sms_202029&utm_content=-): **Migrate from App Engine Blobstore to [Cloud Storage (for Redis)](http://cloud.google.com/storage) v1** - Python 2 @@ -226,8 +234,30 @@ If there is a logical codelab to do immediately after completing one, they will - Python 3 - FINISH: [Module 16 code - Cloud Storage](/mod16-cloudstorage) (_same as Python 2 version_) - RECOMMENDED: - - Module 7 - add App Engine (push) tasks (and migrate to Cloud Tasks in Module 8) + - Module 7 - add App Engine Task Queue push tasks (and migrate to Cloud Tasks in Module 8) + - Module 18 - add App Engine Task Queue pull tasks (and migrate to Cloud Pub/Sub in Module 19) - Module 12 - add App Engine Memcache (and migrate to Cloud Memorystore in Module 13) + - Module 20 - add App Engine Users (and migrate to Cloud Identity Platform in Module 21) + +- [Module 20 codelab](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-20-gaeusers?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrgaeusers_sms_202119&utm_content=-): **Add App Engine Users to existing sample app** + - **Not a migration**: add GAE Users to prepare for migration to Cloud Identity Platform/Firebase Auth + - Python 2 + - START: [Module 1 code - Framework](/mod1-flask) + - FINISH: [Module 20 code - GAE Users](/mod20-gaeusers) + - NEXT: + - Module 21 - migrate App Engine Users to Cloud Identity Platform/Firebase Auth + +- [Module 21 codelab](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-21-idenplat?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgridenplat_sms_202119&utm_content=-): **Migrate from App Engine Users to [Cloud Identity Platform](http://cloud.google.com/identity-platform)/Firebase Auth** + - Python 2 + - START: [Module 20 code - GAE Users](/mod20-gaeusers) + - FINISH: [Module 21 code - Cloud Identity Platform](/mod21a-idenplat)/Firebase Auth + - Python 3 + - FINISH: [Module 21 code - Cloud Identity Platform](/mod21b-idenplat)/Firebase Auth + - RECOMMENDED: + - Module 7 - add App Engine Task Queue push tasks (and migrate to Cloud Tasks in Module 8) + - Module 18 - add App Engine Task Queue pull tasks (and migrate to Cloud Pub/Sub in Module 19) + - Module 12 - add App Engine Memcache (and migrate to Cloud Memorystore in Module 13) + - Module 15 - add App Engine Blobstore (and migrate to Cloud Storage in Module 16) - [Module 3 codelab](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-3-datastore?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrcloudds_201003&utm_content=-): **Migrate from Cloud NDB to [Cloud Datastore](http://cloud.google.com/datastore)** @@ -241,9 +271,11 @@ If there is a logical codelab to do immediately after completing one, they will - START: [Module 2 code - Cloud NDB](/mod2b-cloudndb) - FINISH: [Module 3 code - Cloud Datastore](/mod3b-datastore) - RECOMMENDED: - - Module 7 - add App Engine (push) tasks (and migrate to Cloud Tasks in Module 8) + - Module 7 - add App Engine Task Queue push tasks (and migrate to Cloud Tasks in Module 8) + - Module 18 - add App Engine Task Queue pull tasks (and migrate to Cloud Pub/Sub in Module 19) - Module 12 - add App Engine Memcache (and migrate to Cloud Memorystore in Module 13) - Module 15 - add App Engine Blobstore (and migrate to Cloud Storage in Module 16) + - Module 20 - add App Engine Users (and migrate to Cloud Identity Platform in Module 21) #### Migrations to other Cloud serverless platforms @@ -279,7 +311,8 @@ If there is a logical codelab to do immediately after completing one, they will - START: [Module 2 code - Cloud NDB](/mod2b-cloudndb) - FINISH: [Module 11 code - Cloud Functions](/mod11-functions) - RECOMMENDED: - - Module 7 - add App Engine (push) tasks (and migrate to Cloud Tasks in Module 8) + - Module 7 - add App Engine Task Queue push tasks (and migrate to Cloud Tasks in Module 8) + - Module 18 - add App Engine Task Queue pull tasks (and migrate to Cloud Pub/Sub in Module 19) - Module 12 - add App Engine `memcache` (and migrate to Cloud Memorystore in Module 13) - Module 15 - add App Engine `blobstore` (and migrate to Cloud Storage in Module 16) - OTHER OPTIONS (in somewhat priority order): From 37312b17a05fcbe515ba227a488eba75104182b5 Mon Sep 17 00:00:00 2001 From: wesley chun Date: Mon, 5 Dec 2022 18:11:02 -0800 Subject: [PATCH 48/56] add new videos; minor code tweaks --- README.md | 6 +++--- mod18-gaepull/main.py | 2 +- mod19-pubsub/main.py | 2 +- mod21a-idenplat/templates/index.html | 2 +- mod21b-idenplat/templates/index.html | 2 +- 5 files changed, 7 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index edad757..2b4e724 100644 --- a/README.md +++ b/README.md @@ -97,7 +97,7 @@ The table below summarizes migration module resources currently available along Module | Topic | Video | Codelab | START here | FINISH here --- | --- | --- | --- | --- | --- -0|Baseline app| [link](http://twitter.com/googledevs/status/1407755281786867714?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_smsintro_201023&utm_content=-) | _N/A_ (no tutorial; just review the code) | _N/A_ | Module 0 [code](/mod0-baseline) (2.x) +0|Baseline app| [link](https://twitter.com/googledevs/status/1407755281786867714?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_smsintro_201023&utm_content=-) | _N/A_ (no tutorial; just review the code) | _N/A_ | Module 0 [code](/mod0-baseline) (2.x) 1|Migrate to Flask| [link](https://twitter.com/googledevs/status/1413273119071031299?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrwa2flsk_201008&u%20tm_content=-) | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-1-flask?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrwa2flsk_201008&utm_content=-) | Module 0 [code](/mod0-baseline) (2.x) | Module 1 [code](/mod1-flask) (2.x) (and [code](/mod1b-flask) (3.x)) 2|Migrate to Cloud NDB| [link](https://twitter.com/googledevs/status/1417892493383802883?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrcloudndb_201021&utm_content=-) | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-2-cloudndb?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrcloudndb_201021&utm_content=-) | Module 1 [code](/mod1-flask) (2.x) | Module 2 [code](/mod2a-cloudndb) (2.x) & [code](/mod2b-cloudndb) (3.x) 3|Migrate to Cloud Datastore| [link](http://twitter.com/googledevs/status/1422966928910393347?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrcloudds_201003&utm_content=-) | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-3-datastore?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrcloudds_201003&utm_content=-) | Module 2 [code](/mod2a-cloudndb) (2.x) & [code](/mod2b-cloudndb) (3.x) | Module 3 [code](/mod3a-datastore) (2.x) & [code](/mod3b-datastore) (3.x) @@ -114,8 +114,8 @@ Module | Topic | Video | Codelab | START here | FINISH here 14|Migrate service between projects| _TBD_ | _TBD_ | _TBD_ | _TBD_ 15|Add App Engine `blobstore`| [link](https://twitter.com/googledevs/status/1552384740052934657?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrblobstore_sms_202029&utm_content=-) | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-15-blobstore?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrblobstore_sms_202029&utm_content=-) | Module 0 [code](/mod0-baseline) (2.x) | Module 15 [code](/mod15-blobstore) (2.x) 16|Migrate to Cloud Storage| [link](https://twitter.com/googledevs/status/1559285905961123845?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrcloudstorage_sms_202029&utm_content=-) | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-16-cloudstorage?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrcloudstorage_sms_202029&utm_content=-) | Module 15 [code](/mod15-blobstore) (2.x) | Module 16 [code](/mod16-cloudstorage) (2.x & 3.x) -17|Migrate to Python 3 bundled services| _TBD_ | [link](http://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-17-bundled?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrwormhole_sms_202002&utm_content=-) | Module 1 [code](/mod1-flask) (2.x) | Module 1 [code](/mod1b-flask) (3.x) -18|Add App Engine `taskqueue` pull tasks| _TBD_ | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-18-gaepull?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrgaepull_sms_202013&utm_content=-) | Module 1 [code](/mod1-flask) (2.x) | Module 18 [code](/mod18-gaepull) (2.x) +17|Migrate to Python 3 bundled services| [link](https://twitter.com/googledevs/status/1587855923254624259?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrwormhole_sms_202002&utm_content=-) | [link](http://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-17-bundled?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrwormhole_sms_202002&utm_content=-) | Module 1 [code](/mod1-flask) (2.x) | Module 1 [code](/mod1b-flask) (3.x) +18|Add App Engine `taskqueue` pull tasks| [link](https://twitter.com/googledevs/status/1598013491381665808?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrgaepull_sms_202013&utm_content=-) | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-18-gaepull?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrgaepull_sms_202013&utm_content=-) | Module 1 [code](/mod1-flask) (2.x) | Module 18 [code](/mod18-gaepull) (2.x) 19|Migrate to Cloud Pub/Sub| _TBD_ | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-19-pubsub?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrpubsub_sms_202016&utm_content=-) | Module 18 [code](/mod18-gaepull) (2.x) | Module 19 [code](/mod19-pubsub) (2.x & 3.x) 20|Add App Engine `users` | _TBD_ | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-20-gaeusers?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrgaeusers_sms_202119&utm_content=-)| Module 1 [code](/mod1-flask) (2.x) | Module 20 [code](/mod20-gaeusers) (2.x) 21|Migrate to Cloud Identity Platform | _TBD_ | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-21-idenplat?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgridenplat_sms_202119&utm_content=-)| Module 20 [code](/mod20-gaeusers) (2.x) | Module 21 [code](/mod21a-idenplat) (2.x) & [code](/mod21b-idenplat) (3.x) diff --git a/mod18-gaepull/main.py b/mod18-gaepull/main.py index a10eba2..ce0e1b8 100644 --- a/mod18-gaepull/main.py +++ b/mod18-gaepull/main.py @@ -17,8 +17,8 @@ from google.appengine.ext import ndb HOUR = 3600 -LIMIT = 10 TASKS = 1000 +LIMIT = 10 QNAME = 'pullq' QUEUE = taskqueue.Queue(QNAME) app = Flask(__name__) diff --git a/mod19-pubsub/main.py b/mod19-pubsub/main.py index 276b433..884b3e5 100644 --- a/mod19-pubsub/main.py +++ b/mod19-pubsub/main.py @@ -16,8 +16,8 @@ import google.auth from google.cloud import ndb, pubsub -LIMIT = 10 TASKS = 1000 +LIMIT = 10 TOPIC = 'pullq' SBSCR = 'worker' diff --git a/mod21a-idenplat/templates/index.html b/mod21a-idenplat/templates/index.html index 0d1b97b..2b7696c 100644 --- a/mod21a-idenplat/templates/index.html +++ b/mod21a-idenplat/templates/index.html @@ -69,7 +69,7 @@

-Welcome, (admin) +Welcome, ! (admin)


diff --git a/mod21b-idenplat/templates/index.html b/mod21b-idenplat/templates/index.html index 0d1b97b..2b7696c 100644 --- a/mod21b-idenplat/templates/index.html +++ b/mod21b-idenplat/templates/index.html @@ -69,7 +69,7 @@

-Welcome, (admin) +Welcome, ! (admin)


From 37a1ca1d61b7fb448c0dfd7457ccef03bb54396e Mon Sep 17 00:00:00 2001 From: wesley chun Date: Sat, 10 Dec 2022 03:56:33 -0800 Subject: [PATCH 49/56] add Module 22 code --- README.md | 3 +- mod22-bundled/README.md | 9 +++ mod22-bundled/blobstore2/.gcloudignore | 79 +++++++++++++++++++++++ mod22-bundled/blobstore2/README.md | 3 + mod22-bundled/blobstore2/app.yaml | 21 ++++++ mod22-bundled/blobstore2/main.py | 57 ++++++++++++++++ mod22-bundled/blobstore3/.gcloudignore | 79 +++++++++++++++++++++++ mod22-bundled/blobstore3/README.md | 3 + mod22-bundled/blobstore3/app.yaml | 19 ++++++ mod22-bundled/blobstore3/main.py | 66 +++++++++++++++++++ mod22-bundled/blobstore3/requirements.txt | 2 + mod22-bundled/deferred2/.gcloudignore | 79 +++++++++++++++++++++++ mod22-bundled/deferred2/README.md | 3 + mod22-bundled/deferred2/app.yaml | 24 +++++++ mod22-bundled/deferred2/main.py | 43 ++++++++++++ mod22-bundled/deferred3/.gcloudignore | 79 +++++++++++++++++++++++ mod22-bundled/deferred3/README.md | 3 + mod22-bundled/deferred3/app.yaml | 19 ++++++ mod22-bundled/deferred3/main.py | 42 ++++++++++++ mod22-bundled/deferred3/requirements.txt | 2 + mod22-bundled/mail2/.gcloudignore | 79 +++++++++++++++++++++++ mod22-bundled/mail2/README.md | 3 + mod22-bundled/mail2/app.yaml | 24 +++++++ mod22-bundled/mail2/main.py | 76 ++++++++++++++++++++++ mod22-bundled/mail3/.gcloudignore | 79 +++++++++++++++++++++++ mod22-bundled/mail3/app.yaml | 22 +++++++ mod22-bundled/mail3/main.py | 74 +++++++++++++++++++++ mod22-bundled/mail3/requirements.txt | 2 + 28 files changed, 993 insertions(+), 1 deletion(-) create mode 100644 mod22-bundled/README.md create mode 100644 mod22-bundled/blobstore2/.gcloudignore create mode 100644 mod22-bundled/blobstore2/README.md create mode 100644 mod22-bundled/blobstore2/app.yaml create mode 100644 mod22-bundled/blobstore2/main.py create mode 100644 mod22-bundled/blobstore3/.gcloudignore create mode 100644 mod22-bundled/blobstore3/README.md create mode 100644 mod22-bundled/blobstore3/app.yaml create mode 100644 mod22-bundled/blobstore3/main.py create mode 100644 mod22-bundled/blobstore3/requirements.txt create mode 100644 mod22-bundled/deferred2/.gcloudignore create mode 100644 mod22-bundled/deferred2/README.md create mode 100644 mod22-bundled/deferred2/app.yaml create mode 100644 mod22-bundled/deferred2/main.py create mode 100644 mod22-bundled/deferred3/.gcloudignore create mode 100644 mod22-bundled/deferred3/README.md create mode 100644 mod22-bundled/deferred3/app.yaml create mode 100644 mod22-bundled/deferred3/main.py create mode 100644 mod22-bundled/deferred3/requirements.txt create mode 100644 mod22-bundled/mail2/.gcloudignore create mode 100644 mod22-bundled/mail2/README.md create mode 100644 mod22-bundled/mail2/app.yaml create mode 100644 mod22-bundled/mail2/main.py create mode 100644 mod22-bundled/mail3/.gcloudignore create mode 100644 mod22-bundled/mail3/app.yaml create mode 100644 mod22-bundled/mail3/main.py create mode 100644 mod22-bundled/mail3/requirements.txt diff --git a/README.md b/README.md index 2b4e724..dd7c171 100644 --- a/README.md +++ b/README.md @@ -114,11 +114,12 @@ Module | Topic | Video | Codelab | START here | FINISH here 14|Migrate service between projects| _TBD_ | _TBD_ | _TBD_ | _TBD_ 15|Add App Engine `blobstore`| [link](https://twitter.com/googledevs/status/1552384740052934657?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrblobstore_sms_202029&utm_content=-) | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-15-blobstore?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrblobstore_sms_202029&utm_content=-) | Module 0 [code](/mod0-baseline) (2.x) | Module 15 [code](/mod15-blobstore) (2.x) 16|Migrate to Cloud Storage| [link](https://twitter.com/googledevs/status/1559285905961123845?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrcloudstorage_sms_202029&utm_content=-) | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-16-cloudstorage?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrcloudstorage_sms_202029&utm_content=-) | Module 15 [code](/mod15-blobstore) (2.x) | Module 16 [code](/mod16-cloudstorage) (2.x & 3.x) -17|Migrate to Python 3 bundled services| [link](https://twitter.com/googledevs/status/1587855923254624259?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrwormhole_sms_202002&utm_content=-) | [link](http://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-17-bundled?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrwormhole_sms_202002&utm_content=-) | Module 1 [code](/mod1-flask) (2.x) | Module 1 [code](/mod1b-flask) (3.x) +17|Migrate to Python 3 bundled services (Part 1)| [link](https://twitter.com/googledevs/status/1587855923254624259?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrwormhole_sms_202002&utm_content=-) | [link](http://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-17-bundled?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrwormhole_sms_202002&utm_content=-) | Module 1 [code](/mod1-flask) (2.x) | Module 1 [code](/mod1b-flask) (3.x) 18|Add App Engine `taskqueue` pull tasks| [link](https://twitter.com/googledevs/status/1598013491381665808?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrgaepull_sms_202013&utm_content=-) | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-18-gaepull?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrgaepull_sms_202013&utm_content=-) | Module 1 [code](/mod1-flask) (2.x) | Module 18 [code](/mod18-gaepull) (2.x) 19|Migrate to Cloud Pub/Sub| _TBD_ | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-19-pubsub?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrpubsub_sms_202016&utm_content=-) | Module 18 [code](/mod18-gaepull) (2.x) | Module 19 [code](/mod19-pubsub) (2.x & 3.x) 20|Add App Engine `users` | _TBD_ | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-20-gaeusers?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrgaeusers_sms_202119&utm_content=-)| Module 1 [code](/mod1-flask) (2.x) | Module 20 [code](/mod20-gaeusers) (2.x) 21|Migrate to Cloud Identity Platform | _TBD_ | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-21-idenplat?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgridenplat_sms_202119&utm_content=-)| Module 20 [code](/mod20-gaeusers) (2.x) | Module 21 [code](/mod21a-idenplat) (2.x) & [code](/mod21b-idenplat) (3.x) +22|Migrate to Python 3 bundled services (Part 2)| _TBD_ | _N/A_ | Module 22 [code](/mod22-bundled) (2.x & 3.x) | _(same)_ ### Table of contents diff --git a/mod22-bundled/README.md b/mod22-bundled/README.md new file mode 100644 index 0000000..0526cbb --- /dev/null +++ b/mod22-bundled/README.md @@ -0,0 +1,9 @@ +# Module 22 - Extending support for App Engine bundled services in Python 3 + +There are six folders in this directory that contain full applications using the Blobstore, Deferred, and Mail bundled services. A Python 2 version using the `webapp2` framework is provided along with a modernized Python 3 equivalent using the Flask web framework and the Python 3 App Engine SDK. + +Bundled service | Python 2 `webapp` | Python 3 Flask +--- | --- | --- +Blobstore | [code](blobstore2) | [code](blobstore3) +Deferred | [code](deferred2) | [code](deferred3) +Mail | [code](mail2) | [code](mail3) diff --git a/mod22-bundled/blobstore2/.gcloudignore b/mod22-bundled/blobstore2/.gcloudignore new file mode 100644 index 0000000..af73b5d --- /dev/null +++ b/mod22-bundled/blobstore2/.gcloudignore @@ -0,0 +1,79 @@ +# This file specifies files that are *not* uploaded to Google Cloud Platform +# using gcloud. It follows the same syntax as .gitignore, with the addition of +# "#!include" directives (which insert the entries of the given .gitignore-style +# file at that point). +# +# For more information, run: +# $ gcloud topic gcloudignore +# +.gcloudignore + +# Source code control files +.git/ +.gitignore +.hgignore +.hg/ + +# README/text files +LICENSE +*.md + +# Tests/results (not in .gitignore) +noxfile.py +pylintrc +pylintrc.test + +# most of .gitignore (except `lib`) +# +# Python +*.py[cod] +__pycache__/ +/setup.cfg + +# C extensions +*.so + +# Packages +*.egg +*.egg-info +dist +build +eggs +.eggs +parts +bin +var +sdist +develop-eggs +.installed.cfg +lib64 +*.tgz + +# Installer logs +pip-log.txt + +# Tests/results +.nox/ +.pytest_cache/ +.cache +.pytype +.coverage +coverage.xml +*sponge_log.xml +system_tests/local_test_setup + +# Mac +.DS_Store + +# IDEs/editors +*.sw[op] +*~ +.vscode +.idea + +# Built documentation +docs/_build +docs.metadata + +# Virtual environment +env/ diff --git a/mod22-bundled/blobstore2/README.md b/mod22-bundled/blobstore2/README.md new file mode 100644 index 0000000..e093a2c --- /dev/null +++ b/mod22-bundled/blobstore2/README.md @@ -0,0 +1,3 @@ +# Module 22 - Using Blobstore bundled service (Python 2) + +This repo folder represents the Module 22 Python 2 sample app for the Blobstore bundled service. diff --git a/mod22-bundled/blobstore2/app.yaml b/mod22-bundled/blobstore2/app.yaml new file mode 100644 index 0000000..4b7d197 --- /dev/null +++ b/mod22-bundled/blobstore2/app.yaml @@ -0,0 +1,21 @@ +# Copyright 2022 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +runtime: python27 +threadsafe: yes +api_version: 1 + +handlers: +- url: /.* + script: main.app diff --git a/mod22-bundled/blobstore2/main.py b/mod22-bundled/blobstore2/main.py new file mode 100644 index 0000000..483318f --- /dev/null +++ b/mod22-bundled/blobstore2/main.py @@ -0,0 +1,57 @@ +# Copyright 2022 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import webapp2 +from google.appengine.ext import blobstore, ndb +from google.appengine.ext.webapp import blobstore_handlers + +UPLOAD_FORM = '''\ +

Upload photo:

+
+

+
''' + + +class PhotoUpload(ndb.Model): + 'PhotoUpload entity for registering a photo' + blob_key = ndb.BlobKeyProperty() + + +class PhotoUploadHandler(blobstore_handlers.BlobstoreUploadHandler): + 'PhotoUploadHandler handles a photo upload (POST)' + def post(self): + uploads = self.get_uploads() + blob_id = uploads[0].key() if uploads else None + PhotoUpload(blob_key=blob_id).put() + self.redirect('/view_photo/%s' % blob_id) + + +class ViewPhotoHandler(blobstore_handlers.BlobstoreDownloadHandler): + 'ViewPhotoHandler handles a photo view/download (GET)' + def get(self, blob_key): + self.send_blob(blob_key) if blobstore.get(blob_key) else self.error(404) + + +class MainHandler(webapp2.RequestHandler): + 'main application (GET) handler' + def get(self): + self.response.write( + UPLOAD_FORM % blobstore.create_upload_url('https://github.com/upload_photo')) + + +app = webapp2.WSGIApplication([ + ('/', MainHandler), + ('/upload_photo', PhotoUploadHandler), + ('/view_photo/([^/]+)?', ViewPhotoHandler), +], debug=True) diff --git a/mod22-bundled/blobstore3/.gcloudignore b/mod22-bundled/blobstore3/.gcloudignore new file mode 100644 index 0000000..af73b5d --- /dev/null +++ b/mod22-bundled/blobstore3/.gcloudignore @@ -0,0 +1,79 @@ +# This file specifies files that are *not* uploaded to Google Cloud Platform +# using gcloud. It follows the same syntax as .gitignore, with the addition of +# "#!include" directives (which insert the entries of the given .gitignore-style +# file at that point). +# +# For more information, run: +# $ gcloud topic gcloudignore +# +.gcloudignore + +# Source code control files +.git/ +.gitignore +.hgignore +.hg/ + +# README/text files +LICENSE +*.md + +# Tests/results (not in .gitignore) +noxfile.py +pylintrc +pylintrc.test + +# most of .gitignore (except `lib`) +# +# Python +*.py[cod] +__pycache__/ +/setup.cfg + +# C extensions +*.so + +# Packages +*.egg +*.egg-info +dist +build +eggs +.eggs +parts +bin +var +sdist +develop-eggs +.installed.cfg +lib64 +*.tgz + +# Installer logs +pip-log.txt + +# Tests/results +.nox/ +.pytest_cache/ +.cache +.pytype +.coverage +coverage.xml +*sponge_log.xml +system_tests/local_test_setup + +# Mac +.DS_Store + +# IDEs/editors +*.sw[op] +*~ +.vscode +.idea + +# Built documentation +docs/_build +docs.metadata + +# Virtual environment +env/ diff --git a/mod22-bundled/blobstore3/README.md b/mod22-bundled/blobstore3/README.md new file mode 100644 index 0000000..e866055 --- /dev/null +++ b/mod22-bundled/blobstore3/README.md @@ -0,0 +1,3 @@ +# Module 22 - Using Blobstore bundled service (Python 3) + +This repo folder represents the Module 22 Python 3 sample app for the Blobstore bundled service. diff --git a/mod22-bundled/blobstore3/app.yaml b/mod22-bundled/blobstore3/app.yaml new file mode 100644 index 0000000..bfda3ee --- /dev/null +++ b/mod22-bundled/blobstore3/app.yaml @@ -0,0 +1,19 @@ +# Copyright 2022 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +runtime: python310 +app_engine_apis: true + +env_variables: + NDB_USE_CROSS_COMPATIBLE_PICKLE_PROTOCOL: 'True' diff --git a/mod22-bundled/blobstore3/main.py b/mod22-bundled/blobstore3/main.py new file mode 100644 index 0000000..e56c60a --- /dev/null +++ b/mod22-bundled/blobstore3/main.py @@ -0,0 +1,66 @@ +# Copyright 2022 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from flask import Flask, abort, redirect, request +from google.appengine.api import wrap_wsgi_app +from google.appengine.ext import blobstore, ndb + +UPLOAD_FORM = '''\ +

Upload photo:

+
+

+
''' + +app = Flask(__name__) +app.wsgi_app = wrap_wsgi_app(app.wsgi_app) + + +class PhotoUpload(ndb.Model): + 'PhotoUpload entity for registering a photo' + blob_key = ndb.BlobKeyProperty() + + +class PhotoUploadHandler(blobstore.BlobstoreUploadHandler): + 'PhotoUploadHandler handles a photo upload (POST)' + def post(self): + uploads = self.get_uploads(request.environ) + blob_id = uploads[0].key() if uploads else None + PhotoUpload(blob_key=blob_id).put() + return redirect('/view_photo/%s' % blob_id) + +@app.route('/upload_photo', methods=['POST']) +def upload_photo(): + 'call upload handler for upload (POST) request' + return PhotoUploadHandler().post() + + +class ViewPhotoHandler(blobstore.BlobstoreDownloadHandler): + 'ViewPhotoHandler handles a photo view/download (GET)' + def get(self, blob_key): + if blobstore.get(blob_key): + headers = self.send_blob(request.environ, blob_key) + headers['Content-Type'] = None + return '', headers + abort(404) + +@app.route('/view_photo/') +def view_photo(photo_key): + 'call download handler for view (GET) request' + return ViewPhotoHandler().get(photo_key) + + +@app.route('/') +def upload_form(): + 'display photo upload HTML form' + return UPLOAD_FORM % blobstore.create_upload_url('https://github.com/upload_photo') diff --git a/mod22-bundled/blobstore3/requirements.txt b/mod22-bundled/blobstore3/requirements.txt new file mode 100644 index 0000000..a2e7c7f --- /dev/null +++ b/mod22-bundled/blobstore3/requirements.txt @@ -0,0 +1,2 @@ +flask +appengine-python-standard diff --git a/mod22-bundled/deferred2/.gcloudignore b/mod22-bundled/deferred2/.gcloudignore new file mode 100644 index 0000000..af73b5d --- /dev/null +++ b/mod22-bundled/deferred2/.gcloudignore @@ -0,0 +1,79 @@ +# This file specifies files that are *not* uploaded to Google Cloud Platform +# using gcloud. It follows the same syntax as .gitignore, with the addition of +# "#!include" directives (which insert the entries of the given .gitignore-style +# file at that point). +# +# For more information, run: +# $ gcloud topic gcloudignore +# +.gcloudignore + +# Source code control files +.git/ +.gitignore +.hgignore +.hg/ + +# README/text files +LICENSE +*.md + +# Tests/results (not in .gitignore) +noxfile.py +pylintrc +pylintrc.test + +# most of .gitignore (except `lib`) +# +# Python +*.py[cod] +__pycache__/ +/setup.cfg + +# C extensions +*.so + +# Packages +*.egg +*.egg-info +dist +build +eggs +.eggs +parts +bin +var +sdist +develop-eggs +.installed.cfg +lib64 +*.tgz + +# Installer logs +pip-log.txt + +# Tests/results +.nox/ +.pytest_cache/ +.cache +.pytype +.coverage +coverage.xml +*sponge_log.xml +system_tests/local_test_setup + +# Mac +.DS_Store + +# IDEs/editors +*.sw[op] +*~ +.vscode +.idea + +# Built documentation +docs/_build +docs.metadata + +# Virtual environment +env/ diff --git a/mod22-bundled/deferred2/README.md b/mod22-bundled/deferred2/README.md new file mode 100644 index 0000000..5945d3e --- /dev/null +++ b/mod22-bundled/deferred2/README.md @@ -0,0 +1,3 @@ +# Module 22 - Using Deferred bundled service (Python 2) + +This repo folder represents the Module 22 Python 2 sample app for the Deferred bundled service. diff --git a/mod22-bundled/deferred2/app.yaml b/mod22-bundled/deferred2/app.yaml new file mode 100644 index 0000000..05059b1 --- /dev/null +++ b/mod22-bundled/deferred2/app.yaml @@ -0,0 +1,24 @@ +# Copyright 2022 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +runtime: python27 +threadsafe: yes +api_version: 1 + +builtins: +- deferred: on + +handlers: +- url: /.* + script: main.app diff --git a/mod22-bundled/deferred2/main.py b/mod22-bundled/deferred2/main.py new file mode 100644 index 0000000..27b2229 --- /dev/null +++ b/mod22-bundled/deferred2/main.py @@ -0,0 +1,43 @@ +# Copyright 2022 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import webapp2 +from google.appengine.ext import deferred, ndb + +KEY_NAME = 'SECRET' + +class Counter(ndb.Model): + 'Counter entity: autoincrement integer' + count = ndb.IntegerProperty(indexed=False) + + +def bump_counter_later(key): + 'bump counter in (push) task' + entity = Counter.get_or_insert(key, count=0) + entity.count += 1 + entity.put() + + +class MainHandler(webapp2.RequestHandler): + def get(self): + 'main application (GET) handler' + entity = Counter.get_by_id(KEY_NAME) + self.response.write('Counter at %d... bump requested.' % ( + entity.count if entity else 0)) + deferred.defer(bump_counter_later, KEY_NAME) + + +app = webapp2.WSGIApplication([ + ('/', MainHandler), +], debug=True) diff --git a/mod22-bundled/deferred3/.gcloudignore b/mod22-bundled/deferred3/.gcloudignore new file mode 100644 index 0000000..af73b5d --- /dev/null +++ b/mod22-bundled/deferred3/.gcloudignore @@ -0,0 +1,79 @@ +# This file specifies files that are *not* uploaded to Google Cloud Platform +# using gcloud. It follows the same syntax as .gitignore, with the addition of +# "#!include" directives (which insert the entries of the given .gitignore-style +# file at that point). +# +# For more information, run: +# $ gcloud topic gcloudignore +# +.gcloudignore + +# Source code control files +.git/ +.gitignore +.hgignore +.hg/ + +# README/text files +LICENSE +*.md + +# Tests/results (not in .gitignore) +noxfile.py +pylintrc +pylintrc.test + +# most of .gitignore (except `lib`) +# +# Python +*.py[cod] +__pycache__/ +/setup.cfg + +# C extensions +*.so + +# Packages +*.egg +*.egg-info +dist +build +eggs +.eggs +parts +bin +var +sdist +develop-eggs +.installed.cfg +lib64 +*.tgz + +# Installer logs +pip-log.txt + +# Tests/results +.nox/ +.pytest_cache/ +.cache +.pytype +.coverage +coverage.xml +*sponge_log.xml +system_tests/local_test_setup + +# Mac +.DS_Store + +# IDEs/editors +*.sw[op] +*~ +.vscode +.idea + +# Built documentation +docs/_build +docs.metadata + +# Virtual environment +env/ diff --git a/mod22-bundled/deferred3/README.md b/mod22-bundled/deferred3/README.md new file mode 100644 index 0000000..99036e8 --- /dev/null +++ b/mod22-bundled/deferred3/README.md @@ -0,0 +1,3 @@ +# Module 22 - Using Deferred bundled service (Python 3) + +This repo folder represents the Module 22 Python 3 sample app for the Deferred bundled service. diff --git a/mod22-bundled/deferred3/app.yaml b/mod22-bundled/deferred3/app.yaml new file mode 100644 index 0000000..bfda3ee --- /dev/null +++ b/mod22-bundled/deferred3/app.yaml @@ -0,0 +1,19 @@ +# Copyright 2022 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +runtime: python310 +app_engine_apis: true + +env_variables: + NDB_USE_CROSS_COMPATIBLE_PICKLE_PROTOCOL: 'True' diff --git a/mod22-bundled/deferred3/main.py b/mod22-bundled/deferred3/main.py new file mode 100644 index 0000000..26062dc --- /dev/null +++ b/mod22-bundled/deferred3/main.py @@ -0,0 +1,42 @@ +# Copyright 2022 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from flask import Flask +from google.appengine.api import wrap_wsgi_app +from google.appengine.ext import deferred, ndb + +KEY_NAME = 'SECRET' +app = Flask(__name__) +app.wsgi_app = wrap_wsgi_app(app.wsgi_app, use_deferred=True) + +class Counter(ndb.Model): + 'Counter entity: autoincrement integer' + count = ndb.IntegerProperty(indexed=False) + + +def bump_counter_later(key): + 'bump counter in (push) task' + entity = Counter.get_or_insert(key, count=0) + entity.count += 1 + entity.put() + + +@app.route('/') +def root(): + 'main application (GET) handler' + entity = Counter.get_by_id(KEY_NAME) + output = 'Counter at %d... bump requested.' % ( + entity.count if entity else 0) + deferred.defer(bump_counter_later, KEY_NAME) + return output diff --git a/mod22-bundled/deferred3/requirements.txt b/mod22-bundled/deferred3/requirements.txt new file mode 100644 index 0000000..a2e7c7f --- /dev/null +++ b/mod22-bundled/deferred3/requirements.txt @@ -0,0 +1,2 @@ +flask +appengine-python-standard diff --git a/mod22-bundled/mail2/.gcloudignore b/mod22-bundled/mail2/.gcloudignore new file mode 100644 index 0000000..af73b5d --- /dev/null +++ b/mod22-bundled/mail2/.gcloudignore @@ -0,0 +1,79 @@ +# This file specifies files that are *not* uploaded to Google Cloud Platform +# using gcloud. It follows the same syntax as .gitignore, with the addition of +# "#!include" directives (which insert the entries of the given .gitignore-style +# file at that point). +# +# For more information, run: +# $ gcloud topic gcloudignore +# +.gcloudignore + +# Source code control files +.git/ +.gitignore +.hgignore +.hg/ + +# README/text files +LICENSE +*.md + +# Tests/results (not in .gitignore) +noxfile.py +pylintrc +pylintrc.test + +# most of .gitignore (except `lib`) +# +# Python +*.py[cod] +__pycache__/ +/setup.cfg + +# C extensions +*.so + +# Packages +*.egg +*.egg-info +dist +build +eggs +.eggs +parts +bin +var +sdist +develop-eggs +.installed.cfg +lib64 +*.tgz + +# Installer logs +pip-log.txt + +# Tests/results +.nox/ +.pytest_cache/ +.cache +.pytype +.coverage +coverage.xml +*sponge_log.xml +system_tests/local_test_setup + +# Mac +.DS_Store + +# IDEs/editors +*.sw[op] +*~ +.vscode +.idea + +# Built documentation +docs/_build +docs.metadata + +# Virtual environment +env/ diff --git a/mod22-bundled/mail2/README.md b/mod22-bundled/mail2/README.md new file mode 100644 index 0000000..0d03f68 --- /dev/null +++ b/mod22-bundled/mail2/README.md @@ -0,0 +1,3 @@ +# Module 22 - Using Mail bundled service (Python 2) + +This repo folder represents the Module 22 Python 2 sample app for the Mail bundled service. diff --git a/mod22-bundled/mail2/app.yaml b/mod22-bundled/mail2/app.yaml new file mode 100644 index 0000000..26a9d41 --- /dev/null +++ b/mod22-bundled/mail2/app.yaml @@ -0,0 +1,24 @@ +# Copyright 2022 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +runtime: python27 +threadsafe: yes +api_version: 1 + +inbound_services: +- mail + +handlers: +- url: /.* + script: main.app diff --git a/mod22-bundled/mail2/main.py b/mod22-bundled/mail2/main.py new file mode 100644 index 0000000..f63b6b8 --- /dev/null +++ b/mod22-bundled/mail2/main.py @@ -0,0 +1,76 @@ +# Copyright 2022 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from cgi import escape +import webapp2 +from google.appengine.ext import ndb +from google.appengine.ext.webapp import mail_handlers + +KEY_NAME = 'SECRET' +FIELDS = frozenset(('sender', 'subject', 'date')) +MSG_TMPL = '''\ +

Last message received:

+
+From: %(sender)s
+Subject: %(subject)s
+Date: %(date)s
+
+%(body)s
+
+''' + +class LastMsg(ndb.Model): + 'LastMsg entity for registering last-received email message' + sender = ndb.StringProperty(indexed=False) + subject = ndb.StringProperty(indexed=False) + date = ndb.StringProperty(indexed=False) + body = ndb.StringProperty(indexed=False) + + +class EmailHandler(mail_handlers.InboundMailHandler): + ''' + email receipt (POST) handler: + - extract email message from request payload + - get last message singleton entity (or create empty one) + - add core values: sender, subject, date + - extract and decode text (from first plain text body) + - save entity and return + ''' + def receive(self, msg): + last_msg = LastMsg.get_or_insert(KEY_NAME, sender='') + # quick loop to assign last_msg.FIELD = msg.FIELD for each field + for field in FIELDS: + setattr(last_msg, field, getattr(msg, field)) + last_msg.body = (msg.bodies('text/plain').next())[1].decode() + last_msg.put() + + +class MainHandler(webapp2.RequestHandler): + ''' + main application (GET) handler: + - get last-message entity from Datastore + - convert to dict then escape special characters + - drop values into template and return + ''' + def get(self): + last_msg = LastMsg.get_by_id(KEY_NAME) + msg_dict = last_msg.to_dict() + self.response.write( + MSG_TMPL % dict((k, escape(msg_dict[k])) for k in msg_dict)) + + +app = webapp2.WSGIApplication([ + ('/', MainHandler), + ('/_ah/mail/.+', EmailHandler), +], debug=True) diff --git a/mod22-bundled/mail3/.gcloudignore b/mod22-bundled/mail3/.gcloudignore new file mode 100644 index 0000000..af73b5d --- /dev/null +++ b/mod22-bundled/mail3/.gcloudignore @@ -0,0 +1,79 @@ +# This file specifies files that are *not* uploaded to Google Cloud Platform +# using gcloud. It follows the same syntax as .gitignore, with the addition of +# "#!include" directives (which insert the entries of the given .gitignore-style +# file at that point). +# +# For more information, run: +# $ gcloud topic gcloudignore +# +.gcloudignore + +# Source code control files +.git/ +.gitignore +.hgignore +.hg/ + +# README/text files +LICENSE +*.md + +# Tests/results (not in .gitignore) +noxfile.py +pylintrc +pylintrc.test + +# most of .gitignore (except `lib`) +# +# Python +*.py[cod] +__pycache__/ +/setup.cfg + +# C extensions +*.so + +# Packages +*.egg +*.egg-info +dist +build +eggs +.eggs +parts +bin +var +sdist +develop-eggs +.installed.cfg +lib64 +*.tgz + +# Installer logs +pip-log.txt + +# Tests/results +.nox/ +.pytest_cache/ +.cache +.pytype +.coverage +coverage.xml +*sponge_log.xml +system_tests/local_test_setup + +# Mac +.DS_Store + +# IDEs/editors +*.sw[op] +*~ +.vscode +.idea + +# Built documentation +docs/_build +docs.metadata + +# Virtual environment +env/ diff --git a/mod22-bundled/mail3/app.yaml b/mod22-bundled/mail3/app.yaml new file mode 100644 index 0000000..80be05e --- /dev/null +++ b/mod22-bundled/mail3/app.yaml @@ -0,0 +1,22 @@ +# Copyright 2022 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +runtime: python310 +app_engine_apis: true + +inbound_services: +- mail + +env_variables: + NDB_USE_CROSS_COMPATIBLE_PICKLE_PROTOCOL: 'True' diff --git a/mod22-bundled/mail3/main.py b/mod22-bundled/mail3/main.py new file mode 100644 index 0000000..cbf1d50 --- /dev/null +++ b/mod22-bundled/mail3/main.py @@ -0,0 +1,74 @@ +# Copyright 2022 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from html import escape +from flask import Flask, request +from google.appengine.api import mail, wrap_wsgi_app +from google.appengine.ext import ndb + +KEY_NAME = 'SECRET' +FIELDS = frozenset(('sender', 'subject', 'date')) +MSG_TMPL = '''\ +

Last message received:

+
+From: %(sender)s
+Subject: %(subject)s
+Date: %(date)s
+
+%(body)s
+
+''' + +app = Flask(__name__) +app.wsgi_app = wrap_wsgi_app(app.wsgi_app) + +class LastMsg(ndb.Model): + 'LastMsg entity for registering last-received email message' + sender = ndb.StringProperty(indexed=False) + subject = ndb.StringProperty(indexed=False) + date = ndb.StringProperty(indexed=False) + body = ndb.StringProperty(indexed=False) + + +@app.route('/_ah/mail/', methods=['POST']) +def receive(path): + ''' + email receipt (POST) handler: + - extract email message from request payload + - get last message singleton entity (or create empty one) + - add core values: sender, subject, date + - extract and decode text (from first plain text body) + - save entity and return + ''' + msg = mail.InboundEmailMessage(request.get_data()) + last_msg = LastMsg.get_or_insert(KEY_NAME, sender='') + # quick loop to assign last_msg.FIELD = msg.FIELD for each field + for field in FIELDS: + setattr(last_msg, field, getattr(msg, field)) + last_msg.body = next(msg.bodies('text/plain'))[1].decode() + last_msg.put() + return '' + + +@app.route('/') +def root(): + ''' + main application (GET) handler: + - get last-message entity from Datastore + - convert to dict then escape special characters + - drop values into template and return + ''' + last_msg = LastMsg.get_by_id(KEY_NAME) + msg_dict = last_msg.to_dict() + return MSG_TMPL % {k: escape(msg_dict[k]) for k in msg_dict} diff --git a/mod22-bundled/mail3/requirements.txt b/mod22-bundled/mail3/requirements.txt new file mode 100644 index 0000000..a2e7c7f --- /dev/null +++ b/mod22-bundled/mail3/requirements.txt @@ -0,0 +1,2 @@ +flask +appengine-python-standard From fde3f18844fba2eb1dfba45f13c62422c539378a Mon Sep 17 00:00:00 2001 From: wesley chun Date: Sat, 10 Dec 2022 03:58:07 -0800 Subject: [PATCH 50/56] add missing README --- mod22-bundled/mail3/README.md | 3 +++ 1 file changed, 3 insertions(+) create mode 100644 mod22-bundled/mail3/README.md diff --git a/mod22-bundled/mail3/README.md b/mod22-bundled/mail3/README.md new file mode 100644 index 0000000..8d81935 --- /dev/null +++ b/mod22-bundled/mail3/README.md @@ -0,0 +1,3 @@ +# Module 22 - Using Mail bundled service (Python 3) + +This repo folder represents the Module 22 Python 3 sample app for the Mail bundled service. From bd90f9da2c0bfb43781f9ae1a44e41bfde4c53b3 Mon Sep 17 00:00:00 2001 From: wesley chun Date: Sat, 10 Dec 2022 12:14:41 -0800 Subject: [PATCH 51/56] minor Mod22 tweaks --- README.md | 4 ++-- mod22-bundled/README.md | 2 +- mod22-bundled/blobstore2/main.py | 1 + mod22-bundled/blobstore3/main.py | 1 + mod22-bundled/deferred2/main.py | 7 +++++-- mod22-bundled/deferred3/main.py | 7 +++++-- mod22-bundled/mail2/main.py | 5 +++-- mod22-bundled/mail3/main.py | 1 + 8 files changed, 19 insertions(+), 9 deletions(-) diff --git a/README.md b/README.md index dd7c171..1683c4a 100644 --- a/README.md +++ b/README.md @@ -76,7 +76,7 @@ Topic | Module ordering | Description --- | --- | --- Baseline | 0 ⇒ 1 | Not a migration but a description of the baseline application (review this material before doing any migrations) Web framework | 1 ⇒ _everything else_ | Current App Engine runtimes do not come with a web framework, so this must be the first migration performed. All migrations below can be performed after this one. -Bundled services | 17 | This module is for Python bundled services users interested in how to access those services from Python 3. +Bundled services | 17 ⇒ 22 | These modules are for those who want to continue using Python bundled services from Python 3 App Engine. Datastore | 2 [⇒ 3 [⇒ 6]] | Moving off App Engine `ndb` makes your apps more portable, so the **Module 2** Cloud NDB migration is _recommended_. **Module 3:** Migrating to Cloud Datastore (Firestore in Datastore mode) is _optional_ and only recommended if you have other code using Cloud Datastore. **Module 6**: Migrating to Cloud Firestore (Native mode) is generally _not recommended_ unless you must have the Firebase features it has, and those features will eventually be integrated into Cloud Datastore. (Push) Task Queues | [7 ⇒] 8 [⇒ 9] | Moving off App Engine `taskqueue` makes your apps more portable, so the **Module 8** Cloud Tasks migration is _recommended_ for those using push tasks. Those unfamiliar with push tasks should do **Module 7** first to add push tasks to the sample app. **Module 9:** Migrating to Cloud Datastore (Firestore in Datastore mode), Cloud Tasks (v2), and Python 3 is _optional_ and only recommended if you have other code using Cloud Datastore and considering upgrading to Python 3. (Pull) Task Queues | [18 ⇒] 19 | Moving off App Engine `taskqueue` makes your apps more portable, so the **Module 19** Cloud Pub/Sub migration is _recommended_ for those using pull tasks. The app is also ported to Python 3. Those unfamiliar with pull tasks should do **Module 18** first to add pull tasks to the sample app. @@ -119,7 +119,7 @@ Module | Topic | Video | Codelab | START here | FINISH here 19|Migrate to Cloud Pub/Sub| _TBD_ | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-19-pubsub?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrpubsub_sms_202016&utm_content=-) | Module 18 [code](/mod18-gaepull) (2.x) | Module 19 [code](/mod19-pubsub) (2.x & 3.x) 20|Add App Engine `users` | _TBD_ | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-20-gaeusers?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrgaeusers_sms_202119&utm_content=-)| Module 1 [code](/mod1-flask) (2.x) | Module 20 [code](/mod20-gaeusers) (2.x) 21|Migrate to Cloud Identity Platform | _TBD_ | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-21-idenplat?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgridenplat_sms_202119&utm_content=-)| Module 20 [code](/mod20-gaeusers) (2.x) | Module 21 [code](/mod21a-idenplat) (2.x) & [code](/mod21b-idenplat) (3.x) -22|Migrate to Python 3 bundled services (Part 2)| _TBD_ | _N/A_ | Module 22 [code](/mod22-bundled) (2.x & 3.x) | _(same)_ +22|Migrate to Python 3 bundled services (Part 2)| _TBD_ | _N/A_ | Module 22 [code](/mod22-bundled) (2.x & 3.x) | _(⇐ same folder)_ ### Table of contents diff --git a/mod22-bundled/README.md b/mod22-bundled/README.md index 0526cbb..a41fecd 100644 --- a/mod22-bundled/README.md +++ b/mod22-bundled/README.md @@ -2,7 +2,7 @@ There are six folders in this directory that contain full applications using the Blobstore, Deferred, and Mail bundled services. A Python 2 version using the `webapp2` framework is provided along with a modernized Python 3 equivalent using the Flask web framework and the Python 3 App Engine SDK. -Bundled service | Python 2 `webapp` | Python 3 Flask +Bundled service | Python 2 `webapp2` | Python 3 Flask --- | --- | --- Blobstore | [code](blobstore2) | [code](blobstore3) Deferred | [code](deferred2) | [code](deferred3) diff --git a/mod22-bundled/blobstore2/main.py b/mod22-bundled/blobstore2/main.py index 483318f..f245455 100644 --- a/mod22-bundled/blobstore2/main.py +++ b/mod22-bundled/blobstore2/main.py @@ -17,6 +17,7 @@ from google.appengine.ext.webapp import blobstore_handlers UPLOAD_FORM = '''\ +Module 22 Blobstore sample app

Upload photo:

diff --git a/mod22-bundled/blobstore3/main.py b/mod22-bundled/blobstore3/main.py index e56c60a..7f3bca8 100644 --- a/mod22-bundled/blobstore3/main.py +++ b/mod22-bundled/blobstore3/main.py @@ -17,6 +17,7 @@ from google.appengine.ext import blobstore, ndb UPLOAD_FORM = '''\ +Module 22 Blobstore sample app

Upload photo:

diff --git a/mod22-bundled/deferred2/main.py b/mod22-bundled/deferred2/main.py index 27b2229..dbe78c7 100644 --- a/mod22-bundled/deferred2/main.py +++ b/mod22-bundled/deferred2/main.py @@ -16,6 +16,10 @@ from google.appengine.ext import deferred, ndb KEY_NAME = 'SECRET' +OUTPUT = '''\ +Module 22 Deferred sample app +Counter at %d... bump requested. +''' class Counter(ndb.Model): 'Counter entity: autoincrement integer' @@ -33,8 +37,7 @@ class MainHandler(webapp2.RequestHandler): def get(self): 'main application (GET) handler' entity = Counter.get_by_id(KEY_NAME) - self.response.write('Counter at %d... bump requested.' % ( - entity.count if entity else 0)) + self.response.write(OUTPUT % (entity.count if entity else 0)) deferred.defer(bump_counter_later, KEY_NAME) diff --git a/mod22-bundled/deferred3/main.py b/mod22-bundled/deferred3/main.py index 26062dc..a32e007 100644 --- a/mod22-bundled/deferred3/main.py +++ b/mod22-bundled/deferred3/main.py @@ -17,6 +17,10 @@ from google.appengine.ext import deferred, ndb KEY_NAME = 'SECRET' +OUTPUT = '''\ +Module 22 Deferred sample app +Counter at %d... bump requested. +''' app = Flask(__name__) app.wsgi_app = wrap_wsgi_app(app.wsgi_app, use_deferred=True) @@ -36,7 +40,6 @@ def bump_counter_later(key): def root(): 'main application (GET) handler' entity = Counter.get_by_id(KEY_NAME) - output = 'Counter at %d... bump requested.' % ( - entity.count if entity else 0) + output = OUTPUT % (entity.count if entity else 0) deferred.defer(bump_counter_later, KEY_NAME) return output diff --git a/mod22-bundled/mail2/main.py b/mod22-bundled/mail2/main.py index f63b6b8..fe72c37 100644 --- a/mod22-bundled/mail2/main.py +++ b/mod22-bundled/mail2/main.py @@ -20,6 +20,7 @@ KEY_NAME = 'SECRET' FIELDS = frozenset(('sender', 'subject', 'date')) MSG_TMPL = '''\ +Module 22 Mail sample app

Last message received:

 From: %(sender)s
@@ -66,8 +67,8 @@ class MainHandler(webapp2.RequestHandler):
     def get(self):
         last_msg = LastMsg.get_by_id(KEY_NAME)
         msg_dict = last_msg.to_dict()
-        self.response.write(
-                MSG_TMPL % dict((k, escape(msg_dict[k])) for k in msg_dict))
+        self.response.write(MSG_TMPL %
+                dict((k, escape(msg_dict[k])) for k in msg_dict))
 
 
 app = webapp2.WSGIApplication([
diff --git a/mod22-bundled/mail3/main.py b/mod22-bundled/mail3/main.py
index cbf1d50..28b6cb8 100644
--- a/mod22-bundled/mail3/main.py
+++ b/mod22-bundled/mail3/main.py
@@ -20,6 +20,7 @@
 KEY_NAME = 'SECRET'
 FIELDS = frozenset(('sender', 'subject', 'date'))
 MSG_TMPL = '''\
+Module 22 Mail sample app
 

Last message received:

 From: %(sender)s

From 2255e540c92b0c3e56ebb562d74074daece75d69 Mon Sep 17 00:00:00 2001
From: wesley chun 
Date: Tue, 13 Dec 2022 17:18:26 -0800
Subject: [PATCH 52/56] add Mod19 video; minor README updates

---
 README.md               | 36 ++++++++++++++++++------------------
 mod22-bundled/README.md |  2 +-
 2 files changed, 19 insertions(+), 19 deletions(-)

diff --git a/README.md b/README.md
index 1683c4a..e37e6d0 100644
--- a/README.md
+++ b/README.md
@@ -76,7 +76,7 @@ Topic | Module ordering | Description
 --- | --- | ---
 Baseline | 0 ⇒ 1 | Not a migration but a description of the baseline application (review this material before doing any migrations)
 Web framework | 1 ⇒ _everything else_ | Current App Engine runtimes do not come with a web framework, so this must be the first migration performed. All migrations below can be performed after this one.
-Bundled services | 17 ⇒ 22 | These modules are for those who want to continue using Python bundled services from Python 3 App Engine.
+Bundled services | 17 and 22 | These modules are for those who want to continue using Python bundled services from Python 3 App Engine.
 Datastore | 2 [⇒ 3 [⇒ 6]] | Moving off App Engine `ndb` makes your apps more portable, so the **Module 2** Cloud NDB migration is _recommended_. **Module 3:** Migrating to Cloud Datastore (Firestore in Datastore mode) is _optional_ and only recommended if you have other code using Cloud Datastore. **Module 6**: Migrating to Cloud Firestore (Native mode) is generally _not recommended_ unless you must have the Firebase features it has, and those features will eventually be integrated into Cloud Datastore.
 (Push) Task Queues | [7 ⇒] 8 [⇒ 9] | Moving off App Engine `taskqueue` makes your apps more portable, so the **Module 8** Cloud Tasks migration is _recommended_ for those using push tasks. Those unfamiliar with push tasks should do **Module 7** first to add push tasks to the sample app. **Module 9:** Migrating to Cloud Datastore (Firestore in Datastore mode), Cloud Tasks (v2), and Python 3 is _optional_ and only recommended if you have other code using Cloud Datastore and considering upgrading to Python 3.
 (Pull) Task Queues | [18 ⇒] 19 | Moving off App Engine `taskqueue` makes your apps more portable, so the **Module 19** Cloud Pub/Sub migration is _recommended_ for those using pull tasks. The app is also ported to Python 3. Those unfamiliar with pull tasks should do **Module 18** first to add pull tasks to the sample app.
@@ -97,26 +97,26 @@ The table below summarizes migration module resources currently available along
 
 Module | Topic | Video | Codelab | START here | FINISH here
 --- | --- | --- | --- | --- | ---
-0|Baseline app| [link](https://twitter.com/googledevs/status/1407755281786867714?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_smsintro_201023&utm_content=-) | _N/A_ (no tutorial; just review the code) | _N/A_ | Module 0 [code](/mod0-baseline) (2.x)
-1|Migrate to Flask| [link](https://twitter.com/googledevs/status/1413273119071031299?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrwa2flsk_201008&u%20tm_content=-) | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-1-flask?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrwa2flsk_201008&utm_content=-) | Module 0 [code](/mod0-baseline) (2.x) | Module 1 [code](/mod1-flask) (2.x) (and [code](/mod1b-flask) (3.x))
-2|Migrate to Cloud NDB| [link](https://twitter.com/googledevs/status/1417892493383802883?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrcloudndb_201021&utm_content=-) | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-2-cloudndb?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrcloudndb_201021&utm_content=-) | Module 1 [code](/mod1-flask) (2.x) | Module 2 [code](/mod2a-cloudndb) (2.x) & [code](/mod2b-cloudndb) (3.x)
-3|Migrate to Cloud Datastore| [link](http://twitter.com/googledevs/status/1422966928910393347?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrcloudds_201003&utm_content=-) | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-3-datastore?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrcloudds_201003&utm_content=-) | Module 2 [code](/mod2a-cloudndb) (2.x) & [code](/mod2b-cloudndb) (3.x) | Module 3 [code](/mod3a-datastore) (2.x) & [code](/mod3b-datastore) (3.x)
-4|Migrate to Cloud Run with Docker| [link](https://twitter.com/googledevs/status/1428041270702735362?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrcrdckr_sms_201017&utm_content=-)| [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-4-rundocker?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrcrdckr_sms_201017&utm_content=-) | Module 2 [code](/mod2a-cloudndb) (2.x) & Module 3 [code](/mod3b-datastore) (3.x) | Module 4 [code](/mod4a-rundocker) (2.x) & [code](/mod4b-rundocker) (3.x)
-5|Migrate to Cloud Run with Buildpacks| [link](https://twitter.com/googledevs/status/1433113274984271875?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrcrbdpk_sms_201031&utm_content=-) | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-5-runbldpks?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrcrbdpk_sms_201031&utm_content=-) | Module 2 [code](/mod2b-cloudndb) (3.x) | Module 5 [code](/mod5-runbldpks) (3.x)
+0|Baseline app| [link](https://developers.googleblog.com/2021/06/introducing-serverless-migration.html?utm_source=blog&utm_medium=partner&utm_campaign=CDR_wes_aap-serverless_smsintro_201023)| _N/A_ (no tutorial; just review the code) | _N/A_ | Module 0 [code](/mod0-baseline) (2.x)
+1|Migrate to Flask| [link](https://developers.googleblog.com/2021/07/migrating-from-app-engine-webapp2-to-flask.html?utm_source=blog&utm_medium=partner&utm_campaign=CDR_wes_aap-serverless_mgrwa2flsk_201008)| [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-1-flask?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrwa2flsk_201008&utm_content=-) | Module 0 [code](/mod0-baseline) (2.x) | Module 1 [code](/mod1-flask) (2.x) (and [code](/mod1b-flask) (3.x))
+2|Migrate to Cloud NDB| [link](https://developers.googleblog.com/2021/07/migrating-from-app-engine-ndb-to-cloud-ndb.html?utm_source=blog&utm_medium=partner&utm_campaign=CDR_wes_aap-serverless_mgrcloudndb_201021)| [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-2-cloudndb?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrcloudndb_201021&utm_content=-) | Module 1 [code](/mod1-flask) (2.x) | Module 2 [code](/mod2a-cloudndb) (2.x) & [code](/mod2b-cloudndb) (3.x)
+3|Migrate to Cloud Datastore| [link](https://developers.googleblog.com/2021/08/cloud-ndb-to-cloud-datastore-migration.html?utm_source=blog&utm_medium=partner&utm_campaign=CDR_wes_aap-serverless_mgrcloudds_201003) | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-3-datastore?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrcloudds_201003&utm_content=-) | Module 2 [code](/mod2a-cloudndb) (2.x) & [code](/mod2b-cloudndb) (3.x) | Module 3 [code](/mod3a-datastore) (2.x) & [code](/mod3b-datastore) (3.x)
+4|Migrate to Cloud Run with Docker| [link](https://developers.googleblog.com/2021/08/containerizing-google-app-engine-apps-for-cloud-run.html?utm_source=blog&utm_medium=partner&utm_campaign=CDR_wes_aap-serverless_mgrcrdckr_sms_201017) [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-4-rundocker?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrcrdckr_sms_201017&utm_content=-) | Module 2 [code](/mod2a-cloudndb) (2.x) & Module 3 [code](/mod3b-datastore) (3.x) | Module 4 [code](/mod4a-rundocker) (2.x) & [code](/mod4b-rundocker) (3.x)
+5|Migrate to Cloud Run with Buildpacks| [link](https://developers.googleblog.com/2021/09/an-easier-way-to-move-your-app-engine-to-cloud-run.html?utm_source=blog&utm_medium=partner&utm_campaign=CDR_wes_aap-serverless_mgrcrbdpk_sms_201031) | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-5-runbldpks?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrcrbdpk_sms_201031&utm_content=-) | Module 2 [code](/mod2b-cloudndb) (3.x) | Module 5 [code](/mod5-runbldpks) (3.x)
 6|Migrate to Cloud Firestore| _N/A_ | _N/A_ | Module 3 [code](/mod3b-datastore) (3.x) | _no work required; [Datastore upgrade automatic](https://cloud.google.com/datastore/docs/upgrade-to-firestore)_
-7|Add App Engine `taskqueue` push tasks| [link](https://twitter.com/googledevs/status/1443410302113099778?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrgaetasks_sms_201028&utm_content=-) | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-7-gaetasks?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrgaetasks_sms_201028&utm_content=-) | Module 1 [code](/mod1-flask) (2.x) | Module 7 [code](/mod7-gaetasks) (2.x) & [code](/mod7b-gaetasks) (3.x)
-8|Migrate to Cloud Tasks| [link](https://twitter.com/googledevs/status/1450960021018267656?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrcloudtasks_sms_201112&utm_content=-) | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-8-cloudtasks?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrcloudtasks_sms_201112&utm_content=-) | Module 7 [code](/mod7-gaetasks) (2.x) | Module 8 [code](/mod8-cloudtasks) (2.x)
+7|Add App Engine `taskqueue` push tasks| [link](https://developers.googleblog.com/2021/09/how-to-use-app-engine-push-queues-in.html?utm_source=blog&utm_medium=partner&utm_campaign=CDR_wes_aap-serverless_mgrgaetasks_sms_201028) | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-7-gaetasks?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrgaetasks_sms_201028&utm_content=-) | Module 1 [code](/mod1-flask) (2.x) | Module 7 [code](/mod7-gaetasks) (2.x) & [code](/mod7b-gaetasks) (3.x)
+8|Migrate to Cloud Tasks| [link](https://developers.googleblog.com/2021/10/migrating-app-engine-push-queues-to.html?utm_source=blog&utm_medium=partner&utm_campaign=CDR_wes_aap-serverless_mgrcloudtasks_sms_201112) | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-8-cloudtasks?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrcloudtasks_sms_201112&utm_content=-) | Module 7 [code](/mod7-gaetasks) (2.x) | Module 8 [code](/mod8-cloudtasks) (2.x)
 9|Migrate to Python 3, Cloud Datastore & Cloud Tasks v2| _TBD_ | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-9-py3dstasks?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrpy3fstasks_sms_201126&utm_content=-) | Module 8 [code](/mod8-cloudtasks) (2.x) | Module 9 [code](/mod9-py3dstasks)
 10|Migrate Datastore/Firestore data to another project| _TBD_ | _N/A_ | _N/A_ | _TBD_
-11|Migrate to Cloud Functions| [link](https://twitter.com/googledevs/status/1520206298834481153?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrcloudfuncs_sms_202006&utm_content=-) | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-11-functions?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrcloudfuncs_sms_202006&utm_content=-) | Module 2 [code](/mod2b-cloudndb) (3.x) | Module 11 [code](/mod11-functions) (3.x)
-12|Add App Engine `memcache`| [link](https://twitter.com/googledevs/status/1527303061953126402?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrmemcache_sms_202006&utm_content=-) | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-12-memcache?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrmemcache_sms_202006&utm_content=-) | Module 1 [code](/mod1-flask) (2.x) | Module 12 [code](/mod12-memcache) (2.x) & [code](/mod12b-memcache) (3.x)
-13|Migrate to Cloud Memorystore| [link](https://twitter.com/googledevs/status/1537132939426799616?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrmemorystore_sms_202029&utm_content=-) | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-13-memorystore?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrmemorystore_sms_202029&utm_content=-) | Module 12 [code](/mod12-memcache) (2.x) & [code](/mod12b-memcache) (3.x) | Module 13 [code](/mod13a-memorystore) (2.x) & [code](/mod13b-memorystore) (3.x)
+11|Migrate to Cloud Functions| [link](https://developers.googleblog.com/2022/04/how-can-app-engine-users-take-advantage-of-cloud-functions.html?utm_source=blog&utm_medium=partner&utm_campaign=CDR_wes_aap-serverless_mgrcloudfuncs_sms_202006) | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-11-functions?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrcloudfuncs_sms_202006&utm_content=-) | Module 2 [code](/mod2b-cloudndb) (3.x) | Module 11 [code](/mod11-functions) (3.x)
+12|Add App Engine `memcache`| [link](https://developers.googleblog.com/2022/05/how-to-use-app-engine-memcache-in-flask-apps.html?utm_source=blog&utm_medium=partner&utm_campaign=CDR_wes_aap-serverless_mgrmemcache_sms_202006) | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-12-memcache?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrmemcache_sms_202006&utm_content=-) | Module 1 [code](/mod1-flask) (2.x) | Module 12 [code](/mod12-memcache) (2.x) & [code](/mod12b-memcache) (3.x)
+13|Migrate to Cloud Memorystore| [link](https://developers.googleblog.com/2022/06/Migrating-from-App-Engine-Memcache-to-Cloud-Memorystore-Module-13.html?utm_source=blog&utm_medium=partner&utm_campaign=CDR_wes_aap-serverless_mgrmemorystore_sms_202029) | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-13-memorystore?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrmemorystore_sms_202029&utm_content=-) | Module 12 [code](/mod12-memcache) (2.x) & [code](/mod12b-memcache) (3.x) | Module 13 [code](/mod13a-memorystore) (2.x) & [code](/mod13b-memorystore) (3.x)
 14|Migrate service between projects| _TBD_ | _TBD_ | _TBD_ | _TBD_
-15|Add App Engine `blobstore`| [link](https://twitter.com/googledevs/status/1552384740052934657?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrblobstore_sms_202029&utm_content=-) | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-15-blobstore?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrblobstore_sms_202029&utm_content=-) | Module 0 [code](/mod0-baseline) (2.x) | Module 15 [code](/mod15-blobstore) (2.x)
-16|Migrate to Cloud Storage| [link](https://twitter.com/googledevs/status/1559285905961123845?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrcloudstorage_sms_202029&utm_content=-) | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-16-cloudstorage?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrcloudstorage_sms_202029&utm_content=-) | Module 15 [code](/mod15-blobstore) (2.x) | Module 16 [code](/mod16-cloudstorage) (2.x & 3.x)
-17|Migrate to Python 3 bundled services (Part 1)| [link](https://twitter.com/googledevs/status/1587855923254624259?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrwormhole_sms_202002&utm_content=-) | [link](http://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-17-bundled?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrwormhole_sms_202002&utm_content=-) | Module 1 [code](/mod1-flask) (2.x) | Module 1 [code](/mod1b-flask) (3.x)
-18|Add App Engine `taskqueue` pull tasks| [link](https://twitter.com/googledevs/status/1598013491381665808?utm_source=twitter&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrgaepull_sms_202013&utm_content=-) | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-18-gaepull?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrgaepull_sms_202013&utm_content=-) | Module 1 [code](/mod1-flask) (2.x) | Module 18 [code](/mod18-gaepull) (2.x)
-19|Migrate to Cloud Pub/Sub| _TBD_ | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-19-pubsub?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrpubsub_sms_202016&utm_content=-) | Module 18 [code](/mod18-gaepull) (2.x) | Module 19 [code](/mod19-pubsub) (2.x & 3.x)
+15|Add App Engine `blobstore`| [link](https://developers.googleblog.com/2022/07/how-to-use-app-engine-blobstore-Module15.html?utm_source=blog&utm_medium=partner&utm_campaign=CDR_wes_aap-serverless_mgrblobstore_sms_202029) | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-15-blobstore?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrblobstore_sms_202029&utm_content=-) | Module 0 [code](/mod0-baseline) (2.x) | Module 15 [code](/mod15-blobstore) (2.x)
+16|Migrate to Cloud Storage| [link](https://developers.googleblog.com/2022/08/migrating-from-app-engine-blobstore-to-cloud-storage-module-16.html?utm_source=blog&utm_medium=partner&utm_campaign=CDR_wes_aap-serverless_mgrcloudstorage_sms_202029) | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-16-cloudstorage?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrcloudstorage_sms_202029&utm_content=-) | Module 15 [code](/mod15-blobstore) (2.x) | Module 16 [code](/mod16-cloudstorage) (2.x & 3.x)
+17|Migrate to Python 3 bundled services (Part 1)| [link](https://developers.googleblog.com/2022/10/extending-support-for-app-engine-bundled-services-module-17.html?utm_source=blog&utm_medium=partner&utm_campaign=CDR_wes_aap-serverless_mgrwormhole_sms_202002) | [link](http://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-17-bundled?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrwormhole_sms_202002&utm_content=-) | Module 1 [code](/mod1-flask) (2.x) | Module 1 [code](/mod1b-flask) (3.x)
+18|Add App Engine `taskqueue` pull tasks| [link](https://developers.googleblog.com/2022/11/how-to-use-app-engine-pull-tasks-module-18.html?utm_source=blog&utm_medium=partner&utm_campaign=CDR_wes_aap-serverless_mgrgaepull_sms_202013) | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-18-gaepull?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrgaepull_sms_202013&utm_content=-) | Module 1 [code](/mod1-flask) (2.x) | Module 18 [code](/mod18-gaepull) (2.x)
+19|Migrate to Cloud Pub/Sub| [link](https://developers.googleblog.com/2022/12/migrating-from-app-engine-pull-tasks-to.html?utm_source=blog&utm_medium=partner&utm_campaign=CDR_wes_aap-serverless_mgrpubsub_sms_202016) | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-19-pubsub?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrpubsub_sms_202016&utm_content=-) | Module 18 [code](/mod18-gaepull) (2.x) | Module 19 [code](/mod19-pubsub) (2.x & 3.x)
 20|Add App Engine `users` | _TBD_ | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-20-gaeusers?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrgaeusers_sms_202119&utm_content=-)| Module 1 [code](/mod1-flask) (2.x) | Module 20 [code](/mod20-gaeusers) (2.x)
 21|Migrate to Cloud Identity Platform | _TBD_ | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-21-idenplat?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgridenplat_sms_202119&utm_content=-)| Module 20 [code](/mod20-gaeusers) (2.x) |  Module 21 [code](/mod21a-idenplat) (2.x) & [code](/mod21b-idenplat) (3.x)
 22|Migrate to Python 3 bundled services (Part 2)| _TBD_ | _N/A_ | Module 22  [code](/mod22-bundled) (2.x & 3.x) | _(⇐ same folder)_
@@ -341,7 +341,7 @@ If your original app users does *not* have a user interface, i.e., mobile backen
 
 ## Accessing legacy services in second generation
 
-Many legacy App Engine first generation platform (Python 2, Java 8, PHP 5, and Go 1.11 & older) services are available ([as of Sep 2021](https://twitter.com/googledevs/status/1445916786755571712) for second generation runtimes (Python 3, Java 11/17, PHP 7/8, and Go 1.12 & newer) in a public preview. There are no videos or codelabs yet, however the Module 1 Flask migration using App Engine `ndb` [Python 2 sample](/mod1-flask) is available in [Python 3](/mod1b-flask) if you have access. Similarly, Python 3 editions are also available for Modules 7 and 12 which add usage of App Engine `taskqueue` and `memcache`, respectively. Also see the [documentation on accessing bundled services from Python 3](https://cloud.google.com/appengine/docs/standard/python3/services/access).
+Many legacy App Engine first generation platform (Python 2, Java 8, PHP 5, and Go 1.11 & older) services are available ([as of Sep 2021](https://cloud.google.com/blog/products/serverless/support-for-app-engine-services-in-second-generation-runtimes) for second generation runtimes (Python 3, Java 11/17, PHP 7/8, and Go 1.12 & newer) in a public preview. There are no videos or codelabs yet, however the Module 1 Flask migration using App Engine `ndb` [Python 2 sample](/mod1-flask) is available in [Python 3](/mod1b-flask) if you have access. Similarly, Python 3 editions are also available for Modules 7 and 12 which add usage of App Engine `taskqueue` and `memcache`, respectively. Also see the [documentation on accessing bundled services from Python 3](https://cloud.google.com/appengine/docs/standard/python3/services/access).
 
 
 ## Community
diff --git a/mod22-bundled/README.md b/mod22-bundled/README.md
index a41fecd..8b8b87a 100644
--- a/mod22-bundled/README.md
+++ b/mod22-bundled/README.md
@@ -1,6 +1,6 @@
 # Module 22 - Extending support for App Engine bundled services in Python 3
 
-There are six folders in this directory that contain full applications using the Blobstore, Deferred, and Mail bundled services. A Python 2 version using the `webapp2` framework is provided along with a modernized Python 3 equivalent using the Flask web framework and the Python 3 App Engine SDK.
+There are six folders in this directory that contain (complete) basic applications using the Blobstore, Deferred, and Mail bundled services. A Python 2 version using the `webapp2` framework is provided along with a modernized Python 3 equivalent using the Flask web framework and the App Engine SDK.
 
 Bundled service | Python 2 `webapp2` | Python 3 Flask
 --- | --- | ---

From cb164e3fee9ea9e35516b545a2f575ddcca78256 Mon Sep 17 00:00:00 2001
From: wesley chun 
Date: Thu, 15 Dec 2022 13:36:59 -0800
Subject: [PATCH 53/56] Mod22 README updates

---
 mod22-bundled/blobstore2/README.md | 5 +++++
 mod22-bundled/blobstore3/README.md | 5 +++++
 mod22-bundled/deferred2/README.md  | 5 +++++
 mod22-bundled/deferred3/README.md  | 5 +++++
 mod22-bundled/mail2/README.md      | 5 +++++
 mod22-bundled/mail3/README.md      | 5 +++++
 6 files changed, 30 insertions(+)

diff --git a/mod22-bundled/blobstore2/README.md b/mod22-bundled/blobstore2/README.md
index e093a2c..183853f 100644
--- a/mod22-bundled/blobstore2/README.md
+++ b/mod22-bundled/blobstore2/README.md
@@ -1,3 +1,8 @@
 # Module 22 - Using Blobstore bundled service (Python 2)
 
 This repo folder represents the Module 22 Python 2 sample app for the Blobstore bundled service.
+
+- The app lets end-users upload one photo then displays it to confirm the upload was successful.
+- The Python 2 version of the app uses the `webapp2` framework while the Python 3 version uses Flask and the App Engine SDK to access the bundled services.
+- Also check out both `app.yaml` files for additional changes between runtimes.
+- The Python 3 version of the app uses 3rd-party packages, and as such, has a `requirements.txt` file.
diff --git a/mod22-bundled/blobstore3/README.md b/mod22-bundled/blobstore3/README.md
index e866055..bd891d7 100644
--- a/mod22-bundled/blobstore3/README.md
+++ b/mod22-bundled/blobstore3/README.md
@@ -1,3 +1,8 @@
 # Module 22 - Using Blobstore bundled service (Python 3)
 
 This repo folder represents the Module 22 Python 3 sample app for the Blobstore bundled service.
+
+- The app lets end-users upload one photo then displays it to confirm the upload was successful.
+- The Python 2 version of the app uses the `webapp2` framework while the Python 3 version uses Flask and the App Engine SDK to access the bundled services.
+- Also check out both `app.yaml` files for additional changes between runtimes.
+- The Python 3 version of the app uses 3rd-party packages, and as such, has a `requirements.txt` file.
diff --git a/mod22-bundled/deferred2/README.md b/mod22-bundled/deferred2/README.md
index 5945d3e..7cb066e 100644
--- a/mod22-bundled/deferred2/README.md
+++ b/mod22-bundled/deferred2/README.md
@@ -1,3 +1,8 @@
 # Module 22 - Using Deferred bundled service (Python 2)
 
 This repo folder represents the Module 22 Python 2 sample app for the Deferred bundled service.
+
+- The app implements a simple autoincrement counter which gets bumped for every page visit. The visit displays the current counter value then spawns a deferred task to bump it.
+- The Python 2 version of the app uses the `webapp2` framework while the Python 3 version uses Flask and the App Engine SDK to access the bundled services.
+- Also check out both `app.yaml` files for additional changes between runtimes.
+- The Python 3 version of the app uses 3rd-party packages, and as such, has a `requirements.txt` file.
diff --git a/mod22-bundled/deferred3/README.md b/mod22-bundled/deferred3/README.md
index 99036e8..540a471 100644
--- a/mod22-bundled/deferred3/README.md
+++ b/mod22-bundled/deferred3/README.md
@@ -1,3 +1,8 @@
 # Module 22 - Using Deferred bundled service (Python 3)
 
 This repo folder represents the Module 22 Python 3 sample app for the Deferred bundled service.
+
+- The app implements a simple autoincrement counter which gets bumped for every page visit. The visit displays the current counter value then spawns a deferred task to bump it.
+- The Python 2 version of the app uses the `webapp2` framework while the Python 3 version uses Flask and the App Engine SDK to access the bundled services.
+- Also check out both `app.yaml` files for additional changes between runtimes.
+- The Python 3 version of the app uses 3rd-party packages, and as such, has a `requirements.txt` file.
diff --git a/mod22-bundled/mail2/README.md b/mod22-bundled/mail2/README.md
index 0d03f68..c499f6a 100644
--- a/mod22-bundled/mail2/README.md
+++ b/mod22-bundled/mail2/README.md
@@ -1,3 +1,8 @@
 # Module 22 - Using Mail bundled service (Python 2)
 
 This repo folder represents the Module 22 Python 2 sample app for the Mail bundled service.
+
+- The app can receive email, saving only the most recent message received in Datastore. Visiting the app displays that message along with associated metadata (date, subject, sender).
+- The Python 2 version of the app uses the `webapp2` framework while the Python 3 version uses Flask and the App Engine SDK to access the bundled services.
+- Also check out both `app.yaml` files for additional changes between runtimes.
+- The Python 3 version of the app uses 3rd-party packages, and as such, has a `requirements.txt` file.
diff --git a/mod22-bundled/mail3/README.md b/mod22-bundled/mail3/README.md
index 8d81935..2a0e3f6 100644
--- a/mod22-bundled/mail3/README.md
+++ b/mod22-bundled/mail3/README.md
@@ -1,3 +1,8 @@
 # Module 22 - Using Mail bundled service (Python 3)
 
 This repo folder represents the Module 22 Python 3 sample app for the Mail bundled service.
+
+- The app can receive email, saving only the most recent message received in Datastore. Visiting the app displays that message along with associated metadata (date, subject, sender).
+- The Python 2 version of the app uses the `webapp2` framework while the Python 3 version uses Flask and the App Engine SDK to access the bundled services.
+- Also check out both `app.yaml` files for additional changes between runtimes.
+- The Python 3 version of the app uses 3rd-party packages, and as such, has a `requirements.txt` file.

From 52e5761fdabe5fd5a2bf46ef0dca02d9160b545b Mon Sep 17 00:00:00 2001
From: wesley chun 
Date: Fri, 3 Feb 2023 12:00:52 -0800
Subject: [PATCH 54/56] add Mod 20 & 21 videos

---
 README.md | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/README.md b/README.md
index e37e6d0..aed9acd 100644
--- a/README.md
+++ b/README.md
@@ -101,7 +101,7 @@ Module | Topic | Video | Codelab | START here | FINISH here
 1|Migrate to Flask| [link](https://developers.googleblog.com/2021/07/migrating-from-app-engine-webapp2-to-flask.html?utm_source=blog&utm_medium=partner&utm_campaign=CDR_wes_aap-serverless_mgrwa2flsk_201008)| [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-1-flask?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrwa2flsk_201008&utm_content=-) | Module 0 [code](/mod0-baseline) (2.x) | Module 1 [code](/mod1-flask) (2.x) (and [code](/mod1b-flask) (3.x))
 2|Migrate to Cloud NDB| [link](https://developers.googleblog.com/2021/07/migrating-from-app-engine-ndb-to-cloud-ndb.html?utm_source=blog&utm_medium=partner&utm_campaign=CDR_wes_aap-serverless_mgrcloudndb_201021)| [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-2-cloudndb?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrcloudndb_201021&utm_content=-) | Module 1 [code](/mod1-flask) (2.x) | Module 2 [code](/mod2a-cloudndb) (2.x) & [code](/mod2b-cloudndb) (3.x)
 3|Migrate to Cloud Datastore| [link](https://developers.googleblog.com/2021/08/cloud-ndb-to-cloud-datastore-migration.html?utm_source=blog&utm_medium=partner&utm_campaign=CDR_wes_aap-serverless_mgrcloudds_201003) | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-3-datastore?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrcloudds_201003&utm_content=-) | Module 2 [code](/mod2a-cloudndb) (2.x) & [code](/mod2b-cloudndb) (3.x) | Module 3 [code](/mod3a-datastore) (2.x) & [code](/mod3b-datastore) (3.x)
-4|Migrate to Cloud Run with Docker| [link](https://developers.googleblog.com/2021/08/containerizing-google-app-engine-apps-for-cloud-run.html?utm_source=blog&utm_medium=partner&utm_campaign=CDR_wes_aap-serverless_mgrcrdckr_sms_201017) [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-4-rundocker?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrcrdckr_sms_201017&utm_content=-) | Module 2 [code](/mod2a-cloudndb) (2.x) & Module 3 [code](/mod3b-datastore) (3.x) | Module 4 [code](/mod4a-rundocker) (2.x) & [code](/mod4b-rundocker) (3.x)
+4|Migrate to Cloud Run with Docker| [link](https://developers.googleblog.com/2021/08/containerizing-google-app-engine-apps-for-cloud-run.html?utm_source=blog&utm_medium=partner&utm_campaign=CDR_wes_aap-serverless_mgrcrdckr_sms_201017) | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-4-rundocker?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrcrdckr_sms_201017&utm_content=-) | Module 2 [code](/mod2a-cloudndb) (2.x) & Module 3 [code](/mod3b-datastore) (3.x) | Module 4 [code](/mod4a-rundocker) (2.x) & [code](/mod4b-rundocker) (3.x)
 5|Migrate to Cloud Run with Buildpacks| [link](https://developers.googleblog.com/2021/09/an-easier-way-to-move-your-app-engine-to-cloud-run.html?utm_source=blog&utm_medium=partner&utm_campaign=CDR_wes_aap-serverless_mgrcrbdpk_sms_201031) | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-5-runbldpks?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrcrbdpk_sms_201031&utm_content=-) | Module 2 [code](/mod2b-cloudndb) (3.x) | Module 5 [code](/mod5-runbldpks) (3.x)
 6|Migrate to Cloud Firestore| _N/A_ | _N/A_ | Module 3 [code](/mod3b-datastore) (3.x) | _no work required; [Datastore upgrade automatic](https://cloud.google.com/datastore/docs/upgrade-to-firestore)_
 7|Add App Engine `taskqueue` push tasks| [link](https://developers.googleblog.com/2021/09/how-to-use-app-engine-push-queues-in.html?utm_source=blog&utm_medium=partner&utm_campaign=CDR_wes_aap-serverless_mgrgaetasks_sms_201028) | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-7-gaetasks?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrgaetasks_sms_201028&utm_content=-) | Module 1 [code](/mod1-flask) (2.x) | Module 7 [code](/mod7-gaetasks) (2.x) & [code](/mod7b-gaetasks) (3.x)
@@ -117,9 +117,9 @@ Module | Topic | Video | Codelab | START here | FINISH here
 17|Migrate to Python 3 bundled services (Part 1)| [link](https://developers.googleblog.com/2022/10/extending-support-for-app-engine-bundled-services-module-17.html?utm_source=blog&utm_medium=partner&utm_campaign=CDR_wes_aap-serverless_mgrwormhole_sms_202002) | [link](http://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-17-bundled?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrwormhole_sms_202002&utm_content=-) | Module 1 [code](/mod1-flask) (2.x) | Module 1 [code](/mod1b-flask) (3.x)
 18|Add App Engine `taskqueue` pull tasks| [link](https://developers.googleblog.com/2022/11/how-to-use-app-engine-pull-tasks-module-18.html?utm_source=blog&utm_medium=partner&utm_campaign=CDR_wes_aap-serverless_mgrgaepull_sms_202013) | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-18-gaepull?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrgaepull_sms_202013&utm_content=-) | Module 1 [code](/mod1-flask) (2.x) | Module 18 [code](/mod18-gaepull) (2.x)
 19|Migrate to Cloud Pub/Sub| [link](https://developers.googleblog.com/2022/12/migrating-from-app-engine-pull-tasks-to.html?utm_source=blog&utm_medium=partner&utm_campaign=CDR_wes_aap-serverless_mgrpubsub_sms_202016) | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-19-pubsub?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrpubsub_sms_202016&utm_content=-) | Module 18 [code](/mod18-gaepull) (2.x) | Module 19 [code](/mod19-pubsub) (2.x & 3.x)
-20|Add App Engine `users` | _TBD_ | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-20-gaeusers?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrgaeusers_sms_202119&utm_content=-)| Module 1 [code](/mod1-flask) (2.x) | Module 20 [code](/mod20-gaeusers) (2.x)
-21|Migrate to Cloud Identity Platform | _TBD_ | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-21-idenplat?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgridenplat_sms_202119&utm_content=-)| Module 20 [code](/mod20-gaeusers) (2.x) |  Module 21 [code](/mod21a-idenplat) (2.x) & [code](/mod21b-idenplat) (3.x)
-22|Migrate to Python 3 bundled services (Part 2)| _TBD_ | _N/A_ | Module 22  [code](/mod22-bundled) (2.x & 3.x) | _(⇐ same folder)_
+20|Add App Engine `users` | [link](https://developers.googleblog.com/2022/12/how-to-use-app-engine-users-service-module-20.html?utm_source=blog&utm_medium=partner&utm_campaign=CDR_wes_aap-serverless_mgrgaeusers_sms_202119) | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-20-gaeusers?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrgaeusers_sms_202119&utm_content=-)| Module 1 [code](/mod1-flask) (2.x) | Module 20 [code](/mod20-gaeusers) (2.x)
+21|Migrate to Cloud Identity Platform | [link](https://developers.googleblog.com/2023/01/migrating-from-app-engine-users-to-cloud-identity-module-21.html?utm_source=blog&utm_medium=partner&utm_campaign=CDR_wes_aap-serverless_mgridenplat_sms_202119) | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-21-idenplat?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgridenplat_sms_202119&utm_content=-)| Module 20 [code](/mod20-gaeusers) (2.x) |  Module 21 [code](/mod21a-idenplat) (2.x) & [code](/mod21b-idenplat) (3.x)
+22|Migrate to Python 3 bundled services (Part 2)| [link](http://youtu.be/ZhEBSvnz_BQ?list=PL2pQQBHvYcs0PEecTcLD9_VaLvuhK0_VQ&utm_source=youtube&utm_medium=unpaidsoc&utm_campaign=CDR_wes_aap-serverless_mgrwormhole2_sms_202002&utm_content=info_card) | _N/A_ | Module 22  [code](/mod22-bundled) (2.x & 3.x) | _(⇐ same folder)_
 
 
 ### Table of contents

From 983f92d0861c6333567187cfd8ce68ba742387a5 Mon Sep 17 00:00:00 2001
From: janetvong 
Date: Thu, 9 Mar 2023 12:31:54 -0800
Subject: [PATCH 55/56] Updated Module 20-22 with links.

---
 README.md | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/README.md b/README.md
index e37e6d0..2a8e154 100644
--- a/README.md
+++ b/README.md
@@ -117,9 +117,9 @@ Module | Topic | Video | Codelab | START here | FINISH here
 17|Migrate to Python 3 bundled services (Part 1)| [link](https://developers.googleblog.com/2022/10/extending-support-for-app-engine-bundled-services-module-17.html?utm_source=blog&utm_medium=partner&utm_campaign=CDR_wes_aap-serverless_mgrwormhole_sms_202002) | [link](http://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-17-bundled?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrwormhole_sms_202002&utm_content=-) | Module 1 [code](/mod1-flask) (2.x) | Module 1 [code](/mod1b-flask) (3.x)
 18|Add App Engine `taskqueue` pull tasks| [link](https://developers.googleblog.com/2022/11/how-to-use-app-engine-pull-tasks-module-18.html?utm_source=blog&utm_medium=partner&utm_campaign=CDR_wes_aap-serverless_mgrgaepull_sms_202013) | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-18-gaepull?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrgaepull_sms_202013&utm_content=-) | Module 1 [code](/mod1-flask) (2.x) | Module 18 [code](/mod18-gaepull) (2.x)
 19|Migrate to Cloud Pub/Sub| [link](https://developers.googleblog.com/2022/12/migrating-from-app-engine-pull-tasks-to.html?utm_source=blog&utm_medium=partner&utm_campaign=CDR_wes_aap-serverless_mgrpubsub_sms_202016) | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-19-pubsub?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrpubsub_sms_202016&utm_content=-) | Module 18 [code](/mod18-gaepull) (2.x) | Module 19 [code](/mod19-pubsub) (2.x & 3.x)
-20|Add App Engine `users` | _TBD_ | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-20-gaeusers?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrgaeusers_sms_202119&utm_content=-)| Module 1 [code](/mod1-flask) (2.x) | Module 20 [code](/mod20-gaeusers) (2.x)
-21|Migrate to Cloud Identity Platform | _TBD_ | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-21-idenplat?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgridenplat_sms_202119&utm_content=-)| Module 20 [code](/mod20-gaeusers) (2.x) |  Module 21 [code](/mod21a-idenplat) (2.x) & [code](/mod21b-idenplat) (3.x)
-22|Migrate to Python 3 bundled services (Part 2)| _TBD_ | _N/A_ | Module 22  [code](/mod22-bundled) (2.x & 3.x) | _(⇐ same folder)_
+20|Add App Engine `users` | [link](https://goo.gle/3QlGCfG) | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-20-gaeusers?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgrgaeusers_sms_202119&utm_content=-)| Module 1 [code](/mod1-flask) (2.x) | Module 20 [code](/mod20-gaeusers) (2.x)
+21|Migrate to Cloud Identity Platform | [link](https://t.co/hlxcahRgGu) | [link](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-21-idenplat?utm_source=codelabs&utm_medium=et&utm_campaign=CDR_wes_aap-serverless_mgridenplat_sms_202119&utm_content=-)| Module 20 [code](/mod20-gaeusers) (2.x) |  Module 21 [code](/mod21a-idenplat) (2.x) & [code](/mod21b-idenplat) (3.x)
+22|Migrate to Python 3 bundled services (Part 2)| [link](http://youtu.be/ZhEBSvnz_BQ ) | _N/A_ | Module 22  [code](/mod22-bundled) (2.x & 3.x) | _(⇐ same folder)_
 
 
 ### Table of contents

From 8997c2c72637c120a37f3e3868165467dffe7d13 Mon Sep 17 00:00:00 2001
From: janetvong 
Date: Thu, 23 Mar 2023 11:17:28 -0700
Subject: [PATCH 56/56] Add Python 3 version of Module 15 sample.

---
 mod15b-blobstore/README.md            |  8 +++
 mod15b-blobstore/app.yaml             |  2 +
 mod15b-blobstore/main.py              | 82 +++++++++++++++++++++++++++
 mod15b-blobstore/requirements.txt     |  2 +
 mod15b-blobstore/templates/index.html | 38 +++++++++++++
 5 files changed, 132 insertions(+)
 create mode 100644 mod15b-blobstore/README.md
 create mode 100644 mod15b-blobstore/app.yaml
 create mode 100644 mod15b-blobstore/main.py
 create mode 100644 mod15b-blobstore/requirements.txt
 create mode 100644 mod15b-blobstore/templates/index.html

diff --git a/mod15b-blobstore/README.md b/mod15b-blobstore/README.md
new file mode 100644
index 0000000..94b30ff
--- /dev/null
+++ b/mod15b-blobstore/README.md
@@ -0,0 +1,8 @@
+# Module 15b - Usage of App Engine `blobstore` with Flask framework in Python 3
+
+This repo folder is the corresponding Python 3 version of the Module 15 app.
+
+- All files in this folder are identical to the _Python 2_ code in the [Module 15 repo folder](/mod15-blobstore) **except**:
+    1. `app.yaml` was modified for the Python 3 runtime.
+    1. `appengine_config.py` is unused and thus deleted.
+- The _Python 3_ version of the Module 15 app ([Module 15 repo folder](/mod15-blobstore)) features the use of [Blobstore handlers classes](https://cloud.google.com/appengine/docs/standard/python3/services/blobstore).
\ No newline at end of file
diff --git a/mod15b-blobstore/app.yaml b/mod15b-blobstore/app.yaml
new file mode 100644
index 0000000..c623187
--- /dev/null
+++ b/mod15b-blobstore/app.yaml
@@ -0,0 +1,2 @@
+runtime: python310
+app_engine_apis: true
diff --git a/mod15b-blobstore/main.py b/mod15b-blobstore/main.py
new file mode 100644
index 0000000..fb2d633
--- /dev/null
+++ b/mod15b-blobstore/main.py
@@ -0,0 +1,82 @@
+# Copyright 2022 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import io
+from flask import Flask, abort, redirect, request
+from google.appengine.api import wrap_wsgi_app
+from google.appengine.ext import blobstore, ndb
+
+app = Flask(__name__)
+app.wsgi_app = wrap_wsgi_app(app.wsgi_app, use_deferred=True)
+
+
+class Visit(ndb.Model):
+    'Visit entity registers visitor IP address & timestamp'
+    visitor   = ndb.StringProperty()
+    timestamp = ndb.DateTimeProperty(auto_now_add=True)
+    blob_key = ndb.BlobKeyProperty()
+
+
+def store_visit(remote_addr, user_agent, upload_key):
+    'create new Visit entity in Datastore'
+    Visit(visitor='{}: {}'.format(remote_addr, user_agent),
+            file_blob = upload_key).put()
+
+
+def fetch_visits(limit):
+    'get most recent visits'
+    return Visit.query().order(-Visit.timestamp).fetch(limit)
+
+
+class UploadHandler(blobstore.BlobstoreUploadHandler):
+    'Upload blob (POST) handler'
+    def post(self):
+        uploads = self.get_uploads(request.environ)
+        blob_id = uploads[0].key() if uploads else None
+        store_visit(self.request.remote_addr, self.request.user_agent, blob_id)
+        return redirect('/', code=307)
+
+class ViewBlobHandler(blobstore.BlobstoreDownloadHandler):
+    'view uploaded blob (GET) handler'
+    def get(self, blob_key):
+        if not blobstore.get(blob_key):
+            return "Blobg key not found", 404
+        else:
+            headers = self.send_blob(request.environ, blob_key) 
+
+        # Prevent Flask from setting a default content-type.
+        # GAE sets it to a guessed type if the header is not set.
+            headers['Content-Type'] = None
+            return '', headers
+
+@app.route('/view_photo/')
+def view_photo(photo_key):
+    """View photo given a key."""
+    return ViewBlobHandler().get(photo_key)
+
+
+@app.route('/upload_photo', methods=['POST'])
+def upload_photo():
+    """Upload handler called by blobstore when a blob is uploaded in the test."""
+    return UploadHandler().post()
+
+@app.route('/', methods=['GET', 'POST'])
+def root():
+    'main application (GET/POST) handler'
+    context = {}
+    if request.method == 'GET':
+        context['upload_url'] = url_for('upload')
+    else:
+        context['visits'] = fetch_visits(10)
+    return render_template('index.html', **context)
\ No newline at end of file
diff --git a/mod15b-blobstore/requirements.txt b/mod15b-blobstore/requirements.txt
new file mode 100644
index 0000000..39d910c
--- /dev/null
+++ b/mod15b-blobstore/requirements.txt
@@ -0,0 +1,2 @@
+Flask==2.0.1
+appengine-python-standard>=1.0.0
\ No newline at end of file
diff --git a/mod15b-blobstore/templates/index.html b/mod15b-blobstore/templates/index.html
new file mode 100644
index 0000000..c6376a1
--- /dev/null
+++ b/mod15b-blobstore/templates/index.html
@@ -0,0 +1,38 @@
+
+
+
+VisitMe Example
+
+
+
+

VisitMe example

+{% if upload_url %} + +

Welcome... upload a file? (optional)

+ +

+ + + +{% else %} + +

Last 10 visits

+
    +{% for visit in visits %} +
  • {{ visit.timestamp.ctime() }} + + {% if visit.file_blob %} + (view) + {% else %} + (none) + {% endif %} + + from {{ visit.visitor }} +
  • +{% endfor %} +
+ +{% endif %} + + + \ No newline at end of file