Skip to content

Commit 7878d5c

Browse files
David Cavazosrenovate-botleahecoleengelke
authored
update tensorflow and other dependencies (GoogleCloudPlatform#9827)
* chore(deps): update dependency tensorflow to v2.12.0 * Apply suggestions from code review * Apply suggestions from code review * Update other dependency versions * Update Dockerfile * Update Dockerfile * Update Dockerfile --------- Co-authored-by: Renovate Bot <[email protected]> Co-authored-by: Leah E. Cole <[email protected]> Co-authored-by: Charles Engelke <[email protected]>
1 parent 992f84e commit 7878d5c

File tree

26 files changed

+88
-107
lines changed

26 files changed

+88
-107
lines changed

‎dataflow/gpu-examples/tensorflow-landsat-prime/Dockerfile‎

Lines changed: 12 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -12,30 +12,28 @@
1212
# See the License for the specific language governing permissions and
1313
# limitations under the License.
1414

15-
# Each version of TensorFlow requires a specific CUDA/cuDNN version:
16-
# https://www.tensorflow.org/install/source#gpu
17-
# For a list of all the nvidia images:
18-
# https://ngc.nvidia.com/catalog/containers/nvidia:cuda/tags
19-
FROM nvcr.io/nvidia/cuda:11.2.2-cudnn8-runtime-ubuntu20.04
15+
# NVIDIA CUDA container: https://catalog.ngc.nvidia.com/orgs/nvidia/containers/cuda
16+
# Supported NVIDIA images: https://gitlab.com/nvidia/container-images/cuda/blob/master/doc/supported-tags.md
17+
# TensorFlow/CUDA compatibility: https://www.tensorflow.org/install/source#gpu
18+
FROM nvcr.io/nvidia/cuda:11.8.0-cudnn8-runtime-ubuntu22.04
2019

2120
WORKDIR /pipeline
2221

2322
COPY requirements.txt .
2423
COPY *.py ./
2524

26-
# If you need a different Python version, consider:
27-
# https://launchpad.net/~deadsnakes/+archive/ubuntu/ppa
2825
RUN apt-get update \
29-
&& apt-get install -y --no-install-recommends curl g++ python3.8-dev python3-distutils \
26+
# Install Python and other system dependencies.
27+
&& apt-get install -y --no-install-recommends \
28+
curl g++ python3.11-dev python3-pip python3-distutils \
3029
&& rm -rf /var/lib/apt/lists/* \
31-
&& update-alternatives --install /usr/bin/python python /usr/bin/python3.8 10 \
32-
&& curl https://bootstrap.pypa.io/get-pip.py | python \
33-
# Install the pipeline requirements and check that there are no conflicts.
34-
# Since the image already has all the dependencies installed,
35-
# there's no need to run with the --requirements_file option.
30+
&& update-alternatives --install /usr/bin/python python /usr/bin/python3.11 10 \
31+
# Install the pipeline requirements.
32+
&& pip install --no-cache-dir --upgrade pip \
3633
&& pip install --no-cache-dir -r requirements.txt \
3734
&& pip check
3835

3936
# Set the entrypoint to Apache Beam SDK worker launcher.
40-
COPY --from=apache/beam_python3.8_sdk:2.38.0 /opt/apache/beam /opt/apache/beam
37+
# Check this matches the apache-beam version in the requirements.txt
38+
COPY --from=apache/beam_python3.11_sdk:2.46.0 /opt/apache/beam /opt/apache/beam
4139
ENTRYPOINT [ "/opt/apache/beam/boot" ]

‎dataflow/gpu-examples/tensorflow-landsat-prime/noxfile_config.py‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -22,10 +22,10 @@
2222

2323
TEST_CONFIG_OVERRIDE = {
2424
# You can opt out from the test for specific Python versions.
25-
# > ℹ️ We're opting out of all Python versions except 3.8.
25+
# > ℹ️ Test only on Python 3.11.
2626
# > The Python version used is defined by the Dockerfile, so it's redundant
2727
# > to run multiple tests since they would all be running the same Dockerfile.
28-
"ignored_versions": ["2.7", "3.6", "3.7", "3.9", "3.10", "3.11"],
28+
"ignored_versions": ["2.7", "3.6", "3.7", "3.8", "3.9", "3.10"],
2929
# Old samples are opted out of enforcing Python type hints
3030
# All new samples should feature them
3131
"enforce_type_hints": True,

‎dataflow/gpu-examples/tensorflow-landsat-prime/requirements-test.txt‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,3 @@
11
google-api-python-client==2.86.0
22
google-cloud-storage==2.8.0
3-
pytest==7.0.1
3+
pytest==7.3.1

‎dataflow/gpu-examples/tensorflow-landsat-prime/requirements.txt‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
11
Pillow==9.5.0
2-
apache-beam[gcp]==2.35.0
2+
apache-beam[gcp]==2.46.0
33
rasterio==1.3.6
4-
tensorflow==2.9.3
4+
tensorflow==2.12.0 # Check TensorFlow/CUDA compatibility with Dockerfile: https://www.tensorflow.org/install/source#gpu

‎dataflow/gpu-examples/tensorflow-landsat/Dockerfile‎

Lines changed: 12 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -12,30 +12,28 @@
1212
# See the License for the specific language governing permissions and
1313
# limitations under the License.
1414

15-
# Each version of TensorFlow requires a specific CUDA/cuDNN version:
16-
# https://www.tensorflow.org/install/source#gpu
17-
# For a list of all the nvidia images:
18-
# https://ngc.nvidia.com/catalog/containers/nvidia:cuda/tags
19-
FROM nvcr.io/nvidia/cuda:11.2.2-cudnn8-runtime-ubuntu20.04
15+
# NVIDIA CUDA container: https://catalog.ngc.nvidia.com/orgs/nvidia/containers/cuda
16+
# Supported NVIDIA images: https://gitlab.com/nvidia/container-images/cuda/blob/master/doc/supported-tags.md
17+
# TensorFlow/CUDA compatibility: https://www.tensorflow.org/install/source#gpu
18+
FROM nvcr.io/nvidia/cuda:11.8.0-cudnn8-runtime-ubuntu22.04
2019

2120
WORKDIR /pipeline
2221

2322
COPY requirements.txt .
2423
COPY *.py ./
2524

26-
# If you need a different Python version, consider:
27-
# https://launchpad.net/~deadsnakes/+archive/ubuntu/ppa
2825
RUN apt-get update \
29-
&& apt-get install -y --no-install-recommends curl g++ python3.8-dev python3-distutils \
26+
# Install Python and other system dependencies.
27+
&& apt-get install -y --no-install-recommends \
28+
curl g++ python3.11-dev python3-pip python3-distutils \
3029
&& rm -rf /var/lib/apt/lists/* \
31-
&& update-alternatives --install /usr/bin/python python /usr/bin/python3.8 10 \
32-
&& curl https://bootstrap.pypa.io/get-pip.py | python \
33-
# Install the pipeline requirements and check that there are no conflicts.
34-
# Since the image already has all the dependencies installed,
35-
# there's no need to run with the --requirements_file option.
30+
&& update-alternatives --install /usr/bin/python python /usr/bin/python3.11 10 \
31+
# Install the pipeline requirements.
32+
&& pip install --no-cache-dir --upgrade pip \
3633
&& pip install --no-cache-dir -r requirements.txt \
3734
&& pip check
3835

3936
# Set the entrypoint to Apache Beam SDK worker launcher.
40-
COPY --from=apache/beam_python3.8_sdk:2.38.0 /opt/apache/beam /opt/apache/beam
37+
# Check this matches the apache-beam version in the requirements.txt
38+
COPY --from=apache/beam_python3.11_sdk:2.46.0 /opt/apache/beam /opt/apache/beam
4139
ENTRYPOINT [ "/opt/apache/beam/boot" ]

‎dataflow/gpu-examples/tensorflow-landsat/noxfile_config.py‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -22,10 +22,10 @@
2222

2323
TEST_CONFIG_OVERRIDE = {
2424
# You can opt out from the test for specific Python versions.
25-
# > ℹ️ We're opting out of all Python versions except 3.8.
25+
# > ℹ️ Test only on Python 3.11.
2626
# > The Python version used is defined by the Dockerfile, so it's redundant
2727
# > to run multiple tests since they would all be running the same Dockerfile.
28-
"ignored_versions": ["2.7", "3.6", "3.7", "3.9", "3.10", "3.11"],
28+
"ignored_versions": ["2.7", "3.6", "3.7", "3.8", "3.9", "3.10"],
2929
# Old samples are opted out of enforcing Python type hints
3030
# All new samples should feature them
3131
"enforce_type_hints": True,

‎dataflow/gpu-examples/tensorflow-landsat/requirements-test.txt‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,3 @@
1-
pytest==7.0.1
21
google-api-python-client==2.86.0
32
google-cloud-storage==2.8.0
3+
pytest==7.3.1

‎dataflow/gpu-examples/tensorflow-landsat/requirements.txt‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
pillow==9.5.0
2-
apache-beam[gcp]==2.35.0
1+
Pillow==9.5.0
2+
apache-beam[gcp]==2.46.0
33
rasterio==1.3.6
4-
tensorflow==2.9.3
4+
tensorflow==2.12.0 # Check TensorFlow/CUDA compatibility with Dockerfile: https://www.tensorflow.org/install/source#gpu

‎dataflow/gpu-examples/tensorflow-minimal/Dockerfile‎

Lines changed: 13 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
# Copyright 2021 Google LLC
1+
# Copyright 2020 Google LLC
22
#
33
# Licensed under the Apache License, Version 2.0 (the "License");
44
# you may not use this file except in compliance with the License.
@@ -12,30 +12,28 @@
1212
# See the License for the specific language governing permissions and
1313
# limitations under the License.
1414

15-
# Each version of TensorFlow requires a specific CUDA/cuDNN version:
16-
# https://www.tensorflow.org/install/source#gpu
17-
# For a list of all the nvidia images:
18-
# https://ngc.nvidia.com/catalog/containers/nvidia:cuda/tags
19-
FROM nvcr.io/nvidia/cuda:11.2.2-cudnn8-runtime-ubuntu20.04
15+
# NVIDIA CUDA container: https://catalog.ngc.nvidia.com/orgs/nvidia/containers/cuda
16+
# Supported NVIDIA images: https://gitlab.com/nvidia/container-images/cuda/blob/master/doc/supported-tags.md
17+
# TensorFlow/CUDA compatibility: https://www.tensorflow.org/install/source#gpu
18+
FROM nvcr.io/nvidia/cuda:11.8.0-cudnn8-runtime-ubuntu22.04
2019

2120
WORKDIR /pipeline
2221

2322
COPY requirements.txt .
2423
COPY *.py ./
2524

26-
# If you need a different Python version, consider:
27-
# https://launchpad.net/~deadsnakes/+archive/ubuntu/ppa
2825
RUN apt-get update \
29-
&& apt-get install -y --no-install-recommends curl g++ python3.8-dev python3-distutils \
26+
# Install Python and other system dependencies.
27+
&& apt-get install -y --no-install-recommends \
28+
curl g++ python3.11-dev python3-pip python3-distutils \
3029
&& rm -rf /var/lib/apt/lists/* \
31-
&& update-alternatives --install /usr/bin/python python /usr/bin/python3.8 10 \
32-
&& curl https://bootstrap.pypa.io/get-pip.py | python \
33-
# Install the pipeline requirements and check that there are no conflicts.
34-
# Since the image already has all the dependencies installed,
35-
# there's no need to run with the --requirements_file option.
30+
&& update-alternatives --install /usr/bin/python python /usr/bin/python3.11 10 \
31+
# Install the pipeline requirements.
32+
&& pip install --no-cache-dir --upgrade pip \
3633
&& pip install --no-cache-dir -r requirements.txt \
3734
&& pip check
3835

3936
# Set the entrypoint to Apache Beam SDK worker launcher.
40-
COPY --from=apache/beam_python3.8_sdk:2.39.0 /opt/apache/beam /opt/apache/beam
37+
# Check this matches the apache-beam version in the requirements.txt
38+
COPY --from=apache/beam_python3.11_sdk:2.46.0 /opt/apache/beam /opt/apache/beam
4139
ENTRYPOINT [ "/opt/apache/beam/boot" ]

‎dataflow/gpu-examples/tensorflow-minimal/noxfile_config.py‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -22,10 +22,10 @@
2222

2323
TEST_CONFIG_OVERRIDE = {
2424
# You can opt out from the test for specific Python versions.
25-
# > ℹ️ We're opting out of all Python versions except 3.8.
25+
# > ℹ️ Test only on Python 3.11.
2626
# > The Python version used is defined by the Dockerfile, so it's redundant
2727
# > to run multiple tests since they would all be running the same Dockerfile.
28-
"ignored_versions": ["2.7", "3.6", "3.7", "3.9", "3.10", "3.11"],
28+
"ignored_versions": ["2.7", "3.6", "3.7", "3.8", "3.9", "3.10"],
2929
# Old samples are opted out of enforcing Python type hints
3030
# All new samples should feature them
3131
"enforce_type_hints": True,

0 commit comments

Comments
 (0)