![\"TensorFlow](\"https://codelabs.developers.google.com/codelabs/cloud-tensorflow-mnist/img/93d5f08a4f82d4c.png/")
\n",
+ "\n",
+ "Дополнительно:![\"\n",](\"https://codelabs.developers.google.com/codelabs/cloud-tensorflow-mnist/img/d5222c6e3d15770a.png/")
"
+ ]
+ },
+ {
+ "metadata": {
+ "id": "50ID5ZFXo_JR",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ " Сначала определим переменные и метапеременные.\n",
+ " - Переменные - все те параметры, котрые мы планируем обучать в процессе выполнения алгоритма - веса и смещения.\n",
+ " - Метапеременные - параметры, которые будут заполнены конкретными данными во время обучения - входные данные.\n"
+ ]
+ },
+ {
+ "metadata": {
+ "id": "vG8jFjLNo_JT",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "cell_type": "code",
+ "source": [
+ "image_size = 28\n",
+ "n_classes = 10\n",
+ "\n",
+ "tf.reset_default_graph()\n",
+ "tf.set_random_seed(0)\n",
+ "\n",
+ "#мета для входных картинок\n",
+ "X = tf.placeholder(tf.float32, [None, image_size*image_size], name='train_inputs')\n",
+ "# мета для правильных ответов\n",
+ "Y_ = tf.placeholder(tf.float32, [None, 10], name='train_labels')"
+ ],
+ "execution_count": 0,
+ "outputs": []
+ },
+ {
+ "metadata": {
+ "id": "lVOZaPJ6o_JW",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "Форма тензоров для метапеременных определяется размером картинок 28х28 для X и количеством классов (цифры от 0 до 9) для Y. None - эта размерность указывает на количество образцов в пакете данных."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "MEy1Njyno_JZ",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "## 1.1 Модель"
+ ]
+ },
+ {
+ "metadata": {
+ "id": "jkQN5kbbo_Je",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "![\"drawing\"](\"https://codelabs.developers.google.com/codelabs/cloud-tensorflow-mnist/img/21dabcf6d44e4d6f.png/")
"
+ ]
+ },
+ {
+ "metadata": {
+ "id": "MwoiDFLvo_Jf",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "cell_type": "code",
+ "source": [
+ "#картинка с нейроном\n",
+ "with tf.name_scope('out'):\n",
+ " with tf.name_scope('weights'):\n",
+ " weights = tf.Variable(tf.zeros([image_size*image_size, n_classes]))\n",
+ " with tf.name_scope('biases'):\n",
+ " biases = tf.Variable(tf.zeros([n_classes]))\n",
+ " with tf.name_scope('xw_plus_b'):\n",
+ " Y = tf.nn.softmax(tf.matmul(X, weights) + biases)"
+ ],
+ "execution_count": 0,
+ "outputs": []
+ },
+ {
+ "metadata": {
+ "id": "8ri3QVszo_Jl",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "Сначала задаём веса и смещения. Далее, суммируем с весами 28*28 входных нейронов, добавляем смещения и складываем в переменную - это предсказания сети:\n",
+ "\n",
+ "![\"softmax\"](\"https://codelabs.developers.google.com/codelabs/cloud-tensorflow-mnist/img/604a9797da2a48d7.png/")
\n",
+ "\n",
+ "![\"softmax\"](\"https://codelabs.developers.google.com/codelabs/cloud-tensorflow-mnist/img/206327168bc85294.png/")
\n"
+ ]
+ },
+ {
+ "metadata": {
+ "id": "JGDhtKpjo_Jn",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "## 1.2 Функция потерь"
+ ]
+ },
+ {
+ "metadata": {
+ "id": "tHiGGLSJo_Jn",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "![\"drawing\"](\"https://codelabs.developers.google.com/codelabs/cloud-tensorflow-mnist/img/1d8fc59e6a674f1c.png/")
"
+ ]
+ },
+ {
+ "metadata": {
+ "id": "o5a2j9Ovo_Jp",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "cell_type": "code",
+ "source": [
+ "with tf.name_scope('cross_entropy'):\n",
+ " with tf.name_scope('total'):\n",
+ " cross_entropy = - tf.reduce_sum(Y_ * tf.log(Y))"
+ ],
+ "execution_count": 0,
+ "outputs": []
+ },
+ {
+ "metadata": {
+ "id": "1W8tZ9ovo_Jr",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "Теперь, когда мы знаем предсказания, нужно понять - насколько они хороши? Надо как-то измерить расстояние между правильными и предсказанными ответами. В качестве расстояния возьмём кросс-энтропию.\n",
+ "\n"
+ ]
+ },
+ {
+ "metadata": {
+ "id": "ZROzSPwYo_Jw",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "## 1.3 Оптимизатор"
+ ]
+ },
+ {
+ "metadata": {
+ "id": "_Mh8AEGVo_Jx",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "![\"drawing\"](\"https://codelabs.developers.google.com/codelabs/cloud-tensorflow-mnist/img/34e9e76c7715b719.png/")
"
+ ]
+ },
+ {
+ "metadata": {
+ "id": "ibP4j3k3o_J1",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "cell_type": "code",
+ "source": [
+ "learning_rate = 0.003\n",
+ "with tf.name_scope('train'):\n",
+ " optimizer = tf.train.GradientDescentOptimizer(learning_rate)\n",
+ " train_step = optimizer.minimize(cross_entropy)"
+ ],
+ "execution_count": 0,
+ "outputs": []
+ },
+ {
+ "metadata": {
+ "id": "s53-MxK5o_J7",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "Полученную метрику и будем минимизировать. Для этого используем стандартный градиентный спуск. Темп обучения задаёт шаг вдоль вектора градиента. Чтобы не перескочить минимум, берём маленький шаг."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "xNQmbW7oo_J9",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "## 1.4 Отчёты для TensorBoard"
+ ]
+ },
+ {
+ "metadata": {
+ "id": "jekNNULHo_J-",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "Чтобы получить графики для процесса обучения и результаты проверки на тестовых данных, создаём различные виды отчётов - summary. Те показатели, которые мы укажем сериализуюстся в процессе выполнения сессии и TensorBoard сможет их считать."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "lyhnJsUbo_KC",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "cell_type": "code",
+ "source": [
+ "tf.summary.scalar('cross_entropy', cross_entropy)\n",
+ "\n",
+ "with tf.name_scope('accuracy'):\n",
+ " with tf.name_scope('correct_prediction'):\n",
+ " correct_prediction = tf.equal(tf.argmax(Y, axis=1), tf.argmax(Y_, axis=1))\n",
+ " with tf.name_scope('accuracy'):\n",
+ " accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))\n",
+ "tf.summary.scalar('accuracy', accuracy)\n",
+ "\n",
+ "merged = tf.summary.merge_all()"
+ ],
+ "execution_count": 0,
+ "outputs": []
+ },
+ {
+ "metadata": {
+ "id": "juSpTY0Ho_KL",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "## 1.5 Запускаем сеть"
+ ]
+ },
+ {
+ "metadata": {
+ "id": "Ffq8YFDKo_KN",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "cell_type": "code",
+ "source": [
+ "def get_summary_path(sub_folder):\n",
+ " import os\n",
+ " summ_folder = LOG_DIR#'summaries'\n",
+ " \n",
+ " summary_path = os.path.join(summ_folder, sub_folder)\n",
+ " if not os.path.exists(summ_folder):\n",
+ " os.mkdir(summ_folder)\n",
+ " if not os.path.exists(summary_path):\n",
+ " os.mkdir(summary_path)\n",
+ " \n",
+ " return summary_path"
+ ],
+ "execution_count": 0,
+ "outputs": []
+ },
+ {
+ "metadata": {
+ "id": "ET8f1YsBo_KR",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "Всё, что мы до этого момента делали - лишь конструктор графа. Ничего на расчёт не запускалось. Время проверить, всё ли правильно было сделано.\n",
+ "\n",
+ "Первым делом надо объявить сессию!\n",
+ "\n",
+ "Задаём объекты для сериализации - отдельно для обучения, отдельно для теста.\n",
+ "\n",
+ "В цикле с помощью train_step, который минимизирует кросс-энтропию, обновляются веса и смещения и каждые 10 шагов замеряем точность на тестовых данных. Только замеряем!"
+ ]
+ },
+ {
+ "metadata": {
+ "id": "UPU_V2ZLo_Kc",
+ "colab_type": "code",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 395
+ },
+ "outputId": "48637e47-2283-4fee-e7ef-a5eb9c6fed6d"
+ },
+ "cell_type": "code",
+ "source": [
+ "batch_size = 100\n",
+ "max_step = 2001\n",
+ "\n",
+ "session = tf.InteractiveSession()\n",
+ "\n",
+ "summary_path = get_summary_path('nn_1l')\n",
+ "\n",
+ "train_writer = tf.summary.FileWriter(summary_path + '/train', session.graph)\n",
+ "test_writer = tf.summary.FileWriter(summary_path + '/test')\n",
+ "\n",
+ "tf.global_variables_initializer().run()\n",
+ "\n",
+ "for i in range(max_step):\n",
+ " if i % 10 == 0:\n",
+ " xs, ys = mnist.test.images, mnist.test.labels\n",
+ " summary, acc = session.run([merged, accuracy], feed_dict={X: xs, Y_: ys})\n",
+ " test_writer.add_summary(summary, i)\n",
+ " if i % 100 == 0:\n",
+ " print('Accuracy at step %s: %s' % (i, acc))\n",
+ " else:\n",
+ " xs, ys = mnist.train.next_batch(batch_size)\n",
+ " summary, _ = session.run([merged, train_step], feed_dict={X: xs, Y_: ys})\n",
+ " train_writer.add_summary(summary, i)\n",
+ "\n",
+ "session.close()"
+ ],
+ "execution_count": 14,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "text": [
+ "Accuracy at step 0: 0.098\n",
+ "Accuracy at step 100: 0.8863\n",
+ "Accuracy at step 200: 0.9013\n",
+ "Accuracy at step 300: 0.9054\n",
+ "Accuracy at step 400: 0.9071\n",
+ "Accuracy at step 500: 0.9074\n",
+ "Accuracy at step 600: 0.9109\n",
+ "Accuracy at step 700: 0.9118\n",
+ "Accuracy at step 800: 0.9153\n",
+ "Accuracy at step 900: 0.9147\n",
+ "Accuracy at step 1000: 0.9147\n",
+ "Accuracy at step 1100: 0.9143\n",
+ "Accuracy at step 1200: 0.917\n",
+ "Accuracy at step 1300: 0.9177\n",
+ "Accuracy at step 1400: 0.9159\n",
+ "Accuracy at step 1500: 0.9194\n",
+ "Accuracy at step 1600: 0.9211\n",
+ "Accuracy at step 1700: 0.9203\n",
+ "Accuracy at step 1800: 0.9206\n",
+ "Accuracy at step 1900: 0.9215\n",
+ "Accuracy at step 2000: 0.9212\n"
+ ],
+ "name": "stdout"
+ }
+ ]
+ },
+ {
+ "metadata": {
+ "id": "z4KzpHqVo_Kh",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# 2. Добавим слои\n",
+ "Чтобы улучшить распознавательную способность сети, нам понадобится добавить скрытые слои.\n",
+ "\n",
+ "Дополнительно, некоторые вещи спихнём в функции, что заметно облегчит жизнь при модификации сети."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "5NvU6Aaoo_Ki",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "В качестве функций активации будем использовать классическую сигмоиду:\n",
+ "\n",
+ "![\"drawing\"](\"https://codelabs.developers.google.com/codelabs/cloud-tensorflow-mnist/img/77bc41f211c9fb29.png/")
"
+ ]
+ },
+ {
+ "metadata": {
+ "id": "S81dJik7o_Kj",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "cell_type": "code",
+ "source": [
+ "image_size = 28\n",
+ "n_classes = 10\n",
+ "\n",
+ "tf.reset_default_graph() #сбросим граф\n",
+ "\n",
+ "#мета для входных картинок\n",
+ "X = tf.placeholder(tf.float32, [None, image_size*image_size], name='train_inputs')\n",
+ "# мета для правильных ответов\n",
+ "Y_ = tf.placeholder(tf.float32, [None, 10], name='train_labels')"
+ ],
+ "execution_count": 0,
+ "outputs": []
+ },
+ {
+ "metadata": {
+ "id": "LeF2UdIho_Kt",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "cell_type": "code",
+ "source": [
+ "def variable_summaries(var):\n",
+ " with tf.name_scope('summaries'):\n",
+ " mean = tf.reduce_mean(var)\n",
+ " tf.summary.scalar('mean', mean)\n",
+ " with tf.name_scope('stddev'):\n",
+ " stddev = tf.sqrt(tf.reduce_mean(tf.square(var - mean)))\n",
+ " tf.summary.scalar('stddev', stddev)\n",
+ " tf.summary.scalar('max', tf.reduce_max(var))\n",
+ " tf.summary.scalar('min', tf.reduce_min(var))\n",
+ " tf.summary.histogram('histogram', var)\n",
+ "\n",
+ "def nn_layer(input_tensor, input_dim, output_dim, layer_name, activation):\n",
+ " with tf.name_scope(layer_name):\n",
+ " with tf.name_scope('weights'):\n",
+ " initial = tf.truncated_normal([input_dim, output_dim], stddev=0.1)\n",
+ " weights = tf.Variable(initial)\n",
+ " variable_summaries(weights)\n",
+ " with tf.name_scope('biases'):\n",
+ " initial = tf.truncated_normal([output_dim], stddev=0.1)\n",
+ " biases = tf.Variable(initial)\n",
+ " variable_summaries(biases)\n",
+ " with tf.name_scope('xw_plus_b'):\n",
+ " preactivate = tf.nn.xw_plus_b(input_tensor, weights, biases)\n",
+ " activations = activation(preactivate, name='activation')\n",
+ " \n",
+ " return activations "
+ ],
+ "execution_count": 0,
+ "outputs": []
+ },
+ {
+ "metadata": {
+ "id": "nojI3FX6o_Kv",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "cell_type": "code",
+ "source": [
+ "hidden1 = nn_layer(X, image_size*image_size, 200, 'hidden1', tf.nn.sigmoid)\n",
+ "hidden2 = nn_layer(hidden1, 200, 100, 'hidden2', tf.nn.sigmoid)\n",
+ "hidden3 = nn_layer(hidden2, 100, 60, 'hidden3', tf.nn.sigmoid)\n",
+ "hidden4 = nn_layer(hidden3, 60, 30, 'hidden4', tf.nn.sigmoid)\n",
+ "\n",
+ "Y = nn_layer(hidden4, 30, n_classes, 'out', tf.nn.softmax)"
+ ],
+ "execution_count": 0,
+ "outputs": []
+ },
+ {
+ "metadata": {
+ "id": "XUNObazAo_K9",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "В качестве функций активации будем использовать классическую сигмоиду:\n",
+ "\n",
+ "![\"drawing\"](\"https://codelabs.developers.google.com/codelabs/cloud-tensorflow-mnist/img/e5d46c389470df62.png/")
"
+ ]
+ },
+ {
+ "metadata": {
+ "id": "XmCGJyIto_K-",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "cell_type": "code",
+ "source": [
+ "#БЕЗ ИЗМЕНЕНИЙ\n",
+ "# функция потерь\n",
+ "with tf.name_scope('cross_entropy'):\n",
+ " with tf.name_scope('total'):\n",
+ " cross_entropy = - tf.reduce_sum(Y_ * tf.log(Y))"
+ ],
+ "execution_count": 0,
+ "outputs": []
+ },
+ {
+ "metadata": {
+ "id": "yIzsbYQ7o_LD",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "cell_type": "code",
+ "source": [
+ "#БЕЗ ИЗМЕНЕНИЙ\n",
+ "# Optimizer\n",
+ "learning_rate = 0.003\n",
+ "with tf.name_scope('train'):\n",
+ " optimizer = tf.train.GradientDescentOptimizer(learning_rate)\n",
+ " train_step = optimizer.minimize(cross_entropy)"
+ ],
+ "execution_count": 0,
+ "outputs": []
+ },
+ {
+ "metadata": {
+ "id": "Q8z2vCgio_LG",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "cell_type": "code",
+ "source": [
+ "#БЕЗ ИЗМЕНЕНИЙ\n",
+ "tf.summary.scalar('cross_entropy', cross_entropy)\n",
+ "\n",
+ "with tf.name_scope('accuracy'):\n",
+ " with tf.name_scope('correct_prediction'):\n",
+ " correct_prediction = tf.equal(tf.argmax(Y, axis=1), tf.argmax(Y_, axis=1))\n",
+ " with tf.name_scope('accuracy'):\n",
+ " accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))\n",
+ "tf.summary.scalar('accuracy', accuracy)\n",
+ "\n",
+ "# Merge all summaries together\n",
+ "merged = tf.summary.merge_all()"
+ ],
+ "execution_count": 0,
+ "outputs": []
+ },
+ {
+ "metadata": {
+ "id": "uKjX7swGo_LM",
+ "colab_type": "code",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 451
+ },
+ "outputId": "61902a1c-0afc-4f60-b69b-9ce166f5f125"
+ },
+ "cell_type": "code",
+ "source": [
+ "max_step=2001\n",
+ "batch_size=100\n",
+ "\n",
+ "session = tf.InteractiveSession()\n",
+ "\n",
+ "summary_path = get_summary_path('nn_4hid')\n",
+ "\n",
+ "train_writer = tf.summary.FileWriter(summary_path + '/train', session.graph)\n",
+ "test_writer = tf.summary.FileWriter(summary_path + '/test')\n",
+ "\n",
+ "tf.global_variables_initializer().run()\n",
+ "\n",
+ "for i in range(max_step):\n",
+ " if i % 10 == 0:\n",
+ " xs, ys = mnist.test.images, mnist.test.labels\n",
+ " summary, acc = session.run([merged, accuracy], feed_dict={X: xs, Y_: ys})\n",
+ " test_writer.add_summary(summary, i)\n",
+ " if i % 100 == 0:\n",
+ " print('Accuracy at step %s: %s' % (i, acc))\n",
+ " else:\n",
+ " xs, ys = mnist.train.next_batch(batch_size)\n",
+ " summary, _ = session.run([merged, train_step], feed_dict={X: xs, Y_: ys})\n",
+ " train_writer.add_summary(summary, i)\n",
+ "\n",
+ "session.close()"
+ ],
+ "execution_count": 15,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "text": [
+ "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py:1662: UserWarning: An interactive session is already active. This can cause out-of-memory errors in some cases. You must explicitly call `InteractiveSession.close()` to release resources held by the other session(s).\n",
+ " warnings.warn('An interactive session is already active. This can '\n"
+ ],
+ "name": "stderr"
+ },
+ {
+ "output_type": "stream",
+ "text": [
+ "Accuracy at step 0: 0.098\n",
+ "Accuracy at step 100: 0.1032\n",
+ "Accuracy at step 200: 0.1032\n",
+ "Accuracy at step 300: 0.1135\n",
+ "Accuracy at step 400: 0.1937\n",
+ "Accuracy at step 500: 0.1135\n",
+ "Accuracy at step 600: 0.1135\n",
+ "Accuracy at step 700: 0.1135\n",
+ "Accuracy at step 800: 0.098\n",
+ "Accuracy at step 900: 0.1439\n",
+ "Accuracy at step 1000: 0.1135\n",
+ "Accuracy at step 1100: 0.1135\n",
+ "Accuracy at step 1200: 0.0958\n",
+ "Accuracy at step 1300: 0.1135\n",
+ "Accuracy at step 1400: 0.0974\n",
+ "Accuracy at step 1500: 0.1032\n",
+ "Accuracy at step 1600: 0.1028\n",
+ "Accuracy at step 1700: 0.0958\n",
+ "Accuracy at step 1800: 0.1966\n",
+ "Accuracy at step 1900: 0.2674\n",
+ "Accuracy at step 2000: 0.2894\n"
+ ],
+ "name": "stdout"
+ }
+ ]
+ },
+ {
+ "metadata": {
+ "id": "S2ts4pD4o_LX",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "### упс! точность стала только хуже"
+ ]
+ },
+ {
+ "metadata": {
+ "id": "82-jhqyBo_LX",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# 3. Особенности глубоких сетей"
+ ]
+ },
+ {
+ "metadata": {
+ "id": "qw7dGNgKo_LX",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "### 3.1 Поменяем сигмоиду на RuLU\n",
+ "\n",
+ "Сигмоида, конечно, хорошая, но даёт плохие результаты в глубоком обучении. Это связано с тем, что она всю числовую ось сжимает в отрезок от 0 до 1ю Следовательно, при достаточно большом абсолютном значении входа, она будет выдавать всегда что-то близкое к 0 или к 1, и часть нейронов просто обнулится.\n",
+ "\n",
+ "В современных сетях часто используют ReLU:\n", + "
![\"drawing\"](\"https://codelabs.developers.google.com/codelabs/cloud-tensorflow-mnist/img/60cac06459b3cc08.png/")
"
+ ]
+ },
+ {
+ "metadata": {
+ "id": "enivQC_io_LZ",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "cell_type": "code",
+ "source": [
+ "image_size = 28\n",
+ "n_classes = 10\n",
+ "\n",
+ "tf.reset_default_graph() #сбросим граф\n",
+ "\n",
+ "#мета для входных картинок\n",
+ "X = tf.placeholder(tf.float32, [None, image_size*image_size], name='train_inputs')\n",
+ "# мета для правильных ответов\n",
+ "Y_ = tf.placeholder(tf.float32, [None, 10], name='train_labels')"
+ ],
+ "execution_count": 0,
+ "outputs": []
+ },
+ {
+ "metadata": {
+ "id": "5TSVxIYQo_Lf",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "cell_type": "code",
+ "source": [
+ "hidden1 = nn_layer(X, image_size*image_size, 200, 'hidden1', tf.nn.relu)\n",
+ "hidden2 = nn_layer(hidden1, 200, 100, 'hidden2', tf.nn.relu)\n",
+ "hidden3 = nn_layer(hidden2, 100, 60, 'hidden3', tf.nn.relu)\n",
+ "hidden4 = nn_layer(hidden3, 60, 30, 'hidden4', tf.nn.relu)\n",
+ "\n",
+ "Y = nn_layer(hidden4, 30, n_classes, 'out', tf.nn.softmax)"
+ ],
+ "execution_count": 0,
+ "outputs": []
+ },
+ {
+ "metadata": {
+ "id": "H1xHcctDo_Lk",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "МЕНЯЕМ ТУТ"
+ ]
+ },
+ {
+ "metadata": {
+ "id": "ltCxYZomo_Ll",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "cell_type": "code",
+ "source": [
+ "#БЕЗ ИЗМЕНЕНИЙ\n",
+ "# функция потерь\n",
+ "with tf.name_scope('cross_entropy'):\n",
+ " with tf.name_scope('total'):\n",
+ " cross_entropy = - tf.reduce_sum(Y_ * tf.log(Y))\n",
+ " \n",
+ "#БЕЗ ИЗМЕНЕНИЙ\n",
+ "# Optimizer\n",
+ "learning_rate = 0.003\n",
+ "with tf.name_scope('train'):\n",
+ " optimizer = tf.train.GradientDescentOptimizer(learning_rate)\n",
+ " train_step = optimizer.minimize(cross_entropy)"
+ ],
+ "execution_count": 0,
+ "outputs": []
+ },
+ {
+ "metadata": {
+ "id": "LUCwxpOno_Lx",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "cell_type": "code",
+ "source": [
+ "#БЕЗ ИЗМЕНЕНИЙ\n",
+ "tf.summary.scalar('cross_entropy', cross_entropy)\n",
+ "\n",
+ "with tf.name_scope('accuracy'):\n",
+ " with tf.name_scope('correct_prediction'):\n",
+ " correct_prediction = tf.equal(tf.argmax(Y, axis=1), tf.argmax(Y_, axis=1))\n",
+ " with tf.name_scope('accuracy'):\n",
+ " accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))\n",
+ "tf.summary.scalar('accuracy', accuracy)\n",
+ "\n",
+ "# Merge all summaries together\n",
+ "merged = tf.summary.merge_all()"
+ ],
+ "execution_count": 0,
+ "outputs": []
+ },
+ {
+ "metadata": {
+ "id": "u9PCFQJro_L3",
+ "colab_type": "code",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 451
+ },
+ "outputId": "93b26907-c619-4e85-f7f7-59c0bb383161"
+ },
+ "cell_type": "code",
+ "source": [
+ "max_step=2001\n",
+ "batch_size=100\n",
+ "\n",
+ "session = tf.InteractiveSession()\n",
+ "\n",
+ "summary_path = get_summary_path('nn_4hid_ReLU')\n",
+ "\n",
+ "train_writer = tf.summary.FileWriter(summary_path + '/train', session.graph)\n",
+ "test_writer = tf.summary.FileWriter(summary_path + '/test')\n",
+ "\n",
+ "tf.global_variables_initializer().run()\n",
+ "\n",
+ "for i in range(max_step):\n",
+ " if i % 10 == 0:\n",
+ " xs, ys = mnist.test.images, mnist.test.labels\n",
+ " summary, acc = session.run([merged, accuracy], feed_dict={X: xs, Y_: ys})\n",
+ " test_writer.add_summary(summary, i)\n",
+ " if i % 100 == 0:\n",
+ " print('Accuracy at step %s: %s' % (i, acc))\n",
+ " else:\n",
+ " xs, ys = mnist.train.next_batch(batch_size)\n",
+ " summary, _ = session.run([merged, train_step], feed_dict={X: xs, Y_: ys})\n",
+ " train_writer.add_summary(summary, i)\n",
+ "\n",
+ "session.close()"
+ ],
+ "execution_count": 20,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "text": [
+ "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py:1662: UserWarning: An interactive session is already active. This can cause out-of-memory errors in some cases. You must explicitly call `InteractiveSession.close()` to release resources held by the other session(s).\n",
+ " warnings.warn('An interactive session is already active. This can '\n"
+ ],
+ "name": "stderr"
+ },
+ {
+ "output_type": "stream",
+ "text": [
+ "Accuracy at step 0: 0.0987\n",
+ "Accuracy at step 100: 0.8521\n",
+ "Accuracy at step 200: 0.9075\n",
+ "Accuracy at step 300: 0.9312\n",
+ "Accuracy at step 400: 0.9408\n",
+ "Accuracy at step 500: 0.9192\n",
+ "Accuracy at step 600: 0.943\n",
+ "Accuracy at step 700: 0.952\n",
+ "Accuracy at step 800: 0.9442\n",
+ "Accuracy at step 900: 0.9625\n",
+ "Accuracy at step 1000: 0.951\n",
+ "Accuracy at step 1100: 0.953\n",
+ "Accuracy at step 1200: 0.9655\n",
+ "Accuracy at step 1300: 0.9686\n",
+ "Accuracy at step 1400: 0.9575\n",
+ "Accuracy at step 1500: 0.966\n",
+ "Accuracy at step 1600: 0.9709\n",
+ "Accuracy at step 1700: 0.9557\n",
+ "Accuracy at step 1800: 0.9605\n",
+ "Accuracy at step 1900: 0.9667\n",
+ "Accuracy at step 2000: 0.9664\n"
+ ],
+ "name": "stdout"
+ }
+ ]
+ },
+ {
+ "metadata": {
+ "id": "3iGaY3aCo_L6",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "### 3.2 Сменим оптимизатор\n",
+ "\n",
+ "В многомерных пространствах, как наше, частое явление - седловые точки - они не я вляются минимумами, но в них градиент равен 0. Поэтому градиентный спуск может в них засесть и дальше не двигаться. Поэтому заменим его на другой метод."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "oLEnrupxo_L7",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "cell_type": "code",
+ "source": [
+ "image_size = 28\n",
+ "n_classes = 10\n",
+ "\n",
+ "tf.reset_default_graph() #сбросим граф\n",
+ "\n",
+ "#мета для входных картинок\n",
+ "X = tf.placeholder(tf.float32, [None, image_size*image_size], name='train_inputs')\n",
+ "# мета для правильных ответов\n",
+ "Y_ = tf.placeholder(tf.float32, [None, 10], name='train_labels')"
+ ],
+ "execution_count": 0,
+ "outputs": []
+ },
+ {
+ "metadata": {
+ "id": "tXbfm3iWo_L-",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "cell_type": "code",
+ "source": [
+ "hidden1 = nn_layer(X, image_size*image_size, 200, 'hidden1', tf.nn.relu)\n",
+ "hidden2 = nn_layer(hidden1, 200, 100, 'hidden2', tf.nn.relu)\n",
+ "hidden3 = nn_layer(hidden2, 100, 60, 'hidden3', tf.nn.relu)\n",
+ "hidden4 = nn_layer(hidden3, 60, 30, 'hidden4', tf.nn.relu)\n",
+ "\n",
+ "Y = nn_layer(hidden4, 30, n_classes, 'out', tf.nn.softmax)"
+ ],
+ "execution_count": 0,
+ "outputs": []
+ },
+ {
+ "metadata": {
+ "id": "b9GpYeuzo_ME",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "cell_type": "code",
+ "source": [
+ "#БЕЗ ИЗМЕНЕНИЙ\n",
+ "# функция потерь\n",
+ "with tf.name_scope('cross_entropy'):\n",
+ " with tf.name_scope('total'):\n",
+ " cross_entropy = - tf.reduce_sum(Y_ * tf.log(Y))\n",
+ " #cross_entropy = tf.losses.softmax_cross_entropy(onehot_labels=Y_, logits=Y)\n",
+ " \n",
+ "# Optimizer\n",
+ "learning_rate = 0.003\n",
+ "with tf.name_scope('train'):\n",
+ " optimizer = tf.train.AdamOptimizer(learning_rate) #<-------------------\n",
+ " train_step = optimizer.minimize(cross_entropy)"
+ ],
+ "execution_count": 0,
+ "outputs": []
+ },
+ {
+ "metadata": {
+ "id": "NVzETbtko_MJ",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "cell_type": "code",
+ "source": [
+ "#БЕЗ ИЗМЕНЕНИЙ\n",
+ "tf.summary.scalar('cross_entropy', cross_entropy)\n",
+ "\n",
+ "with tf.name_scope('accuracy'):\n",
+ " with tf.name_scope('correct_prediction'):\n",
+ " correct_prediction = tf.equal(tf.argmax(Y, axis=1), tf.argmax(Y_, axis=1))\n",
+ " with tf.name_scope('accuracy'):\n",
+ " accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))\n",
+ "tf.summary.scalar('accuracy', accuracy)\n",
+ "\n",
+ "# Merge all summaries together\n",
+ "merged = tf.summary.merge_all()"
+ ],
+ "execution_count": 0,
+ "outputs": []
+ },
+ {
+ "metadata": {
+ "id": "kYBnsRUYo_ML",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "cell_type": "code",
+ "source": [
+ "max_step=2001\n",
+ "batch_size=100\n",
+ "\n",
+ "session = tf.InteractiveSession()\n",
+ "\n",
+ "summary_path = get_summary_path('nn_4hid_ReLU_Adam')\n",
+ "\n",
+ "train_writer = tf.summary.FileWriter(summary_path + '/train', session.graph)\n",
+ "test_writer = tf.summary.FileWriter(summary_path + '/test')\n",
+ "\n",
+ "tf.global_variables_initializer().run()\n",
+ "\n",
+ "for i in range(max_step):\n",
+ " if i % 10 == 0:\n",
+ " xs, ys = mnist.test.images, mnist.test.labels\n",
+ " summary, acc = session.run([merged, accuracy], feed_dict={X: xs, Y_: ys})\n",
+ " test_writer.add_summary(summary, i)\n",
+ " if i % 100 == 0:\n",
+ " print('Accuracy at step %s: %s' % (i, acc))\n",
+ " else:\n",
+ " xs, ys = mnist.train.next_batch(batch_size)\n",
+ " summary, _ = session.run([merged, train_step], feed_dict={X: xs, Y_: ys})\n",
+ " train_writer.add_summary(summary, i)\n",
+ "\n",
+ "session.close()"
+ ],
+ "execution_count": 0,
+ "outputs": []
+ },
+ {
+ "metadata": {
+ "id": "KMWr2qoWo_MQ",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# 4. Затухание темпа обучения\n",
+ "\n",
+ "Мы уже получаем довольно хорошую точность - 96%!\n",
+ "\n",
+ "Кривые зашумлены, скорее всего это означает что мы идём слишком большим шагом. Но мы не можем просто так его уменьшить, в этом случае обучение будет слишком долгим. Самое лучшее что можно сделать - начать с большого шага и постепенно его уменьшать."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "H2kLRXMro_MR",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "cell_type": "code",
+ "source": [
+ "image_size = 28\n",
+ "n_classes = 10\n",
+ "\n",
+ "tf.reset_default_graph() #сбросим граф\n",
+ "\n",
+ "#мета для входных картинок\n",
+ "X = tf.placeholder(tf.float32, [None, image_size*image_size], name='train_inputs')\n",
+ "# мета для правильных ответов\n",
+ "Y_ = tf.placeholder(tf.float32, [None, 10], name='train_labels')\n",
+ "\n",
+ "hidden1 = nn_layer(X, image_size*image_size, 200, 'hidden1', tf.nn.relu)\n",
+ "hidden2 = nn_layer(hidden1, 200, 100, 'hidden2', tf.nn.relu)\n",
+ "hidden3 = nn_layer(hidden2, 100, 60, 'hidden3', tf.nn.relu)\n",
+ "hidden4 = nn_layer(hidden3, 60, 30, 'hidden4', tf.nn.relu)\n",
+ "\n",
+ "Y = nn_layer(hidden4, 30, n_classes, 'out', tf.identity)\n",
+ "\n",
+ "#БЕЗ ИЗМЕНЕНИЙ\n",
+ "# функция потерь\n",
+ "with tf.name_scope('cross_entropy'):\n",
+ " with tf.name_scope('total'):\n",
+ " #ТУТ НАДО ЗАМЕНИТЬ НА БИБЛИОТЕЧНУЮ. т.к. она корректно обрабатывает NaN.\n",
+ " #При этом, нужно будет на последнем слое softmax убрать\n",
+ " #cross_entropy = - tf.reduce_sum(Y_ * tf.log(Y)) \n",
+ " cross_entropy = tf.losses.softmax_cross_entropy(onehot_labels=Y_, logits=Y)"
+ ],
+ "execution_count": 0,
+ "outputs": []
+ },
+ {
+ "metadata": {
+ "id": "a7VviSxEo_Mb",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "cell_type": "code",
+ "source": [
+ "step = tf.placeholder(tf.int32)\n",
+ "learning_rate = 0.0001 + tf.train.exponential_decay(0.003, step, 2000, 1/math.e)\n",
+ "with tf.name_scope('train'):\n",
+ " optimizer = tf.train.AdamOptimizer(learning_rate)\n",
+ " train_step = optimizer.minimize(cross_entropy)"
+ ],
+ "execution_count": 0,
+ "outputs": []
+ },
+ {
+ "metadata": {
+ "id": "8nBipySDo_Mf",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "Чтобы уемньшать шаг в зависимости от итерации, нужно задать placeholder для передачи номера итерации во время вычисления графа. На основе этого шага и сформируем экспоненциально затухающий темп."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "2XNUWIUno_Mf",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "cell_type": "code",
+ "source": [
+ "#БЕЗ ИЗМЕНЕНИЙ\n",
+ "tf.summary.scalar('cross_entropy', cross_entropy)\n",
+ "\n",
+ "with tf.name_scope('accuracy'):\n",
+ " with tf.name_scope('correct_prediction'):\n",
+ " correct_prediction = tf.equal(tf.argmax(Y, axis=1), tf.argmax(Y_, axis=1))\n",
+ " with tf.name_scope('accuracy'):\n",
+ " accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))\n",
+ "tf.summary.scalar('accuracy', accuracy)\n",
+ "\n",
+ "# Merge all summaries together\n",
+ "merged = tf.summary.merge_all()"
+ ],
+ "execution_count": 0,
+ "outputs": []
+ },
+ {
+ "metadata": {
+ "id": "lnxRXx0Go_Mh",
+ "colab_type": "code",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 1817
+ },
+ "outputId": "f150ba32-f5bf-4d1b-9c09-6a5a5cbb59ca"
+ },
+ "cell_type": "code",
+ "source": [
+ "max_step=10000\n",
+ "batch_size=100\n",
+ "\n",
+ "session = tf.InteractiveSession()\n",
+ "\n",
+ "summary_path = get_summary_path('nn_4hid_ReLU_Adam_lrdecay')\n",
+ "\n",
+ "train_writer = tf.summary.FileWriter(summary_path + '/train', session.graph)\n",
+ "test_writer = tf.summary.FileWriter(summary_path + '/test')\n",
+ "\n",
+ "tf.global_variables_initializer().run()\n",
+ "\n",
+ "for i in range(max_step):\n",
+ " if i % 10 == 0:\n",
+ " xs, ys = mnist.test.images, mnist.test.labels\n",
+ " summary, acc = session.run([merged, accuracy], feed_dict={X: xs, Y_: ys})\n",
+ " test_writer.add_summary(summary, i)\n",
+ " if i % 100 == 0:\n",
+ " print('Accuracy at step %s: %s' % (i, acc))\n",
+ " else:\n",
+ " xs, ys = mnist.train.next_batch(batch_size)\n",
+ " summary, _ = session.run([merged, train_step], feed_dict={X: xs, Y_: ys, step: i}) # <-----------step: i---------------\n",
+ " train_writer.add_summary(summary, i)\n",
+ "\n",
+ "session.close()"
+ ],
+ "execution_count": 20,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "text": [
+ "Accuracy at step 0: 0.0912\n",
+ "Accuracy at step 100: 0.888\n",
+ "Accuracy at step 200: 0.9331\n",
+ "Accuracy at step 300: 0.95\n",
+ "Accuracy at step 400: 0.9535\n",
+ "Accuracy at step 500: 0.9545\n",
+ "Accuracy at step 600: 0.961\n",
+ "Accuracy at step 700: 0.9549\n",
+ "Accuracy at step 800: 0.9693\n",
+ "Accuracy at step 900: 0.9672\n",
+ "Accuracy at step 1000: 0.9687\n",
+ "Accuracy at step 1100: 0.9674\n",
+ "Accuracy at step 1200: 0.9678\n",
+ "Accuracy at step 1300: 0.9682\n",
+ "Accuracy at step 1400: 0.973\n",
+ "Accuracy at step 1500: 0.9739\n",
+ "Accuracy at step 1600: 0.9752\n",
+ "Accuracy at step 1700: 0.9745\n",
+ "Accuracy at step 1800: 0.9732\n",
+ "Accuracy at step 1900: 0.9738\n",
+ "Accuracy at step 2000: 0.976\n",
+ "Accuracy at step 2100: 0.9738\n",
+ "Accuracy at step 2200: 0.9769\n",
+ "Accuracy at step 2300: 0.975\n",
+ "Accuracy at step 2400: 0.9766\n",
+ "Accuracy at step 2500: 0.9745\n",
+ "Accuracy at step 2600: 0.9758\n",
+ "Accuracy at step 2700: 0.9774\n",
+ "Accuracy at step 2800: 0.9777\n",
+ "Accuracy at step 2900: 0.9764\n",
+ "Accuracy at step 3000: 0.9753\n",
+ "Accuracy at step 3100: 0.9754\n",
+ "Accuracy at step 3200: 0.9748\n",
+ "Accuracy at step 3300: 0.9751\n",
+ "Accuracy at step 3400: 0.9798\n",
+ "Accuracy at step 3500: 0.978\n",
+ "Accuracy at step 3600: 0.9782\n",
+ "Accuracy at step 3700: 0.9782\n",
+ "Accuracy at step 3800: 0.9775\n",
+ "Accuracy at step 3900: 0.9764\n",
+ "Accuracy at step 4000: 0.9786\n",
+ "Accuracy at step 4100: 0.9783\n",
+ "Accuracy at step 4200: 0.9781\n",
+ "Accuracy at step 4300: 0.9778\n",
+ "Accuracy at step 4400: 0.9797\n",
+ "Accuracy at step 4500: 0.9799\n",
+ "Accuracy at step 4600: 0.9783\n",
+ "Accuracy at step 4700: 0.98\n",
+ "Accuracy at step 4800: 0.9781\n",
+ "Accuracy at step 4900: 0.9771\n",
+ "Accuracy at step 5000: 0.9796\n",
+ "Accuracy at step 5100: 0.9787\n",
+ "Accuracy at step 5200: 0.9805\n",
+ "Accuracy at step 5300: 0.9794\n",
+ "Accuracy at step 5400: 0.9801\n",
+ "Accuracy at step 5500: 0.9809\n",
+ "Accuracy at step 5600: 0.9796\n",
+ "Accuracy at step 5700: 0.9804\n",
+ "Accuracy at step 5800: 0.9805\n",
+ "Accuracy at step 5900: 0.9795\n",
+ "Accuracy at step 6000: 0.979\n",
+ "Accuracy at step 6100: 0.9804\n",
+ "Accuracy at step 6200: 0.9808\n",
+ "Accuracy at step 6300: 0.9806\n",
+ "Accuracy at step 6400: 0.9802\n",
+ "Accuracy at step 6500: 0.9798\n",
+ "Accuracy at step 6600: 0.9795\n",
+ "Accuracy at step 6700: 0.9796\n",
+ "Accuracy at step 6800: 0.9793\n",
+ "Accuracy at step 6900: 0.9799\n",
+ "Accuracy at step 7000: 0.9805\n",
+ "Accuracy at step 7100: 0.9809\n",
+ "Accuracy at step 7200: 0.9805\n",
+ "Accuracy at step 7300: 0.9794\n",
+ "Accuracy at step 7400: 0.9804\n",
+ "Accuracy at step 7500: 0.9797\n",
+ "Accuracy at step 7600: 0.979\n",
+ "Accuracy at step 7700: 0.9808\n",
+ "Accuracy at step 7800: 0.9804\n",
+ "Accuracy at step 7900: 0.9804\n",
+ "Accuracy at step 8000: 0.9798\n",
+ "Accuracy at step 8100: 0.9802\n",
+ "Accuracy at step 8200: 0.9805\n",
+ "Accuracy at step 8300: 0.9804\n",
+ "Accuracy at step 8400: 0.9803\n",
+ "Accuracy at step 8500: 0.9799\n",
+ "Accuracy at step 8600: 0.98\n",
+ "Accuracy at step 8700: 0.9798\n",
+ "Accuracy at step 8800: 0.9801\n",
+ "Accuracy at step 8900: 0.9803\n",
+ "Accuracy at step 9000: 0.9806\n",
+ "Accuracy at step 9100: 0.9803\n",
+ "Accuracy at step 9200: 0.9805\n",
+ "Accuracy at step 9300: 0.9801\n",
+ "Accuracy at step 9400: 0.9804\n",
+ "Accuracy at step 9500: 0.98\n",
+ "Accuracy at step 9600: 0.9804\n",
+ "Accuracy at step 9700: 0.9809\n",
+ "Accuracy at step 9800: 0.9809\n",
+ "Accuracy at step 9900: 0.9805\n"
+ ],
+ "name": "stdout"
+ }
+ ]
+ },
+ {
+ "metadata": {
+ "id": "fcDb0wi5o_Mm",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "Т.к. ввели новую метапеременную (step), нужно передавать её при выполнении сессии"
+ ]
+ },
+ {
+ "metadata": {
+ "id": "edJPWz3uo_Mn",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Прореживание\n",
+ "\n",
+ "В какой-то момент на больших итерациях тестовая кросс-энтропия начинает расти - дальше её уже ничто не остановит. Это переобучение. Ведь алгоритм настраивается только на кросс-энтропию обучающих данных. Просто наша сеть смогла запомнить почти все образцы обучающей выборки и потеряла обобщающую способность.\n",
+ "\n",
+ "В этом случае мы можем попробовать прореживание, или dropuot - один из видов регуляризации:\n", + "
![](\"https://codelabs.developers.google.com/codelabs/cloud-tensorflow-mnist/img/5ee25552f4c216c.png/")
"
+ ]
+ },
+ {
+ "metadata": {
+ "id": "R0yXXwgho_Mp",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "cell_type": "code",
+ "source": [
+ "image_size = 28\n",
+ "n_classes = 10\n",
+ "\n",
+ "tf.reset_default_graph()\n",
+ "\n",
+ "#мета для входных картинок\n",
+ "X = tf.placeholder(tf.float32, [None, image_size*image_size], name='train_inputs')\n",
+ "# мета для правильных ответов\n",
+ "Y_ = tf.placeholder(tf.float32, [None, 10], name='train_labels')"
+ ],
+ "execution_count": 0,
+ "outputs": []
+ },
+ {
+ "metadata": {
+ "id": "FqcWSr4no_Mu",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "cell_type": "code",
+ "source": [
+ "def variable_summaries(var):\n",
+ " with tf.name_scope('summaries'):\n",
+ " mean = tf.reduce_mean(var)\n",
+ " tf.summary.scalar('mean', mean)\n",
+ " with tf.name_scope('stddev'):\n",
+ " stddev = tf.sqrt(tf.reduce_mean(tf.square(var - mean)))\n",
+ " tf.summary.scalar('stddev', stddev)\n",
+ " tf.summary.scalar('max', tf.reduce_max(var))\n",
+ " tf.summary.scalar('min', tf.reduce_min(var))\n",
+ " tf.summary.histogram('histogram', var)\n",
+ "\n",
+ "def nn_layer(input_tensor, input_dim, output_dim, layer_name, activation):\n",
+ " with tf.name_scope(layer_name):\n",
+ " with tf.name_scope('weights'):\n",
+ " initial = tf.truncated_normal([input_dim, output_dim], stddev=0.1)\n",
+ " weights = tf.Variable(initial)\n",
+ " variable_summaries(weights)\n",
+ " with tf.name_scope('biases'):\n",
+ " initial = tf.truncated_normal([output_dim], stddev=0.1)\n",
+ " biases = tf.Variable(initial)\n",
+ " variable_summaries(biases)\n",
+ " with tf.name_scope('xw_plus_b'):\n",
+ " preactivate = tf.nn.xw_plus_b(input_tensor, weights, biases)\n",
+ " activations = activation(preactivate, name='activation')\n",
+ " \n",
+ " return activations "
+ ],
+ "execution_count": 0,
+ "outputs": []
+ },
+ {
+ "metadata": {
+ "id": "4O4I4_kEo_My",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "cell_type": "code",
+ "source": [
+ "pkeep = tf.placeholder(tf.float32)\n",
+ "\n",
+ "hidden1 = nn_layer(X, image_size*image_size, 200, 'hidden1', tf.nn.relu)\n",
+ "hidden1_d = tf.nn.dropout(hidden1, pkeep)\n",
+ "\n",
+ "hidden2 = nn_layer(hidden1_d, 200, 100, 'hidden2', tf.nn.relu)\n",
+ "hidden2_d = tf.nn.dropout(hidden2, pkeep)\n",
+ "\n",
+ "hidden3 = nn_layer(hidden2_d, 100, 60, 'hidden3', tf.nn.relu)\n",
+ "hidden3_d = tf.nn.dropout(hidden3, pkeep)\n",
+ "\n",
+ "hidden4 = nn_layer(hidden3_d, 60, 30, 'hidden4', tf.nn.relu)\n",
+ "hidden4_d = tf.nn.dropout(hidden4, pkeep)\n",
+ "\n",
+ "Y = nn_layer(hidden4_d, 30, n_classes, 'out', tf.identity)"
+ ],
+ "execution_count": 0,
+ "outputs": []
+ },
+ {
+ "metadata": {
+ "id": "Xutg2WuJo_M2",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "Дропаут заключается в том, что на каждой итерации обучения некоторые нейроны полностью выбрасываются из сети. Какие именно нейроны это будут, определяется параметром pkeep, который задаёт вероятность того, что нейрон останестя. При работе с тестовыми данными уже нельзя выкидывать нейроны - сеть обучена - и поэтому параметр выставляется в 1."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "owkaFBM7o_M2",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "cell_type": "code",
+ "source": [
+ "#БЕЗ ИЗМЕНЕНИЙ\n",
+ "# функция потерь\n",
+ "with tf.name_scope('cross_entropy'):\n",
+ " with tf.name_scope('total'):\n",
+ " cross_entropy = tf.losses.softmax_cross_entropy(onehot_labels=Y_, logits=Y)\n",
+ " \n",
+ "step = step = tf.placeholder(tf.int32)\n",
+ "learning_rate = 0.0001 + tf.train.exponential_decay(0.003, step, 2000, 1/math.e)\n",
+ "with tf.name_scope('train'):\n",
+ " optimizer = tf.train.AdamOptimizer(learning_rate)\n",
+ " train_step = optimizer.minimize(cross_entropy)\n",
+ " \n",
+ "#БЕЗ ИЗМЕНЕНИЙ\n",
+ "tf.summary.scalar('cross_entropy', cross_entropy)\n",
+ "\n",
+ "with tf.name_scope('accuracy'):\n",
+ " with tf.name_scope('correct_prediction'):\n",
+ " correct_prediction = tf.equal(tf.argmax(Y, axis=1), tf.argmax(Y_, axis=1))\n",
+ " with tf.name_scope('accuracy'):\n",
+ " accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))\n",
+ "tf.summary.scalar('accuracy', accuracy)\n",
+ "\n",
+ "# Merge all summaries together\n",
+ "merged = tf.summary.merge_all()"
+ ],
+ "execution_count": 0,
+ "outputs": []
+ },
+ {
+ "metadata": {
+ "id": "JqqTPaI2o_M5",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "cell_type": "code",
+ "source": [
+ "max_step=10000\n",
+ "batch_size=100\n",
+ "\n",
+ "session = tf.InteractiveSession()\n",
+ "\n",
+ "summary_path = get_summary_path('nn_4hid_ReLU_Adam_lrdecay_dp')\n",
+ "\n",
+ "train_writer = tf.summary.FileWriter(summary_path + '/train', session.graph)\n",
+ "test_writer = tf.summary.FileWriter(summary_path + '/test')\n",
+ "\n",
+ "tf.global_variables_initializer().run()\n",
+ "\n",
+ "for i in range(max_step):\n",
+ " if i % 10 == 0:\n",
+ " xs, ys = mnist.test.images, mnist.test.labels\n",
+ " summary, acc = session.run([merged, accuracy], feed_dict={X: xs, Y_: ys, pkeep: 1.0}) #<----------pkeep: 1.0--------------\n",
+ " test_writer.add_summary(summary, i)\n",
+ " if i % 100 == 0:\n",
+ " print('Accuracy at step %s: %s' % (i, acc))\n",
+ " else:\n",
+ " xs, ys = mnist.train.next_batch(batch_size)\n",
+ " summary, _ = session.run([merged, train_step], feed_dict={X: xs, Y_: ys, step: i, pkeep: 0.75}) #<----------pkeep: 0.75-------------\n",
+ " train_writer.add_summary(summary, i)\n",
+ "\n",
+ "session.close()"
+ ],
+ "execution_count": 0,
+ "outputs": []
+ },
+ {
+ "metadata": {
+ "id": "YeaHar6Fo_M8",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "Опять ввели новую метапеременную - нужно передать её значение, т.к. мы обещали графу, что на момент вычислений там что-нибудь окажется"
+ ]
+ }
+ ]
+}
\ No newline at end of file