WL#6599 - New Data Dictionary and I_S integration.

This WL defines INFORMATION_SCHEMA (I_S) system view over DD tables,
representing a I_S table. This will eliminate the need of preparing a
temporary table for each I_S table during execution and enable faster
execution of I_S queries.

A. Functional changes introduced:
---------------------------------
A.1) Implements following I_S tables as a system view.

  CHARACTER_SETS
  COLLATIONS
  COLLATION_CHARACTER_SET_APPLICABILITY
  SCHEMATA
  TABLE_NAMES
  TABLES
  VIEWS
  COLUMNS
  STATISTICS
  KEY_COLUMN_USAGE
  TABLE_CONSTRAINTS

A.2) Implements following SHOW commands to use A.1).

  SHOW CHARSET
  SHOW COLLATION
  SHOW DATABASES
  SHOW TABLES
  SHOW TABLE STATUS
  SHOW COLUMNS
  SHOW KEYS/INDEXES
  SHOW STATISTICS
  DESCRIBE

A.3)
  Introduces a session variable 'information_schema_stats' that
  enables use to decide how to get the table/index dynamic
  statistics. Setting it to 'latest' would get latest statistics
  from storage engine. And setting it to 'cached' would get
  statistics stored in mysql.table_stats/index_stats. These are two
  new DD tables introduced by this WL. These tables are updated
  when user runs ANALYZE TABLE on a table. For more details see HLS
  2.1.4) and 2.1.12).

A.4) Introduced following new DD columns, see HLS 2.1.8) for more
     details.

  COLUMNS.COLUMN_KEY
  COLUMNS.COLUMN_TYPE
  COLUMNS.DEFAULT_VALUE_UTF8
  TABLES.ROW_FORMAT

A.5)
  Implements special handling for SHOW COLUMNS/KEYS for temporary
  tables as the meta data of temporary tables are not stored in DD.

A.6)
  I_S tables are temporary tables on 5.7. This make them
  possible to access under LOCK TABLE mode with explicitly
  locking I_S tables. Now with this WL, the query execution
  procedure fails to process I_S system view as the under laying DD
  table are not locked.

  So, this WL makes the SQL server to open the DD tables used by
  a I_S system view without requiring them to be locked explicitly.

B. Performance improvements:
----------------------------

See mysql wiki page WL6599_I_S_performance_with_data_dictionary
for more details. (HLS section 5)

C. Compatibility issues:
------------------------
There are few differences in the way mysql server would behave
after this WL. The detailed list of these change in behavior is
listed under HLS section 6).

D. Source files:
----------------
Most of code changes are placed in sql/dd/info_schema folder.

* Refer code in sql/dd/info_schema/show.* to know how SHOW
  commands are implemented.

* Refer code in sql/dd/info_schema/stats.* to know how the
  retrieval of dynamic statistics are implemented.

E. Related WL's and Bugs:
-------------------------

* WL#7167 Change DDL to update rows for view columns in
  DD.COLUMNS and other dependent values.

  I_S.COLUMNS will get column meta data for views from
  mysql.columns. The view's column meta data are stored in
  mysql.columns by this WL. The WL takes care of enabling several
  test cases that is disabled by this WL#6599. So WL#6599 and
  WL#7167 would be pushed together. They are QA'ed together.

* WL#7464 - InnoDB: provide a way to do non-locking reads.
  This WL enables execution of I_S queries do to non-locking reads.

* WL#8232 - New data-dictionary: change approach to table
            row_format and index algorithm validation/handling

Upcoming WL:
* WL#9494   Implement INFORMATION_SCHEMA system views for
            SP/TRIGGERS/EVENTS/REFERENTIAL_CONSTRAINTS

* WL#9495   Update schema tables of dynamic plugins into data
            dictionary.

The following bugs would be fixed by this WL:
  Bug#11748044
  Bug#13878164

F. Disabled tests:
------------------

thread_pool.thread_pool_i_s : Would be fixed by WL#9495.
i_innodb.innodb_bug14150372 : WL6599_INNODB_SPORADIC

Author: gopshank

client/dump/mysql_crawler.cc

@@ -122,6 +122,15 @@ void Mysql_crawler::enumerate_database_objects(const Database& db)
 void Mysql_crawler::enumerate_tables(const Database& db)
 {
   Mysql::Tools::Base::Mysql_query_runner* runner= this->get_runner();
+
+  /*
+    Get statistics from SE by setting information_schema_stats=LATEST
+    for this session. This makes the queries IS queries retrieve latest
+    statistics and avoids getting outdated statistics.
+  */
+  std::vector<const Mysql::Tools::Base::Mysql_query_runner::Row*> t;
+  runner->run_query_store("SET SESSION information_schema_stats=latest", &t);
+
   std::vector<const Mysql::Tools::Base::Mysql_query_runner::Row*> tables;
 
   runner->run_query_store("SHOW TABLE STATUS FROM "

client/dump/mysqldump_tool_chain_maker_options.cc

@@ -137,6 +137,28 @@ void Mysqldump_tool_chain_maker_options::process_positional_options(
       db_name, positional_options[i]));
   }
 
+  /*
+    INFORMATION_SCHEMA DB content dump is only used to reload the data
+    into another tables for analysis purpose. This feature is not the
+    core responsibility of mysqlpump tool. INFORMATION_SCHEMA DB
+    content can even be dumped using other methods like SELECT INTO
+    OUTFILE... for such purpose.
+    Hence reporting error if INFORMATION_SCHEMA DB is in databases list.
+  */
+  for (auto database : m_object_filter.m_databases_included)
+  {
+    auto db_name= std::get<1>(database);
+
+    if (!my_strcasecmp(&my_charset_latin1, db_name.c_str(),
+                       INFORMATION_SCHEMA_DB_NAME))
+    {
+      m_mysql_chain_element_options->get_program()->error(
+        Mysql::Tools::Base::Message_data(1, "Dumping "
+        "\'INFORMATION_SCHEMA\' DB content is not supported.",
+        Mysql::Tools::Base::Message_type_error));
+    }
+  }
+
   /*
     We add standard exclusions only if objects are included by default, i.e.
     there are exclusions or there is no exclusions and inclusions.

client/mysqldump.cc

@@ -588,6 +588,7 @@ static int dump_databases(char **);
 static int dump_all_databases();
 static char *quote_name(const char *name, char *buff, my_bool force);
 char check_if_ignore_table(const char *table_name, char *table_type);
+bool is_infoschema_db(const char *db);
 static char *primary_key_fields(const char *table_name);
 static my_bool get_view_structure(char *table, char* db);
 static my_bool dump_all_views_in_db(char *database);
@@ -2872,9 +2873,12 @@ static uint get_table_structure(char *table, char *db, char *table_type,
       */
       while ((row= mysql_fetch_row(result)))
       {
-        complete_insert|=
-          strcmp(row[SHOW_EXTRA], "STORED GENERATED") == 0 ||
-          strcmp(row[SHOW_EXTRA], "VIRTUAL GENERATED") == 0;
+        if (row[SHOW_EXTRA])
+        {
+          complete_insert|=
+            strcmp(row[SHOW_EXTRA], "STORED GENERATED") == 0 ||
+            strcmp(row[SHOW_EXTRA], "VIRTUAL GENERATED") == 0;
+        }
       }
       mysql_free_result(result);
 
@@ -2915,9 +2919,14 @@ static uint get_table_structure(char *table, char *db, char *table_type,
     colno= 0;
     while ((row= mysql_fetch_row(result)))
     {
-      real_columns[colno]=
-        strcmp(row[SHOW_EXTRA], "STORED GENERATED") != 0 &&
-        strcmp(row[SHOW_EXTRA], "VIRTUAL GENERATED") != 0;
+      if (row[SHOW_EXTRA])
+      {
+        real_columns[colno]=
+          strcmp(row[SHOW_EXTRA], "STORED GENERATED") != 0 &&
+          strcmp(row[SHOW_EXTRA], "VIRTUAL GENERATED") != 0;
+      }
+      else
+        real_columns[colno]= TRUE;
 
       if (real_columns[colno++] && complete_insert)
       {
@@ -2953,9 +2962,12 @@ static uint get_table_structure(char *table, char *db, char *table_type,
       */
       while ((row= mysql_fetch_row(result)))
       {
-        complete_insert|=
-          strcmp(row[SHOW_EXTRA], "STORED GENERATED") == 0 ||
-          strcmp(row[SHOW_EXTRA], "VIRTUAL GENERATED") == 0;
+        if (row[SHOW_EXTRA])
+        {
+          complete_insert|=
+            strcmp(row[SHOW_EXTRA], "STORED GENERATED") == 0 ||
+            strcmp(row[SHOW_EXTRA], "VIRTUAL GENERATED") == 0;
+        }
       }
       mysql_free_result(result);
 
@@ -3013,9 +3025,14 @@ static uint get_table_structure(char *table, char *db, char *table_type,
     {
       ulong *lengths= mysql_fetch_lengths(result);
 
-      real_columns[colno]=
-        strcmp(row[SHOW_EXTRA], "STORED GENERATED") != 0 &&
-        strcmp(row[SHOW_EXTRA], "VIRTUAL GENERATED") != 0;
+      if (row[SHOW_EXTRA])
+      {
+        real_columns[colno]=
+          strcmp(row[SHOW_EXTRA], "STORED GENERATED") != 0 &&
+          strcmp(row[SHOW_EXTRA], "VIRTUAL GENERATED") != 0;
+      }
+      else
+        real_columns[colno]= TRUE;
 
       if (!real_columns[colno++])
         continue;
@@ -3057,7 +3074,7 @@ static uint get_table_structure(char *table, char *db, char *table_type,
         }
         if (!row[SHOW_NULL][0])
           fputs(" NOT NULL", sql_file);
-        if (row[SHOW_EXTRA][0])
+        if (row[SHOW_EXTRA] && row[SHOW_EXTRA][0])
           fprintf(sql_file, " %s",row[SHOW_EXTRA]);
         check_io(sql_file);
       }
@@ -4435,6 +4452,9 @@ static int dump_databases(char **db_names)
 
   for (db= db_names ; *db ; db++)
   {
+    if (is_infoschema_db(*db))
+      die(EX_USAGE, "Dumping \'%s\' DB content is not supported", *db);
+
     if (dump_all_tables_in_db(*db))
       result=1;
   }
@@ -4860,6 +4880,9 @@ static int dump_selected_tables(char *db, char **table_names, int tables)
   char **dump_tables, **pos, **end;
   DBUG_ENTER("dump_selected_tables");
 
+  if (is_infoschema_db(db))
+    die(EX_USAGE, "Dumping \'%s\' DB content is not supported", db);
+
   if (init_dumping(db, init_dumping_tables))
     DBUG_RETURN(1);
 
@@ -5425,6 +5448,38 @@ char check_if_ignore_table(const char *table_name, char *table_type)
 }
 
 
+/**
+  Check if the database is 'information_schema' and write a verbose message
+  stating that dumping the database is not supported.
+
+  @param  db        Database Name.
+
+  @retval true      If database should be ignored.
+  @retval false     If database shouldn't be ignored.
+*/
+
+bool is_infoschema_db(const char *db)
+{
+  DBUG_ENTER("is_infoschema_db");
+
+  /*
+    INFORMATION_SCHEMA DB content dump is only used to reload the data into
+    another tables for analysis purpose. This feature is not the core
+    responsibility of mysqldump tool. INFORMATION_SCHEMA DB content can even be
+    dumped using other methods like SELECT INTO OUTFILE... for such purpose.
+    Hence ignoring INFORMATION_SCHEMA DB here.
+  */
+  if (mysql_get_server_version(mysql) >= FIRST_INFORMATION_SCHEMA_VERSION &&
+      !my_strcasecmp(&my_charset_latin1, db, INFORMATION_SCHEMA_DB_NAME))
+  {
+    verbose_msg("Dumping \'%s\' DB content is not supported", db);
+    DBUG_RETURN(true);
+  }
+
+  DBUG_RETURN(false);
+}
+
+
 /*
   Get string of comma-separated primary key field names
 

mysql-test/extra/binlog_tests/blackhole.test

@@ -94,6 +94,7 @@ INSERT INTO t1 VALUES('MySQL has now support', 'for full-text search'),
                           ('Only MyISAM tables','support collections'),
              ('Function MATCH ... AGAINST()','is used to do a search'),
         ('Full-text search in MySQL', 'implements vector space model');
+ANALYZE TABLE t1;
 SHOW INDEX FROM t1;
 
 # nl search

mysql-test/include/commit.inc

@@ -653,7 +653,7 @@ drop table t2;
 set sql_mode=no_engine_substitution;
 create temporary table t2 (a int);
 --disable_warnings
-call p_verify_status_increment(3, 0, 0, 0);
+call p_verify_status_increment(5, 0, 0, 0);
 --enable_warnings
 set sql_mode=default;
 --echo # 19. A function changes temp-trans-table.
@@ -878,7 +878,7 @@ call p_verify_status_increment(6, 0, 6, 0);
 commit;
 call p_verify_status_increment(0, 0, 0, 0);
 drop view v1;
-call p_verify_status_increment(3, 0, 3, 0);
+call p_verify_status_increment(5, 0, 3, 0);
 --enable_warnings
 
 --echo #

mysql-test/include/common-tests.inc

@@ -1804,6 +1804,7 @@ show full columns from t2;
 show full columns from t2 from test like 'f%';
 --replace_column 8 #
 show full columns from t2 from test like 's%';
+analyze table t2;
 --replace_column 7 #
 show keys from t2;
 

mysql-test/include/handler.inc

@@ -786,7 +786,8 @@ connection default;
 # Bug#44151 using handler commands on information_schema tables crashes server
 #
 USE information_schema;
---error ER_WRONG_USAGE
+--replace_result columns COLUMNS
+--error ER_WRONG_OBJECT
 HANDLER COLUMNS OPEN;
 USE test;
 

mysql-test/include/mtr_check.sql

@@ -1,4 +1,4 @@
--- Copyright (c) 2008, 2016, Oracle and/or its affiliates. All rights reserved.
+-- Copyright (c) 2008, 2016 Oracle and/or its affiliates. All rights reserved.
 --
 -- This program is free software; you can redistribute it and/or modify
 -- it under the terms of the GNU General Public License as published by
@@ -68,15 +68,16 @@ BEGIN
 
   -- Dump all databases, there should be none
   -- except those that was created during bootstrap
-  SELECT * FROM INFORMATION_SCHEMA.SCHEMATA;
+  SELECT * FROM INFORMATION_SCHEMA.SCHEMATA ORDER BY SCHEMA_NAME;
 
   -- Dump all tablespaces, there should be none
   SELECT * FROM INFORMATION_SCHEMA.FILES WHERE 
     FILE_TYPE !='TEMPORARY' AND TABLE_SCHEMA='test' ORDER BY FILE_ID;
 
   -- The test database should not contain any tables
   SELECT table_name AS tables_in_test FROM INFORMATION_SCHEMA.TABLES
-    WHERE table_schema='test';
+    WHERE table_schema='test'
+    ORDER BY TABLE_NAME;
 
   -- Show "mysql" database, tables and columns
   SELECT CONCAT(table_schema, '.', table_name) AS tables_in_mysql
@@ -90,7 +91,7 @@ BEGIN
          collation_name, column_type, column_key, extra, column_comment
     FROM INFORMATION_SCHEMA.COLUMNS
       WHERE table_schema='mysql' AND table_name != 'ndb_apply_status'
-        ORDER BY columns_in_mysql;
+        ORDER BY columns_in_mysql, column_name;
 
   -- Dump all events, there should be none
   SELECT * FROM INFORMATION_SCHEMA.EVENTS;
@@ -112,7 +113,8 @@ BEGIN
          CHARACTER_SET_CLIENT,COLLATION_CONNECTION,DATABASE_COLLATION
     FROM INFORMATION_SCHEMA.ROUTINES;
   -- Dump all views, only those in the sys schema should exist
-  SELECT * FROM INFORMATION_SCHEMA.VIEWS;
+  SELECT * FROM INFORMATION_SCHEMA.VIEWS
+    ORDER BY TABLE_SCHEMA, TABLE_NAME;
 
   SHOW GLOBAL STATUS LIKE 'slave_open_temp_tables';
 

mysql-test/t/mysqlshow.test renamed to mysql-test/include/mysqlshow.inc

@@ -1782,6 +1782,7 @@ show full columns from t2;
 show full columns from t2 from test like 'f%';
 --replace_column 8 #
 show full columns from t2 from test like 's%';
+analyze table t2;
 show keys from t2;
 
 drop table t4, t3, t2, t1;
@@ -2235,6 +2236,7 @@ drop table t1,t2;
 create table t1 (f1 int not null auto_increment primary key, f2 varchar(10));
 create table t11 like t1;
 insert into t1 values(1,""),(2,"");
+analyze table t1, t11;
 --replace_column 7 X 8 X 9 X 10 X 11 X 12 X 13 X 14 X
 show table status like 't1%';
 select 123 as a from t1 where f1 is null;

mysql-test/include/select.inc

@@ -1,9 +1,9 @@
 select schema_name from information_schema.schemata where schema_name not in ('sys', 'performance_schema');
-schema_name
-information_schema
-mtr
+SCHEMA_NAME
 mysql
+information_schema
 test
+mtr
 show tables in mysql where Tables_in_mysql != 'ndb_binlog_index';
 Tables_in_mysql
 column_stats