Skip to content

Commit 270fd34

Browse files
abhi5466dahlerlend
abhi5466
authored and
dahlerlend
committed
WL#6392 : Upgrade to Transactional Data Dictionary
With the implementation of the new data dictionary, the metadata will be stored in the InnoDB tables. For the users to upgrade from the older MySQL version(5.7) to mysql-8.0, dictionary tables should be created and populated in old data directory from various metadata sources. When the Mysql-8.0 server comes up, it detects for the presence of dictionary tables. If dictionary tables are not present, they will be created in old data directory and populated with the metadata. Then the server will proceed with normal start. If dictionary tables are present, server will proceed with normal startup. Upgrade Steps for Users ------------------------ 1. Do a pre requisite check on 5.7 data directory. 2. Do a slow shutdown of mysql-5.7 server. Slow shutdown involves setting global variable 'innodb_fast_shutdown' to zero before server shutdown. 3. Start mysql-8.0 server on 5.7 data directory. mysql-8.0 server will detect if Dictionary tables are present or not. If dictionary tables are not present, they will be created in old data directory. Metadata will be populated in dictionary tables. Then server will proceed with normal start. 4. Execute mysql_upgrade client tool. 5. Shutdown and start server again with normal configuration. (Recommended) Reviewed-by: Nisha Gopalakrishnan <[email protected]> Reviewed-by: Thayumanavar Sachithanantham <[email protected]> Reviewed-by: Dmitry Lenev <[email protected]>
1 parent 8a2163c commit 270fd34

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

59 files changed

+8126
-299
lines changed

‎mysql-test/r/dd_upgrade_error.result

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
call mtr.add_suppression("Plugin * after shutdown");
2+
call mtr.add_suppression(".* Native table .* has the wrong structure");
3+
call mtr.add_suppression("Resizing redo log from");
4+
call mtr.add_suppression("Starting to delete and rewrite log files");
5+
call mtr.add_suppression("New log files created");
6+
call mtr.add_suppression("Info table is not ready to be used.");
7+
call mtr.add_suppression("must use supported storage engine");
8+
call mtr.add_suppression("after shutdown");
9+
call mtr.add_suppression("will be forced to shutdown");
10+
call mtr.add_suppression("Table 'mysql.component' doesn't exist");
11+
call mtr.add_suppression("table is missing or has an incorrect definition");
12+
call mtr.add_suppression("is expected to be transactional");
13+
call mtr.add_suppression("Upgrading redo log");
14+
# Stop DB server which was created by MTR default
15+
# Copy the remote tablespace & DB zip files from suite location to working location.
16+
# Check that the file exists in the working folder.
17+
# Unzip the zip file.
18+
# Remove innodb log files due to an existing bug
19+
MYSQL_TMP_DIR/bootstrap.log
20+
# Starting the DB server will fail since the data dir contains
21+
# tables with non native partitioning.
22+
# Restart the server with default options.
23+
# restart

‎mysql-test/r/dd_upgrade_test.result

Lines changed: 508 additions & 0 deletions
Large diffs are not rendered by default.

‎mysql-test/std_data/data57.zip

2.59 MB
Binary file not shown.

‎mysql-test/t/dd_upgrade_error.test

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
# This test scripts covers upgrade of tables, views, tablespaces,
2+
# events, stored routines and triggers from 5.7 to 8.0.
3+
4+
--source include/not_embedded.inc
5+
6+
call mtr.add_suppression("Plugin * after shutdown");
7+
call mtr.add_suppression(".* Native table .* has the wrong structure");
8+
call mtr.add_suppression("Resizing redo log from");
9+
call mtr.add_suppression("Starting to delete and rewrite log files");
10+
call mtr.add_suppression("New log files created");
11+
call mtr.add_suppression("Info table is not ready to be used.");
12+
call mtr.add_suppression("must use supported storage engine");
13+
call mtr.add_suppression("after shutdown");
14+
call mtr.add_suppression("will be forced to shutdown");
15+
call mtr.add_suppression("Table 'mysql.component' doesn't exist");
16+
call mtr.add_suppression("table is missing or has an incorrect definition");
17+
call mtr.add_suppression("is expected to be transactional");
18+
call mtr.add_suppression("Upgrading redo log");
19+
20+
# Set different paths for --datadir
21+
let $MYSQLD_DATADIR1 = $MYSQL_TMP_DIR/data57;
22+
23+
--echo # Stop DB server which was created by MTR default
24+
--exec echo "wait" > $MYSQLTEST_VARDIR/tmp/mysqld.1.expect
25+
--source include/shutdown_mysqld.inc
26+
27+
--echo # Copy the remote tablespace & DB zip files from suite location to working location.
28+
--copy_file $MYSQLTEST_VARDIR/std_data/data57.zip $MYSQL_TMP_DIR/data57.zip
29+
30+
--echo # Check that the file exists in the working folder.
31+
--file_exists $MYSQL_TMP_DIR/data57.zip
32+
33+
--echo # Unzip the zip file.
34+
--exec unzip -qo $MYSQL_TMP_DIR/data57.zip -d $MYSQL_TMP_DIR
35+
36+
--echo # Remove innodb log files due to an existing bug
37+
--remove_file $MYSQL_TMP_DIR/data57/ib_logfile0
38+
--remove_file $MYSQL_TMP_DIR/data57/ib_logfile1
39+
40+
# Create a bootstrap file in temp location
41+
--replace_result $MYSQL_TMP_DIR MYSQL_TMP_DIR
42+
--exec echo $MYSQL_TMP_DIR/bootstrap.log
43+
44+
--echo # Starting the DB server will fail since the data dir contains
45+
--echo # tables with non native partitioning.
46+
let MYSQLD_LOG= $MYSQL_TMP_DIR/server.log;
47+
--error 1
48+
--exec $MYSQLD $extra_args --secure-file-priv="" --innodb_undo_tablespaces=0 --log-error=$MYSQLD_LOG --show_compatibility_56=1 --datadir=$MYSQLD_DATADIR1
49+
50+
let SEARCH_FILE= $MYSQLD_LOG;
51+
--let SEARCH_PATTERN= Error in creating TABLE_SHARE from p4_hash.frm file.
52+
--source include/search_pattern_in_file.inc
53+
54+
#Remove copied files
55+
--remove_file $MYSQLD_LOG
56+
--remove_file $MYSQL_TMP_DIR/data57.zip
57+
--force-rmdir $MYSQL_TMP_DIR/data57
58+
59+
--echo # Restart the server with default options.
60+
--source include/start_mysqld.inc

‎mysql-test/t/dd_upgrade_test.test

Lines changed: 177 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,177 @@
1+
# This test scripts covers upgrade of tables, views, tablespaces,
2+
# events, stored routines and triggers from 5.7 to 8.0.
3+
4+
--source include/not_embedded.inc
5+
6+
call mtr.add_suppression("Plugin * after shutdown");
7+
call mtr.add_suppression(".* Native table .* has the wrong structure");
8+
call mtr.add_suppression("Resizing redo log from");
9+
call mtr.add_suppression("Starting to delete and rewrite log files");
10+
call mtr.add_suppression("New log files created");
11+
call mtr.add_suppression("Info table is not ready to be used.");
12+
call mtr.add_suppression("must use supported storage engine");
13+
call mtr.add_suppression("after shutdown");
14+
call mtr.add_suppression("will be forced to shutdown");
15+
call mtr.add_suppression("Table 'mysql.component' doesn't exist");
16+
call mtr.add_suppression("ACL DDLs will not work unless mysql_upgrade is executed");
17+
call mtr.add_suppression("table is missing or has an incorrect definition");
18+
call mtr.add_suppression("is expected to be transactional");
19+
call mtr.add_suppression("Upgrading redo log");
20+
21+
# Set different paths for --datadir
22+
let $MYSQLD_DATADIR1 = $MYSQL_TMP_DIR/data57;
23+
24+
--echo # Stop DB server which was created by MTR default
25+
--exec echo "wait" > $MYSQLTEST_VARDIR/tmp/mysqld.1.expect
26+
--shutdown_server
27+
--source include/wait_until_disconnected.inc
28+
29+
--echo # Copy the remote tablespace & DB zip files from suite location to working location.
30+
--copy_file $MYSQLTEST_VARDIR/std_data/data57.zip $MYSQL_TMP_DIR/data57.zip
31+
32+
--echo # Check that the file exists in the working folder.
33+
--file_exists $MYSQL_TMP_DIR/data57.zip
34+
35+
--echo # Unzip the zip file.
36+
--exec unzip -qo $MYSQL_TMP_DIR/data57.zip -d $MYSQL_TMP_DIR
37+
38+
# These files are added to test error scenario, delete from for upgrade testing.
39+
--remove_file $MYSQL_TMP_DIR/data57/test/55_temporal.frm
40+
--remove_file $MYSQL_TMP_DIR/data57/test/55_temporal.MYD
41+
--remove_file $MYSQL_TMP_DIR/data57/test/55_temporal.MYI
42+
# Remove myisam partitioned tables. There are used for negative testing.
43+
--remove_files_wildcard $MYSQL_TMP_DIR/data57/partitions *
44+
--rmdir $MYSQL_TMP_DIR/data57/partitions
45+
46+
--echo #Files in 'test' dir
47+
--list_files $MYSQL_TMP_DIR/data57/
48+
49+
# Create a bootstrap file in temp location
50+
--replace_result $MYSQL_TMP_DIR MYSQL_TMP_DIR
51+
--exec echo $MYSQL_TMP_DIR/bootstrap.log
52+
53+
54+
--echo # Start the DB server.
55+
--exec echo "restart: --show_compatibility_56=1 --innodb_undo_tablespaces=0 --datadir=$MYSQLD_DATADIR1" > $MYSQLTEST_VARDIR/tmp/mysqld.1.expect
56+
--enable_reconnect
57+
--source include/wait_until_connected_again.inc
58+
59+
60+
--echo # Sanity Testing after Server start
61+
CREATE SCHEMA test123;
62+
CREATE TABLE test123.t1(a int);
63+
INSERT INTO test123.t1 values(1);
64+
SELECT * FROM test123.t1;
65+
DROP SCHEMA test123;
66+
67+
--echo # Execute mysql_upgrade
68+
69+
--source include/mysql_upgrade_preparation.inc
70+
--exec $MYSQL_UPGRADE --skip-verbose --force 2>&1
71+
--source include/mysql_upgrade_cleanup.inc
72+
73+
SHOW DATABASES;
74+
#Check for table
75+
SHOW CREATE TABLE mysql.user;
76+
77+
SHOW CREATE TABLE mysql.innodb_table_stats;
78+
SHOW CREATE TABLE mysql.innodb_index_stats;
79+
80+
SELECT NAME,LEN FROM INFORMATION_SCHEMA.INNODB_SYS_COLUMNS WHERE TABLE_ID IN
81+
(SELECT TABLE_ID FROM INFORMATION_SCHEMA.INNODB_SYS_TABLES WHERE NAME LIKE '%innodb_%_stats')
82+
AND NAME LIKE 'table_name';
83+
84+
SHOW CREATE TABLE test.t_compressed;
85+
SHOW CREATE TABLE test.t_compressed2;
86+
SHOW CREATE TABLE test.t_compressed3;
87+
SHOW CREATE TABLE test.t_dynamic;
88+
SHOW CREATE TABLE test.t_index;
89+
90+
SHOW CREATE TABLE test.vt2;
91+
SELECT * FROM test.vt2;
92+
SHOW CREATE TABLE test.t_gen_stored;
93+
SHOW CREATE TABLE test.jemp;
94+
SHOW CREATE TABLE test.t_gen_stored_myisam;
95+
SHOW CREATE TABLE test.t_gen_stored_myisam2;
96+
SHOW CREATE TABLE test.jemp_myisam;
97+
98+
99+
SELECT TABLE_SCHEMA, TABLE_NAME, TABLE_TYPE, ENGINE, VERSION, ROW_FORMAT
100+
FROM INFORMATION_SCHEMA.tables WHERE table_schema='test';
101+
# Check for views
102+
#Invalid view
103+
SHOW CREATE VIEW aview.view_invalid;
104+
#Valid views
105+
SHOW CREATE VIEW aview.view_2;
106+
SHOW CREATE VIEW aview.second_view;
107+
SELECT * FROM aview.view_2;
108+
SELECT * FROM aview.second_view;
109+
110+
SHOW CREATE VIEW aview.mixed_view;
111+
SHOW CREATE VIEW aview.mixed_view2;
112+
SELECT * FROM aview.mixed_view2;
113+
114+
#view with different definer
115+
SHOW CREATE VIEW aview.view_user;
116+
SHOW CREATE VIEW aview.view_user2;
117+
#view with different character_set and connection collation
118+
SHOW CREATE VIEW aview.view_character_set;
119+
120+
121+
#Check for SP/SF
122+
SHOW CREATE PROCEDURE sp.simpleproc;
123+
SHOW CREATE PROCEDURE sp.proc_123;
124+
SHOW CREATE PROCEDURE sp.proc_456;
125+
SHOW CREATE FUNCTION sp.hello;
126+
#Check for events
127+
SHOW CREATE EVENT events.e_totals;
128+
#Check for fulltext index
129+
SHOW CREATE TABLE test.opening_lines;
130+
131+
# Check for General tablespace
132+
SHOW CREATE TABLE tablespace.t2;
133+
SHOW CREATE TABLE tablespace.t3;
134+
SELECT * FROM tablespace.t3;
135+
SHOW CREATE TABLE tablespace.t4;
136+
137+
#Check for table with blackhole engine
138+
SHOW CREATE TABLE test.t_blackhole;
139+
140+
#Check for table with special character in name
141+
SHOW CREATE TABLE `test`.`t_sc~!@#$%^&*(`;
142+
143+
#Check for JSON types
144+
SHOW CREATE TABLE test.t_json;
145+
146+
#Check for geometry data types
147+
SHOW CREATE TABLE test.geom;
148+
149+
#Check for Triggers. Operations on t1 inserts data in t2.
150+
INSERT INTO triggers.t1 VALUES(1);
151+
UPDATE triggers.t1 SET a=2 WHERE a=1;
152+
SELECT * FROM triggers.t2;
153+
154+
SELECT TRIGGER_SCHEMA, TRIGGER_NAME, EVENT_MANIPULATION, ACTION_TIMING,
155+
ACTION_ORDER FROM INFORMATION_SCHEMA.TRIGGERS WHERE TRIGGER_SCHEMA='triggers';
156+
157+
--replace_column 7 #
158+
SHOW CREATE TRIGGER triggers.trg_t1_after_insert_3;
159+
--replace_column 7 #
160+
SHOW CREATE TRIGGER triggers.trg1;
161+
162+
--replace_column 7 #
163+
SHOW CREATE TRIGGER triggers.t1_bi;
164+
165+
#I_S query to check Foreign_keys
166+
SELECT * FROM INFORMATION_SCHEMA.REFERENTIAL_CONSTRAINTS WHERE
167+
CONSTRAINT_SCHEMA='foreign_keys' ORDER BY CONSTRAINT_SCHEMA,CONSTRAINT_NAME;
168+
--echo # Stop the server
169+
--source include/shutdown_mysqld.inc
170+
171+
#Remove copied files
172+
--remove_file $MYSQL_TMP_DIR/data57.zip
173+
174+
--force-rmdir $MYSQL_TMP_DIR/data57
175+
176+
--echo # Restart the server with default options.
177+
--source include/start_mysqld.inc

‎sql/CMakeLists.txt

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -71,6 +71,7 @@ SET(DD_SOURCES
7171
dd/dd_tablespace.cc
7272
dd/dd_trigger.cc
7373
dd/dd_view.cc
74+
dd/dd_upgrade.cc
7475

7576
dd/impl/bootstrapper.cc
7677
dd/impl/dd.cc

‎sql/dd/cache/dictionary_client.h

Lines changed: 23 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -802,6 +802,22 @@ class Dictionary_client
802802
template <typename T>
803803
bool update_and_invalidate(T* object);
804804

805+
806+
/**
807+
This function is a wrapper to the function below.
808+
809+
It calls add_and_reset_id() with value 'false' for the
810+
second parameter.
811+
812+
@tparam T Dictionary object type.
813+
@param object Object to be added to the shared cache
814+
and the object registry.
815+
*/
816+
817+
template <typename T>
818+
void add_and_reset_id(T* object);
819+
820+
805821
/**
806822
Add a new dictionary object and assign an id.
807823
@@ -811,20 +827,23 @@ class Dictionary_client
811827
client and added to the local registry. The object must be released
812828
afterwards,
813829
830+
The id should reset to 1 if we have cleared DD cache.
831+
814832
@note This function is only to be used during server start.
815833
816834
@note The new object will be owned by the shared cache. Thus, the
817835
dictionary user may not delete the object. Instead, the
818836
object must be released in the same way as other dictionary
819837
objects.
820838
821-
@tparam T Dictionary object type.
822-
@param object Object to be added to the shared cache
823-
and the object registry.
839+
@tparam T Dictionary object type.
840+
@param reset_id Reset id to 1
841+
@param object Object to be added to the shared cache
842+
and the object registry.
824843
*/
825844

826845
template <typename T>
827-
void add_and_reset_id(T* object);
846+
void add_and_reset_id(T* object, bool reset_id);
828847

829848

830849
/**

‎sql/dd/dd.h

Lines changed: 15 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -20,17 +20,27 @@ namespace dd {
2020

2121
///////////////////////////////////////////////////////////////////////////
2222

23+
// enum type to pass to init() function.
24+
enum class enum_dd_init_type
25+
{
26+
DD_INITIALIZE= 1,
27+
DD_RESTART_OR_UPGRADE,
28+
DD_POPULATE_UPGRADE,
29+
DD_DELETE
30+
};
31+
32+
2333
/**
24-
Initialize data dictionary upon server startup or install data
25-
dictionary for the first time.
34+
Initialize data dictionary upon server startup, server startup on old
35+
data directory or install data dictionary for the first time.
2636
27-
@param install - If true, creates data dictionary tables. Else
28-
it initializes data dictionary.
37+
@param dd_init - Option for initialization, population or deletion
38+
of data dictionary.
2939
3040
@return false - On success
3141
@return true - On error
3242
*/
33-
bool init(bool install);
43+
bool init(enum_dd_init_type dd_init);
3444

3545

3646
/**

0 commit comments

Comments
 (0)