|
1 | 1 | Name: sqlite
|
2 | 2 | URL: https://sqlite.org/
|
3 |
| -Version: 3.31.1 |
| 3 | +Version: 3.34.0 |
| 4 | +CPEPrefix: cpe:/a:sqlite:sqlite:3.34.0 |
4 | 5 | Included In Release: Yes
|
5 | 6 | Security Critical: Yes
|
6 | 7 | License: Public domain
|
7 | 8 |
|
8 |
| -1) Managing differences between SQLite core and Chromium's version. |
9 |
| -2) Making changes to Chromium SQLite. |
10 |
| -3) Import new release of SQLite. |
11 |
| -4) Running SQLite's test suite within the Chromium checkout. |
| 9 | +Chromium maintains a mirror of the official SQLite Git repository at |
| 10 | +https://chromium.googlesource.com/chromium/deps/sqlite. Generally the stock |
| 11 | +upstream release branch is used, but occasionally unreleased upstream bug |
| 12 | +fixes are backported to the release branch. |
12 | 13 |
|
13 |
| ---- |
14 |
| - |
15 |
| -1) Managing differences between SQLite core and Chromium's version. |
16 |
| - |
17 |
| -Chromium maintains some differences WRT SQLite, for reasons beyond this |
18 |
| -document's remit. Some differences are bugs we have found and fixed (and |
19 |
| -hopefully upstreamed), some are fixes we've backported from a later version of |
20 |
| -SQLite, and some our local changes unlikely to ever be upstreamed. New versions |
21 |
| -of SQLite are imported every year or two, at which point the changes need to be |
22 |
| -reviewed for continued applicability, and sometimes adjusted to reflect upstream |
23 |
| -code changes. |
24 |
| - |
25 |
| -To this end, the repository contains a reference copy of the SQLite source code |
26 |
| -as of the last import, plus a series of patches which can be applied to |
27 |
| -re-create the current trunk code. These patches are generated and processed by |
28 |
| -git, with the intention of re-creating a commit series so that importers can use |
29 |
| -their regular revision-control knowledge to manage import merges. |
30 |
| - |
31 |
| -The directory structure is as follows. Files common to all third_party projects |
32 |
| -(BUILD.GN, OWNERS, LICENSE) are omitted. |
33 |
| - |
34 |
| -* sqlite-src-*/ - Upstream source code, without any modifications. The number |
35 |
| - after src- is a release version or a snapshot number (which is |
36 |
| - a commit time). See https://www.sqlite.org/cgi/src/timeline |
37 |
| -* patches/ - Our patches to the currently used release, formatted by git using |
38 |
| - the UNIX mailbox format. The patches can be applied with git am, |
39 |
| - and created with git format-patch. |
40 |
| -* patched/ - The currently used source code, with our patches applied. |
41 |
| -* amalgamation/ - The supported method of using SQLite is via an amalgamation |
42 |
| - build, which merges all the code in one .c file and one .h |
43 |
| - file. See https://www.sqlite.org/amalgamation.html |
44 |
| -* amalgamation/config.h - Linux build configuration |
45 |
| -* scripts/ - Scripts that generate the files in the amalgamation |
46 |
| -* sqlite.h - The header used by the rest of Chromium to include SQLite. This |
47 |
| - forwards to amalgamation/sqlite3.h |
48 |
| -* fuzz/ - Google OSS-Fuzz (ClusterFuzz) testing for Chromium's SQLite build |
49 |
| - |
50 |
| ---- |
51 |
| - |
52 |
| -2) Making changes to Chromium SQLite. |
53 |
| - |
54 |
| -third_party/sqlite/patched is the patched source from SQLite. This is used to |
55 |
| -generate the amalgamation, a concatenation of all of the files into a giant |
56 |
| -sqlite3.c. To prototype, edit in patched/, then call |
57 |
| - ./scripts/generate_amalgamation.sh |
58 |
| -to regenerate sqlite3.c. The code in patched/ is much easier to edit, and the |
59 |
| -SQLite test framework can easily be run. During development it may be |
60 |
| -convenient to modify BUILD.gn based on patched/main.mk to just pull in the |
61 |
| -patched/ files rather than sqlite3.c. |
62 |
| - |
63 |
| -Once your patch is complete, squash it down into a reasonable CL, then |
64 |
| -re-generate the patches. This is a truncated version of the import flow. The |
65 |
| -following is written like a shell script to allow copy/paste to a shell, ignore |
66 |
| -comments and change the obvious lines. These instructions should work on Linux |
67 |
| -or OSX. They may assume a modern version of git. |
68 |
| - |
69 |
| -# The steps below are easier if done in the SQLite directory. |
70 |
| -cd third_party/sqlite |
71 |
| - |
72 |
| -# Must match the version in //third_party/sqlite/sqlite-src-xxxxxxx. |
73 |
| -# This is SQLite's version number, and uses upstream's convention. |
74 |
| -export BASE=3310100 |
75 |
| -export GNU_SED=sed # OSX: "brew install gnu-sed", then use "gsed" here. |
76 |
| - |
77 |
| -#### Create a reference branch. |
78 |
| -git new-branch sqlite-base |
79 |
| -git rm -rf patched |
80 |
| -cp -r sqlite-src-${BASE}/ patched |
81 |
| -# Clean up trailing whitespace and CRLF so any patches look clean. |
82 |
| -find patched/ -type f -not -iname "*.db" -not -iname "*.eps" \ |
83 |
| - -not -iname "*.ico" -not -iname "*.jpg" -not -iname "*.pfx" \ |
84 |
| - -not -iname "*.png" -not -iname "*.tiff" -not -iname "*.vsix" \ |
85 |
| - -exec ${GNU_SED} --in-place 's/[[:space:]]\+$//' {} \+ |
86 |
| -git add patched/ |
87 |
| -git clean -i -d -x patched # Make sure no file is git-ignored. |
88 |
| -git commit -m "Squash: Reset SQLite patched/ to sqlite-src-${BASE}." |
89 |
| -# This branch will not build. It will be used for rebasing, then deleted. |
90 |
| - |
91 |
| -#### Create a reference branch with patches applied. |
92 |
| -git new-branch --upstream-current sqlite-dev |
93 |
| -git am --keep-non-patch --ignore-space-change patches/*.patch |
94 |
| -git diff origin/master patched/ |
95 |
| -# This branch should be identical to master. |
96 |
| - |
97 |
| -#### Develop and validate the change, or cherry-pick it from a dev branch. |
98 |
| -# The goal is to have a set of reasonably-independent CLs which can be |
99 |
| -# understood separately, so that future importers can sensibly determine how to |
100 |
| -# handle conflicts. So use git-rebase and slipstream fixups back into existing |
101 |
| -# patches, or add a new patch. |
102 |
| -./scripts/generate_amalgamation.sh |
103 |
| -git cl format amalgamation/rename_exports.h |
104 |
| -cd ../.. |
105 |
| -ninja -C out/Default |
106 |
| -# Check that extract_sqlite_api.py added chrome_ to all exported symbols. |
107 |
| -# Only "_fini" and "_init" should be unprefixed. |
108 |
| -nm -B out/Default/libchromium_sqlite3.so | cut -c 18- | sort | grep '^T' |
109 |
| -out/Default/sql_unittests |
110 |
| -third_party/blink/tools/run_web_tests.py -t Default storage/websql/ |
111 |
| -cd third_party/sqlite |
112 |
| - |
113 |
| -#### Create the review. |
114 |
| -# Rebuild the patch set. |
115 |
| -git rm patches/* |
116 |
| -git format-patch --output-directory=patches --zero-commit \ |
117 |
| - sqlite-base..sqlite-dev |
118 |
| - |
119 |
| -### Document and link any backported patches to the upstream repository URL |
120 |
| -### and crbug. |
121 |
| -# Under the *.patch's added by this change, the below information should be |
122 |
| -# added between the "Subject: *" line and the first "---" line. For an example, |
123 |
| -# please reference https://crrev.com/c/1480822/2/third_party/sqlite/patches/0008-Adjustments-to-the-page-cache-to-try-to-avoid-harmle.patch. |
124 |
| -# |
125 |
| -# This backports https://www.sqlite.org/src/info/XXX |
126 |
| -# |
127 |
| -# Bug: XXX |
128 |
| - |
129 |
| -### Commit and create CL |
130 |
| -git add amalgamation/ |
131 |
| -git add patches/ |
132 |
| -git commit -m "Squash: regenerate amalgamation and patches." |
133 |
| -git branch --set-upstream-to=origin/master |
134 |
| -git cl upload --squash |
135 |
| - |
136 |
| ---- |
137 |
| - |
138 |
| -3) Import a new SQLite release. |
139 |
| - |
140 |
| -Importing a new SQLite involves merging our local changes with SQLite's changes. |
141 |
| -Like any other merge, this may involve modifying some commits and dropping |
142 |
| -others. The basic idea below is to generate git branches to work with: |
143 |
| - |
144 |
| -* sqlite-new-upstream - new release code archived in a separate directory |
145 |
| -* sqlite-old-base - current release without patches |
146 |
| -* sqlite-old - current release with patches mapped to git commits |
147 |
| -* sqlite-new-base - new release without patches |
148 |
| -* sqlite-new - new release with patches mapped to git commits |
149 |
| -* sqlite-new-cl - new release in one git commit, for git cl upload |
150 |
| - |
151 |
| -We will upload sqlite-new-upstream as a massive (800k LOC+) CL that cannot |
152 |
| -possibly be reviewed, but is generated in an automated fashion. We will then |
153 |
| -squash sqlite-new to sqlite-new-upstream and obtain one CL that only contains |
154 |
| -diffs. The second CL is still large, but it's a fraction of the first |
155 |
| -(automated) CL. |
156 |
| - |
157 |
| -# The steps below are easier if done in the SQLite directory. |
158 |
| -cd third_party/sqlite |
159 |
| - |
160 |
| -# The numbers below are SQLite version numbers, and use upstream's convention |
161 |
| -# for tagging release binaries and source zipballs. |
162 |
| -export OLD=3310100 |
163 |
| -export NEW=3320000 |
164 |
| -export GNU_SED=sed # OSX: "brew install gnu-sed", then use "gsed" here. |
165 |
| -export YEAR=2020 |
166 |
| - |
167 |
| -#### Download and unpack the new SQLite release. |
168 |
| -git new-branch sqlite-new-upstream |
169 |
| -# URL from "Alternative Source Code Formats" at https://sqlite.org/download.html |
170 |
| -curl https://sqlite.org/${YEAR}/sqlite-src-${NEW}.zip > upstream.zip |
171 |
| -mkdir sqlite-src-${NEW} |
172 |
| -unzip ./upstream.zip -d sqlite-src-${NEW} |
173 |
| -rm ./upstream.zip |
174 |
| -mv sqlite-src-${NEW}/sqlite-*/* sqlite-src-${NEW}/ |
175 |
| -rm -r sqlite-src-${NEW}/sqlite-*/.fossil-settings |
176 |
| -rmdir sqlite-src-${NEW}/sqlite-*/ |
177 |
| -rm -r sqlite-src-${NEW}/compat |
178 |
| -xdg-open sqlite-src-${NEW} # Make sure everything looks right. |
179 |
| - |
180 |
| -#### Add the new release code in a separate CL, for code review sanity. |
181 |
| -git add sqlite-src-${NEW} # Committing the code as downloaded, on purpose. |
182 |
| -git clean -i -d -x sqlite-src-${NEW} # Make sure no file is git-ignored. |
183 |
| -git commit -m "sqlite: Add code for release ${NEW}" |
184 |
| -git cl upload # Have the new code in a separate CL. |
185 |
| - |
186 |
| -#### Create a branch for the old SQLite release's upstream version. |
187 |
| -git new-branch sqlite-old-base |
188 |
| -git rm -rf patched |
189 |
| -cp -r sqlite-src-${OLD}/ patched |
190 |
| -# Clean up trailing whitespace and CRLF so any patches look clean. |
191 |
| -find patched/ -type f -not -iname "*.db" -not -iname "*.eps" \ |
192 |
| - -not -iname "*.ico" -not -iname "*.jpg" -not -iname "*.pfx" \ |
193 |
| - -not -iname "*.png" -not -iname "*.tiff" -not -iname "*.vsix" \ |
194 |
| - -exec ${GNU_SED} --in-place 's/[[:space:]]\+$//' {} \+ |
195 |
| -git add patched/ |
196 |
| -git clean -i -d -x patched # Make sure no file is git-ignored. |
197 |
| -git commit -m "Squash: Reset SQLite patched/ to sqlite-src-${OLD}." |
198 |
| -# This branch will not build. It will be used for rebasing, then deleted. |
199 |
| - |
200 |
| -#### Create a branch for our old SQLite code, with patches mapped to commits. |
201 |
| -git new-branch --upstream-current sqlite-old |
202 |
| -git am --keep-non-patch --ignore-space-change patches/*.patch |
203 |
| -git diff origin/master patched/ |
204 |
| -# This branch should be identical to master. |
205 |
| - |
206 |
| -#### Create a branch for the new SQLite release's upstream version. |
207 |
| -git checkout sqlite-old-base |
208 |
| -git new-branch --upstream-current sqlite-new-base |
209 |
| -git rm -rf patched |
210 |
| -git checkout sqlite-new-upstream -- sqlite-src-${NEW}/ |
211 |
| -git mv sqlite-src-${NEW}/ patched |
212 |
| -# Clean up trailing whitespace and CRLF so any patches look clean. |
213 |
| -find patched/ -type f -not -iname "*.db" -not -iname "*.eps" \ |
214 |
| - -not -iname "*.ico" -not -iname "*.jpg" -not -iname "*.pfx" \ |
215 |
| - -not -iname "*.png" -not -iname "*.tiff" -not -iname "*.vsix" \ |
216 |
| - -exec ${GNU_SED} --in-place 's/[[:space:]]\+$//' {} \+ |
217 |
| -git add patched/ |
218 |
| -git clean -i -d -x patched # Make sure no file is git-ignored. |
219 |
| -git commit -m "Squash: Reset SQLite patched/ to sqlite-src-${NEW}." |
220 |
| -# This branch will not build. It will be used for rebasing, then deleted. |
221 |
| - |
222 |
| -#### Create a branch for updating our patches. |
223 |
| -git checkout sqlite-old |
224 |
| -git new-branch --upstream-current sqlite-new |
225 |
| -# Rebase our patches, which are mapped to separate commits, onto the new |
226 |
| -# release. There will be merge conflicts that must be fixed. This is the |
227 |
| -# interesting part of the work. |
228 |
| -git rebase sqlite-new-base |
229 |
| - |
230 |
| -#### Finally, create the branch that we'll upload. |
231 |
| -git new-branch --upstream-current sqlite-new-cl |
232 |
| -./scripts/generate_amalgamation.sh |
233 |
| -git cl format amalgamation/rename_exports.h |
234 |
| - |
235 |
| -#### Copy any new entries to the seed-corpus for dbfuzz2. |
236 |
| -# We use a set of seed databases for fuzzing SQLite's resilience to database |
237 |
| -# corruption. Sometimes, new seed databases are added upstream. |
238 |
| -# Find any files of the pattern patched/test/dbfuzz2-seed*.db, and copy them to |
239 |
| -# the fuzz/db_corpus directory. |
240 |
| -cp --no-clobber patched/test/dbfuzz2-seed*.db fuzz/db_corpus |
241 |
| - |
242 |
| -#### Validate the upgrade. |
243 |
| -# The goal is to have a set of reasonably-independent CLs which can be |
244 |
| -# understood separately, so that future importers can sensibly determine how to |
245 |
| -# handle conflicts. So use git-rebase and slipstream fixups back into their |
246 |
| -# original CL until everything builds and works. |
247 |
| -cd ../.. |
248 |
| -autoninja -C out/Default |
249 |
| -# Check that extract_sqlite_api.py added chrome_ to all exported symbols. |
250 |
| -# Only "_fini" and "_init" should be unprefixed. |
251 |
| -nm -B out/Default/libchromium_sqlite3.so | cut -c 18- | sort | grep '^T' |
252 |
| -out/Default/sql_unittests |
253 |
| -third_party/blink/tools/run_web_tests.py -t Default storage/websql/ |
254 |
| -cd third_party/sqlite |
255 |
| - |
256 |
| -#### Create the review. |
257 |
| -# Rebuild the patch set. |
258 |
| -git rm patches/* |
259 |
| -git format-patch --output-directory=patches --zero-commit \ |
260 |
| - sqlite-new-base..sqlite-new |
261 |
| -git add amalgamation/ |
262 |
| -git add patches/ |
263 |
| -git commit -m "Squash: regenerate amalgamation and patches." |
264 |
| -git branch --set-upstream-to=origin/master |
265 |
| -git cl upload --squash |
266 |
| -# Example commit message (https://crrev.com/c/1699400): |
267 |
| -# sqlite: Upgrade from 3.28.0 to 3.29.0 |
268 |
| -# |
269 |
| -# sqlite 3.29.0 update changes: |
270 |
| -# https://www.sqlite.org/releaselog/3_29_0.html |
271 |
| -# |
272 |
| -# Bug: 983713 |
273 |
| - |
274 |
| -#### Drop the old version of SQLite. |
275 |
| -git new-branch sqlite-rm-old |
276 |
| -git rm -r sqlite-src-${OLD} |
277 |
| -git commit -m "sqlite: Remove source code for old release ${OLD}." |
278 |
| -git cl upload |
279 |
| - |
280 |
| -### Update README.chromium to reference the new sqlite version number. |
281 |
| -# Update: |
282 |
| -# (1) the header's "Version" |
283 |
| -# (2) `export OLD` and `export NEW` values. |
284 |
| - |
285 |
| --------------------------------------------- |
286 |
| - |
287 |
| -4) Running SQLite's test suite within the Chromium checkout. |
288 |
| - |
289 |
| -TODO(pwnall): This hasn't been tried out for at least a year. |
290 |
| - |
291 |
| -Prerequisites: The test suite requires tcl-dev and libicu-dev. Install those on |
292 |
| -Ubuntu like: |
293 |
| - sudo apt-get install tcl8.6-dev libicu-dev |
294 |
| -On macOS, I use Homebrew: |
295 |
| - brew install icu4c tcl-tk |
296 |
| - export PATH="$(brew --prefix icu4c)/bin:$(brew --prefix tcl-tk)/bin:$PATH" |
297 |
| - |
298 |
| -Run the commands in scripts/generate_amalgamation.sh, but replace the "make" |
299 |
| -command with "make test". |
300 |
| - |
301 |
| -make test > /tmp/test.log |
302 |
| -egrep 'errors out of' /tmp/test.log |
303 |
| -# Show broken tests: |
304 |
| -egrep 'Failures on these tests:' /tmp/test.log |
305 |
| -# Broken tests will also show lines ending in "..." instead of "... Ok". |
306 |
| - |
307 |
| -In version 3.10.2 on OSX 10.11.2, I see: |
308 |
| - 6 errors out of 139819 tests |
309 |
| -The failed tests are: |
310 |
| - pager4-1.3 pager4-1.4 pager4-1.5 pager4-1.9 pager4-1.10 pager4-1.11 |
311 |
| -This is due to the change in os_unix.c fileHasMoved() to support WebDatabase. |
312 |
| -Commenting out the early return allows them to succeed. |
313 |
| - |
314 |
| -In version 3.10.2 on Ubuntu 14.04.3 I see: |
315 |
| - 9 errors out of 140309 tests |
316 |
| -The failed tests are: |
317 |
| - oserror-1.1.1 oserror-1.1.2 oserror-1.1.3 |
318 |
| - pager4-1.3 pager4-1.4 pager4-1.5 pager4-1.9 pager4-1.10 pager4-1.11 |
319 |
| -The oserror tests fail because there are too many fds available, and can be |
320 |
| -fixed by running "ulimit -n 1024" before the test. The pager4 tests are failing |
321 |
| -for the same reason as for OSX. |
322 |
| - |
323 |
| --- |
324 |
| - |
325 |
| -NOTE(pwnall): On Ubuntu it is possible to run the tests in a tmpfs something |
326 |
| -like: |
327 |
| - |
328 |
| -TMPFS=/dev/shm/sqlite_build |
329 |
| -BUILD=$PWD |
330 |
| -mkdir $TMPFS |
331 |
| -(cd $TMPFS ; $BUILD/testfixture $BUILD/../test/veryquick.test >/tmp/test.log) |
332 |
| - |
333 |
| -This is faster, but it is plausible that different things are being tested than |
334 |
| -real-world use. |
| 14 | +The official upstream SQLite Git mirror is at https://github.com/sqlite/sqlite. |
0 commit comments